Polyglot persistence: utilizing open source databases as a Swiss pocket knife

在过去的几年里,Vidaxl已经成为欧洲慢消费品在线零售市场的领导者。当一家公司在过去的9年中实现了超过50%的同比增长时,几乎没有足够的时间来检修现有系统。这意味着现有系统将被扩展到其最大能力,并且通常通过使用各种各样的数据存储来获得额外的性能。
在快速增长的环境中,polyglot持久性占据主导地位,传统的一刀切的单语言策略已经结束。
VidaXL拥有广泛的数据存储领域,从传统的SQL数据存储(如MySQL或PostgreSQL)以及最新的负载平衡技术(如ProxysQL)到文档存储(如MongoDB)和搜索引擎(如Solr和ElasticSearch)。

展开查看详情

1. Polyglot persistence: utilizing open source databases as a Swiss pocket knife Art van Scheppingen - Senior (No)SQL DBA @ VidaXL Bart Oleś - Senior Support Engineer @ Severalnines

2.Who are we?

3.VidaXL Company Stats • Online retailer in (mostly) slow moving goods • Founded 2008 • 350M turnover, 40% growth yearly • 1500 employees (US, CN, AU, IN, RO, UA) • HQ in the Netherlands • 4 warehouses worldwide (NL, US and AU)

4.How does VidaXL sell its goods? • Own webshop platform in EU, US and AU • Warehouses in NL, US and AU • Selling on other platforms, e.g. Amazon, eBay • Allow selling on our own platform using Mirakl • B2B drop-shipments

5.VidaXL Technical Foundations • SAP as ERP system • Genesys as CS system • Webshop • Open source web-based development strategy • PHP / NodeJS • Docker • Cloudflare workers

6.VidaXL DevOps Datastores • MySQL • MariaDB (Galera) clusters • MySQL replication • PostgreSQL • SOLR • Elasticsearch • ELK • MongoDB • Couchbase • (RabbitMQ) • Prometheus

7.What is Polyglot Persistence? Using multiple specialized persistent stores rather than one single general-purpose database

8.Where does the term come from? • The way we work is changing • Enterprise applications are becoming more complex • Separate (devops/agile) teams • Ownership of applications • (Micro)services • Everyone has their preference • Various programming languages • Various storage systems

9.Where does the term come from? • Monoglot Programming • Only one programming language allowed • Readability • All code is in the same language • Support • One platform to support • Knowledge • Everybody is an expert • Is there a jack-of-all-trades language?

10.Where does the term come from? • Monoglot Programming • Only one programming language allowed • Readability • All code is in the same language • Support • One platform to support • Knowledge • Everybody is an expert • Is there a jack-of-all-trades language?

11.Monoglot programming •

12.Monoglot programming Carpenters actually use a broad variety of tools

13.Polyglot Programming • Polyglot Programming • Use programming languages for what they are good at • Flexibility • Use Java for a secure API • Use Scala for real time stream processing • Use Python for text analysis • Tie everything together using AngularJS • Knowledge • Everybody is expert at one or more languages

14.Polyglot Programming

15.Monoglot Persistence

16.Data storage landscape changes • Relational data stores (RDBMS) • Key-Value data stores (“NoSQL”) • Columnar data stores (OLAP) • Document data stores (NoSQL) • Graph data stores (GDB) • Big Data

17.Data storage landscape changes Software RDBMS Oracle, MySQL, PostgreSQL Key-Value Redis, Riak Columnar InfiniDB, Clickhouse Document MongoDB, Couchbase Graph Neo4J, Janusgraph Big Data Hadoop

18.Data storage landscape changes Software AWS Google RDBMS Oracle, MySQL, RDS, Aurora CloudSQL, Spanner PostgreSQL Key-Value Redis, Riak DynamoDB Datastore Columnar InfiniDB, Clickhouse Redshift BigQuery Document MongoDB, SimpleDB Bigtable Couchbase Graph Neo4J, Janusgraph Neptune Big Data Hadoop EMR Cloud Dataproc

19.Even Hadoop has become a polyglot

20.Polyglot Persistence • Complex problems require different storage systems • Use the right tool for the job, for example • Use PostgreSQL for financial data • Use MySQL for website contents • Use MongoDB for user profiles • Use Cassandra for real time streams • Use Neo4J for recommendation analysis

21.Use the right tool for the right job Document storage: MongoDB

22.Use the right tool for the right job Columnar storage: Cassandra

23.Use the right tool for the right job Graph storage: Neo4J

24.Polyglot Persistence

25.Polyglot Persistence at VidaXL Yes we certainly are polyglots!

26.Quick recap on our data stores • MySQL • MariaDB (Galera) clusters • MySQL replication • ProxySQL • PostgreSQL • SOLR • Elasticsearch • ELK • MongoDB • Couchbase • (RabbitMQ) • Prometheus

27.How did this happen? • Continuous growth • Hardly any time to overhaul existing systems • Transition from monolith to microservice architecture • For each microservice the most optimal solution has been chosen • Early adopters of new technology • Gaining advantage over competition

28.From monolith to microservice

29.From monolith to microservice