100% Containers Powered Carpooling

1 .100% Containers Powered Carpooling 

2 .Maxime Fouilleul Database Reliability Engineer 

3 . BlaBlaCar - Facts & Figures Infrastructure Ecosystem - 100% containers powered carpooling Today’s agenda Stateful Services into containers - MariaDB as an example Next challenges - Kubernetes, the Cloud 

4 .BlaBlaCar Facts & Figures 

5 .Facts and Figures 60 million 30 million mobile members app downloads iPhone and Android Founded 15 million in 2006 travellers /quarter 1 million tonnes Currently in less CO2 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, In the past year Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. 

6 . Our prod data ecosystem MariaDB ElasticSearch Redis Cassandra PostgreSQL Kafka Transactional Search Volatile Distributed Spatial Stream 

7 .Infrastructure Ecosystem 100% containers powered carpooling 

8 .Why containers? 

9 .Homogeneous Hardware From this srv_001 srv_006 srv_009 srv_013 svc_001 svc_006 svc_009 svc_013 srv_007 srv_002 svc_007 svc_002 srv_003 srv_010 srv_014 svc_003 srv_008 svc_010 svc_014 srv_004 svc_008 srv_011 svc_004 svc_011 srv_005 srv_012 svc_005 svc_012 

10 .Homogeneous Hardware To that srv_001 srv_003 srv_005 srv_007 svc_001 svc_008 svc_013 svc_002 svc_003 svc_004 svc_005 svc_011 srv_002 srv_004 srv_006 srv_008 svc_006 svc_009 svc_014 svc_007 svc_010 svc_012 

11 .Homogeneous Hardware - “Pets vs Cattle” Easier to replace broken hardware Cost Effective Easier to manage 

12 . Homogeneous Deployment trip-meeting-point application redis trip-meeting-point cat ./prod-dc1/services/trip-meeting-point/service-manifest.yml cat ./prod-dc1/services/redis-meeting-point/service-manifest.yml --- --- containers: containers: - aci.blbl.cr/aci-trip-meeting-point:20180928.145115-v-979da34 - aci.blbl.cr/aci-redis:4.0.2-1 - aci.blbl.cr/aci-go-synapse:15-40 - aci.blbl.cr/aci-redis-dictator:20 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-logshipper:27 - aci.blbl.cr/aci-prometheus-redis-exporter:0.12.2-1 nodes: nodes: - hostname: trip-meeting-point1 - hostname: redis-meeting-point1 gelf: fleet: level: INFO - MachineMetadata=rack=110 fleet: - Conflicts=*redis-meeting-point* - MachineMetadata=rack=110 - Conflicts=*trip-meeting-point* - hostname: redis-meeting-point2 - hostname: trip-meeting-point2 fleet: fleet: - MachineMetadata=rack=210 - MachineMetadata=rack=210 - Conflicts=*redis-meeting-point* - Conflicts=*trip-meeting-point* - hostname: trip-meeting-point3 - hostname: redis-meeting-point3 fleet: fleet: - MachineMetadata=rack=310 - MachineMetadata=rack=310 - Conflicts=*trip-meeting-point* - Conflicts=*redis-meeting-point* ggn prod-dc1 trip-meeting-point update -y ggn prod-dc1 redis-meeting-point update -y 

13 . Volatile by design trip-meeting-point dependencies cat ./prod-dc1/services/trip-meeting-point/service-manifest.yml trip-meeting-point --- containers: - aci.blbl.cr/aci-trip-meeting-point:20180928.145115-v-979da34 aci-trip-meeting-point aci-go-synapse aci-go-nerve aci-logshipper - aci.blbl.cr/aci-go-synapse:15-41 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-logshipper:27 [...] cat ./aci-trip-meeting-point/aci-manifest.yml --- name: aci.blbl.cr/aci-trip-meeting-point:{{.version}} aci: aci-java aci-hindsight dependencies: - aci.blbl.cr/aci-java:1.8.181-2 [...] cat ./aci-java/aci-manifest.yml --- name: aci.blbl.cr/aci-java:1.8.181-2 aci: dependencies: - aci.blbl.cr/aci-debian:9.5-9 aci-debian aci-common - aci.blbl.cr/aci-common:7 

14 .Volatile - When should I redeploy? A change in my own app/container: “immutable” A change on a sidecar container or its dependencies Noisy neighbours: “mutualization” When you are ready for instability your are HA 

15 .How? 

16 . Infrastructure Ecosystem front1 php nerve nginx mysql-main1 zookeeper Service Discovery Service Codebase create monitoring mysqld synapse dgr build nerve monitoring synapse nerve store Container Registry etcd run rkt PODs “Distributed init system” ggn fleet cluster 1 type of hardware Hardware CoreOS host bare-metal servers 3 disk profiles 

17 . Infrastructure Ecosystem front1 php nerve nginx mysql-main1 zookeeper Service Discovery Service Codebase create monitoring mysqld synapse dgr build nerve monitoring synapse nerve store Container Registry etcd run rkt PODs “Distributed init system” ggn helm fleet kubernetes 1 type of Hardware host bare-metal servers CoreOS hardware 3 disk profiles 

18 .Service Discovery Zookeeper backend pod client pod go-synapse go-nerve /database/node1 /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends 

19 .Stateful Services into containers MariaDB as an example 

20 .“Stateful” and “volatile by design”? 

21 .The recipe/prereqs/pillars to succeed: Abolish Slavery Be Quiet! Build Smart “For a given service, “A node should be able “Services can be every node have the to restart without operate by any SRE” same role” impacting the app” 

22 .MariaDB as an example 

23 . Abolish Slavery “For a given service, every node have the same role” 

24 .Asynchronous vs. Synchronous MariaDB Cluster means Master No Single Point of Slave Slave Slave Failure No Replication Lag Auto States Transfers wsrep wsrep wsrep wsrep wsrep MariaDB Cluster As fast as the slowest 

25 .The Target Writes Reads MariaDB Cluster Containers wsrep wsrep wsrep wsrep wsrep Writes go on one MariaDB Cluster node Reads are balanced on the others 

26 .How to hit the target? Service Discovery 

27 . Nerve - Track and report service status # cat env/prod-dc1/services/mysql-main/attributes/nerve.yml # zookeepercli -c lsr /services/mysql/main --- mysql-main1_192.168.1.2_ba0f1f8b override: mysql-main2_192.168.1.3_734d63da nerve: mysql-main3_192.168.1.4_dde45787 services: # zookeepercli -c get - name: "mysql-main" /services/mysql/main/mysql-main1_192.168.1.2_ba0f1f8b3 port: 3306 { reporters: "available":true, - {type: zookeeper, path: /services/mysql/main} "host":"192.168.1.2", checks: "port":3306, - type: sql "name":"mysql-main1", driver: mysql "weight":255, datasource: "local_mon:local_mon@tcp(127.0.0.1:3306)/" "labels":{ "host":"r10-srv4" } } 

28 . Synapse  -  Service discovery router # cat env/prod-dc1/services/tripsearch/attributes/synapse.yml # cat env/prod-dc1/services/tripsearch/attributes/tripsearch.yml --- —- override: override: synapse: tripsearch: services: database: - name: mysql-main_read read: path: /services/mysql/main host: localhaproxy port: 3307 database: tripsearch - name: mysql-main_write user: tripsearch_rd path: /services/mysql/main port: 3307 port: 3308 write: serverOptions: backup host: localhaproxy serverSort: date database: tripsearch user: tripsearch_wr port: 3308 

29 . Be Quiet! “A node should be able to restart without impacting the app”