100% Containers Powered Carpooling

从我们将100%的生产工作量转移到容器中到现在已经3年了。我们将概述我们的基础设施和环境,介绍我们的服务发现解决方案,并展示我们所称的“后端高可用性支柱”。我们将揭示成功地将状态应用程序操作到容器中的秘密,并详细说明我们过去和未来的挑战:
-2014年:配置管理(Chef)致力于Prem服务器
-2015年:RKT集装箱通过车队在数百台Prem服务器上协调
-2017年:通过Kubernetes在数百台prem服务器上协调的RKT容器
-2018年:通过云提供商(WIP)上的Kubernetes协调的Docker容器

展开查看详情

1.100% Containers Powered Carpooling

2.Maxime Fouilleul Database Reliability Engineer

3. BlaBlaCar - Facts & Figures Infrastructure Ecosystem - 100% containers powered carpooling Today’s agenda Stateful Services into containers - MariaDB as an example Next challenges - Kubernetes, the Cloud

4.BlaBlaCar Facts & Figures

5.Facts and Figures 60 million 30 million mobile members app downloads iPhone and Android Founded 15 million in 2006 travellers /quarter 1 million tonnes Currently in less CO2 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, In the past year Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey.

6. Our prod data ecosystem MariaDB ElasticSearch Redis Cassandra PostgreSQL Kafka Transactional Search Volatile Distributed Spatial Stream

7.Infrastructure Ecosystem 100% containers powered carpooling

8.Why containers?

9.Homogeneous Hardware From this srv_001 srv_006 srv_009 srv_013 svc_001 svc_006 svc_009 svc_013 srv_007 srv_002 svc_007 svc_002 srv_003 srv_010 srv_014 svc_003 srv_008 svc_010 svc_014 srv_004 svc_008 srv_011 svc_004 svc_011 srv_005 srv_012 svc_005 svc_012

10.Homogeneous Hardware To that srv_001 srv_003 srv_005 srv_007 svc_001 svc_008 svc_013 svc_002 svc_003 svc_004 svc_005 svc_011 srv_002 srv_004 srv_006 srv_008 svc_006 svc_009 svc_014 svc_007 svc_010 svc_012

11.Homogeneous Hardware - “Pets vs Cattle” Easier to replace broken hardware Cost Effective Easier to manage

12. Homogeneous Deployment trip-meeting-point application redis trip-meeting-point cat ./prod-dc1/services/trip-meeting-point/service-manifest.yml cat ./prod-dc1/services/redis-meeting-point/service-manifest.yml --- --- containers: containers: - aci.blbl.cr/aci-trip-meeting-point:20180928.145115-v-979da34 - aci.blbl.cr/aci-redis:4.0.2-1 - aci.blbl.cr/aci-go-synapse:15-40 - aci.blbl.cr/aci-redis-dictator:20 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-logshipper:27 - aci.blbl.cr/aci-prometheus-redis-exporter:0.12.2-1 nodes: nodes: - hostname: trip-meeting-point1 - hostname: redis-meeting-point1 gelf: fleet: level: INFO - MachineMetadata=rack=110 fleet: - Conflicts=*redis-meeting-point* - MachineMetadata=rack=110 - Conflicts=*trip-meeting-point* - hostname: redis-meeting-point2 - hostname: trip-meeting-point2 fleet: fleet: - MachineMetadata=rack=210 - MachineMetadata=rack=210 - Conflicts=*redis-meeting-point* - Conflicts=*trip-meeting-point* - hostname: trip-meeting-point3 - hostname: redis-meeting-point3 fleet: fleet: - MachineMetadata=rack=310 - MachineMetadata=rack=310 - Conflicts=*trip-meeting-point* - Conflicts=*redis-meeting-point* ggn prod-dc1 trip-meeting-point update -y ggn prod-dc1 redis-meeting-point update -y

13. Volatile by design trip-meeting-point dependencies cat ./prod-dc1/services/trip-meeting-point/service-manifest.yml trip-meeting-point --- containers: - aci.blbl.cr/aci-trip-meeting-point:20180928.145115-v-979da34 aci-trip-meeting-point aci-go-synapse aci-go-nerve aci-logshipper - aci.blbl.cr/aci-go-synapse:15-41 - aci.blbl.cr/aci-go-nerve:21-27 - aci.blbl.cr/aci-logshipper:27 [...] cat ./aci-trip-meeting-point/aci-manifest.yml --- name: aci.blbl.cr/aci-trip-meeting-point:{{.version}} aci: aci-java aci-hindsight dependencies: - aci.blbl.cr/aci-java:1.8.181-2 [...] cat ./aci-java/aci-manifest.yml --- name: aci.blbl.cr/aci-java:1.8.181-2 aci: dependencies: - aci.blbl.cr/aci-debian:9.5-9 aci-debian aci-common - aci.blbl.cr/aci-common:7

14.Volatile - When should I redeploy? A change in my own app/container: “immutable” A change on a sidecar container or its dependencies Noisy neighbours: “mutualization” When you are ready for instability your are HA

15.How?

16. Infrastructure Ecosystem front1 php nerve nginx mysql-main1 zookeeper Service Discovery Service Codebase create monitoring mysqld synapse dgr build nerve monitoring synapse nerve store Container Registry etcd run rkt PODs “Distributed init system” ggn fleet cluster 1 type of hardware Hardware CoreOS host bare-metal servers 3 disk profiles

17. Infrastructure Ecosystem front1 php nerve nginx mysql-main1 zookeeper Service Discovery Service Codebase create monitoring mysqld synapse dgr build nerve monitoring synapse nerve store Container Registry etcd run rkt PODs “Distributed init system” ggn helm fleet kubernetes 1 type of Hardware host bare-metal servers CoreOS hardware 3 disk profiles

18.Service Discovery Zookeeper backend pod client pod go-synapse go-nerve /database/node1 /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends

19.Stateful Services into containers MariaDB as an example

20.“Stateful” and “volatile by design”?

21.The recipe/prereqs/pillars to succeed: Abolish Slavery Be Quiet! Build Smart “For a given service, “A node should be able “Services can be every node have the to restart without operate by any SRE” same role” impacting the app”

22.MariaDB as an example

23. Abolish Slavery “For a given service, every node have the same role”

24.Asynchronous vs. Synchronous MariaDB Cluster means Master No Single Point of Slave Slave Slave Failure No Replication Lag Auto States Transfers wsrep wsrep wsrep wsrep wsrep MariaDB Cluster As fast as the slowest

25.The Target Writes Reads MariaDB Cluster Containers wsrep wsrep wsrep wsrep wsrep Writes go on one MariaDB Cluster node Reads are balanced on the others

26.How to hit the target? Service Discovery

27. Nerve - Track and report service status # cat env/prod-dc1/services/mysql-main/attributes/nerve.yml # zookeepercli -c lsr /services/mysql/main --- mysql-main1_192.168.1.2_ba0f1f8b override: mysql-main2_192.168.1.3_734d63da nerve: mysql-main3_192.168.1.4_dde45787 services: # zookeepercli -c get - name: "mysql-main" /services/mysql/main/mysql-main1_192.168.1.2_ba0f1f8b3 port: 3306 { reporters: "available":true, - {type: zookeeper, path: /services/mysql/main} "host":"192.168.1.2", checks: "port":3306, - type: sql "name":"mysql-main1", driver: mysql "weight":255, datasource: "local_mon:local_mon@tcp(127.0.0.1:3306)/" "labels":{ "host":"r10-srv4" } }

28. Synapse  -  Service discovery router # cat env/prod-dc1/services/tripsearch/attributes/synapse.yml # cat env/prod-dc1/services/tripsearch/attributes/tripsearch.yml --- —- override: override: synapse: tripsearch: services: database: - name: mysql-main_read read: path: /services/mysql/main host: localhaproxy port: 3307 database: tripsearch - name: mysql-main_write user: tripsearch_rd path: /services/mysql/main port: 3307 port: 3308 write: serverOptions: backup host: localhaproxy serverSort: date database: tripsearch user: tripsearch_wr port: 3308

29. Be Quiet! “A node should be able to restart without impacting the app”