Automate App Operation

1 .Automate App Operation Hongchao Deng hongchao.deng@coreos.com 

2 .App = ? 

3 .App = Code + Config 

4 .故事开始于… 

5 .

6 .

7 .

8 .Development Deployment DNS 想法 程序 LoadBalancer 打包 实现 发布 docker build docker push 

9 .Demo 

10 .Demo 

11 .Deploy App Container • Docker/OCI • Standard app packaging format • Kubernetes/Swarm • Resource scheduling, cluster management 

12 .It is easy to deploy stateless apps. But how to deploy stateful apps? 

13 .How to Deploy • Database: PostgreSQL, MySQL, TiDB • Coordination service: etcd, ZooKeeper • Streaming: Kafka, Heron • Big data: Spark, Hadoop • Storage: Ceph, GlusterFS • Logging: ElasticSearch • Monitoring: Prometheus 

14 . Deploying those are much harder than stateless web apps 

15 .Prometheus 

16 . Complex dependencies Alert Manager Service Service Monitor Prometheu Alerting Rule 

17 .etcd 

18 . Membership Configuration etcd --name=example-etcd-cluster-0002 ... --initial-cluster= example-etcd-cluster-0001=http://example-etcd-cluster-0001.example-etcd- cluster.default.svc.cluster.local:2380, example-etcd-cluster-0000=http://example-etcd-cluster-0000.example-etcd- cluster.default.svc.cluster.local:2380, example-etcd-cluster-0002=http://example-etcd-cluster-0002.example-etcd- cluster.default.svc.cluster.local:2380 

19 .Self-updating Kubernetes 

20 . Update Strategy Target Version APIServer: v1.6.0 etcd: v3.1.4 

21 . Update Strategy Target Version APIServer: v1.6.0 -Backup before upgrade - Rolling upgrade etcd: v3.1.4 - Check upgrade path - Rollback within a minor 

22 .What makes them difficult to deploy and manage? 

23 .Complex operation logic 

24 .Introduce Operator 

25 . Current state of a cluster Observe • cluster size is 3 • cluster version is v1.2.3 Differences from desired config Analyze • cluster size should be 5 • cluster version should be v1.3.0 How to get there Act • configure and add members to 5 • upgrade member to v1.3.0 safely one-by-one 

26 . etcd Operator Common Tasks Advanced • Resize • Restore • Upgrade • TLS • Backup • Monitoring/Alerting • Failover 

27 .Declarative API 

28 . Create a cluster 

29 .kubectl create - f