网易云 刘超 - 《基于Service+Mesh的海量容器管理平台实践》

1 .基于Service Mesh的 海量容器管理平台实践 

2 .关于我 • 刘超 • 网易云 解决方案总架构师 • 10余年云计算领域研发及架构经验，先后在EMC，CCTV证券资 讯频道，HP，华为，网易从事云计算和大数据架构工作 • 毕业于上海交通大学。 • 曾出版《Lucene应用开发揭秘》 • 多次作为邀请讲师参加Dockone容器技术大会，Segmentfault 开发者大会，InfoQ全球架构师峰会（明星讲师），CSDN SDCC大 会，51CTO WOTA大会等 • 知名技术博主，博客可搜索popsuper1982，多篇文章推荐至全 球最大IT社区CSDN首页及《程序员》杂志 • 在工作中积累了大量运营商系统，互联网金融系统，电商系统等 容器化和微服务化经验 www.163yun.com 

3 .目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com 

4 .企业上云三大架构 快速迭代 高并发 开发 应用架构 CAPEX 大数据分析 运维 IT架构 数据架构 运营 OPEX 数字化运营 www.163yun.com 

5 .容器技术的三种视角 微服务的交付形式Kubernetes 应用架构 IT架构 数据架构 轻量级的IT运维模式Swarm 资源利用率高的任务执行模式Mesos www.163yun.com 

6 .Kubernetes + Docker 是 Dev 和 Ops 融合的一个桥梁。 开发，构建，测试 服务发现 配置中心 Dev 熔断降级 Kubernetes + Docker Dockerfile Ops 镜像环境交付 提供资源，部署，运维 www.163yun.com 

7 .Kubernetes更加适合微服务和DevOps的设计 微服务设计 Kubernetes功能 设计要点一：API 网关 Ingress 设计要点二：无状态化，区分有状态的和无状态的应用。 无状态对应Deployment，有状态对应 StatefulSet 设计要点三：数据库的横向扩展。 headless service指向PaaS服务，或者StatefulSet部署 设计要点四：缓存 headless service指向PaaS服务，或者StatefulSet部署 设计要点五：服务拆分和服务发现 Service 设计要点六：服务编排与弹性伸缩 Deployment的Replicas 设计要点七：统一配置中心 ConfigMap 设计要点八：统一的日志中心 DaemonSet部署日志Agent 设计要点九：熔断，限流，降级 Service Mesh 设计要点十：全方位的监控 Cadvisor，DaemonSet部署监控Agent www.163yun.com 

8 .目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com 

9 .Service Mesh的范例Istio www.163yun.com 

10 .一切从envoy开始 • 轻量级的proxy • 静态配置，热加载，热重启 • 动态配置，拉取模式 Listener LDS Routes RDS Clusters CDS Endpoints EDS www.163yun.com 

11 .Envoy之静态配置 admin: access_log_path: /tmp/admin_access.log address: socket_address: { address: 127.0.0.1, port_value: 9901 } static_resources: listeners: - name: listener_0 address: Listener socket_address: { address: 127.0.0.1, port_value: 10000 } filter_chains: - filters: - name: envoy.http_connection_manager config: Routes stat_prefix: ingress_http codec_type: AUTO route_config: name: local_route virtual_hosts: - name: local_service Clusters domains: ["*"] routes: Endpoints - match: { prefix: "/" } route: { cluster: some_service } http_filters: - name: envoy.router clusters: - name: some_service connect_timeout: 0.25s type: STATIC lb_policy: ROUND_ROBIN hosts: [{ socket_address: { address: 127.0.0.2, port_value: 1234 }}] www.163yun.com 

12 .Envoy之动态配置 admin: access_log_path: /tmp/admin_access.log address: socket_address: { address: 127.0.0.1, port_value: 9901 } dynamic_resources: lds_config: api_config_source: api_type: GRPC cluster_names: [xds_cluster] cds_config: api_config_source: api_type: GRPC cluster_names: [xds_cluster] static_resources: clusters: - name: xds_cluster connect_timeout: 0.25s type: STATIC lb_policy: ROUND_ROBIN http2_protocol_options: {} hosts: [{ socket_address: { address: 127.0.0.3, port_value: 5678 }}] www.163yun.com 

13 .加入控制面Pilot www.163yun.com 

14 .Pilot路由策略 www.163yun.com 

15 .Pilot的负载均衡 www.163yun.com 

16 .目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com 

17 .这个简单例子背后发生的事情 www.163yun.com 

18 .Productpage是一个python程序 @app.route('/api/v1/products') def productsRoute(): return json.dumps(getProducts()), 200, {'Content-Type': 'application/json'} details = { @app.route('/api/v1/products/<product_id>') "name" : "http://details:9080", def productRoute(product_id): "endpoint" : "details", headers = getForwardHeaders(request) "children" : [] status, details = getProductDetails(product_id, headers) } return json.dumps(details), status, {'Content-Type': 'application/json'} ratings = { "name" : "http://ratings:9080", @app.route('/api/v1/products/<product_id>/reviews') "endpoint" : "ratings", def reviewsRoute(product_id): "children" : [] headers = getForwardHeaders(request) } status, reviews = getProductReviews(product_id, headers) return json.dumps(reviews), status, {'Content-Type': 'application/json'} reviews = { "name" : "http://reviews:9080", "endpoint" : "reviews", @app.route('/api/v1/products/<product_id>/ratings') "children" : [ratings] def ratingsRoute(product_id): } headers = getForwardHeaders(request) status, ratings = getProductRatings(product_id, headers) return json.dumps(ratings), status, {'Content-Type': 'application/json'} www.163yun.com 

19 .使用Kubernetes编排productpage apiVersion: v1 apiVersion: extensions/v1beta1 kind: Service kind: Deployment metadata: metadata: name: productpage name: productpage-v1 labels: spec: app: productpage replicas: 1 spec: template: ports: metadata: - port: 9080 labels: name: http app: productpage selector: version: v1 app: productpage spec: --- containers: - name: productpage image: istio/examples-bookinfo-productpage- v1:1.5.0 imagePullPolicy: IfNotPresent ports: - containerPort: 9080 --- www.163yun.com 

20 .使用Kubernetes编排reviews apiVersion: v1 kind: Service metadata: name: reviews labels: app: reviews spec: ports: - port: 9080 name: http selector: app: reviews --- apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: Deployment kind: Deployment kind: Deployment metadata: metadata: metadata: name: reviews-v1 name: reviews-v2 name: reviews-v3 spec: spec: spec: replicas: 1 replicas: 1 replicas: 1 template: template: template: metadata: metadata: metadata: labels: labels: labels: app: reviews app: reviews app: reviews version: v1 version: v2 version: v3 spec: spec: spec: containers: containers: containers: - name: reviews - name: reviews - name: reviews image: istio/examples-bookinfo-reviews-v1:1.5.0 image: istio/examples-bookinfo-reviews-v2:1.5.0 image: istio/examples-bookinfo-reviews-v3:1.5.0 imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent ports: ports: ports: - containerPort: 9080 - containerPort: 9080 - containerPort: 9080 --- --- --- www.163yun.com 

21 .嵌入proxy_init作为InitContainer Init Containers: istio-init: Image: docker.io/istio/proxy_init:0.7.1 Port: <none> Host Port: <none> Args: -p 15001 -u 1337 Environment: <none> Mounts: <none> enable-core-dump: Image: alpine Port: <none> Host Port: <none> Command: /bin/sh Args: -c sysctl -w kernel.core_pattern=/etc/istio/proxy/core.%e.%p.%t && ulimit -c unlimited Environment: <none> Mounts: <none> www.163yun.com 

22 .proxy_init设置iptables规则 /usr/local/bin/prepare_proxy.sh -p PORT -u UID iptables -t nat -A ISTIO_OUTPUT -m owner -- 15001 uid-owner ${ENVOY_UID} -j RETURN iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port ${ENVOY_PORT} 9080 iptables -t nat -A PREROUTING -j ISTIO_REDIRECT iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT iptables -t nat -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_REDIRECT www.163yun.com 

23 .嵌入proxy容器作为sidecar istio-proxy: Image: docker.io/istio/proxy_debug:0.7.1 Port: <none> Host Port: <none> Args: proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:8080 --discoveryRefreshDelay www.163yun.com 1s 

24 .SideCar内启动的进程 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND istio-p+ 1 0.0 0.2 32656 19108 ? Ssl 08:55 0:00 /usr/local/bin/pilot-agent proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage --drainDuration 45s -- parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:8080 --discoveryRefreshDelay 1s -- zipkinAddress zipkin.istio-system:9411 --connectTimeout 10s --statsdUdpAddress istio-mixer.istio-system:9125 -- proxyAdminPort 15000 --controlPlaneAuthPolicy NONE istio-p+ 11 3.9 0.5 127352 41068 ? Sl 08:55 13:19 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster productpage --service-node sidecar~192.168.1.114~productpage-v1-8666ffbd7c-mss5p.default~default.svc.cluster.local --max-obj-name-len 189 -l info --v2-config-only www.163yun.com 

25 .Envoy进程的配置 "admin": { "dynamic_resources": { "static_resources": { "access_log_path": "/dev/stdout", "lds_config": { "clusters": [ "address": { "api_config_source": { { "socket_address": { "api_type": "REST_LEGACY", "name": "rds", "address": "127.0.0.1", "refresh_delay": {"seconds": 1, "type": "STRICT_DNS", "port_value": 15000 "nanos": 0}, "connect_timeout": {"seconds": 10, } "cluster_names": [ "nanos": 0}, } "rds" "lb_policy": "ROUND_ROBIN", }, ] } "hosts": [ }, { "cds_config": { "socket_address": {"address": "istio- "api_config_source": { pilot.istio-system", "port_value": 8080} "api_type": "REST_LEGACY", } "refresh_delay": {"seconds": 1, ] "nanos": 0}, "cluster_names": [ }, "rds" ] 管理端口 } } 动态资源 静态资源 www.163yun.com 

26 .Pilot Agent的作用 Wrapper of envoy 当envoy的静态配置改变的时候，对envoy进行热重启后生效(例如TLS) www.163yun.com 

27 . apiVersion: config.istio.io/v1alpha2 Pilot的工作机制 kind: RouteRule metadata: name: reviews-default namespace: default spec: destination: name: reviews precedence: 1 route: - labels: version: v1 weight: 50 - labels: version: v3 weight: 50 www.163yun.com 

28 .在pilot上配置route apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: reviews-default spec: destination: name: reviews precedence: 1 route: - labels: version: v1 www.163yun.com 

29 .查看envoy的管理端口 curl http://127.0.0.1:15000/clusters { istio-proxy@productpage-v1-8666ffbd7c-mss5p:/$ curl http://127.0.0.1:15000/clusters | grep reviews "name": "reviews.default.svc.cluster.local|http", % Total % Received % Xferd Average Speed Time Time Time Current "domains": [ Dload Upload Total Spent Left Speed "reviews:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_connections::1024 "reviews", 100 52770 0 52770 0 0 16.1M 0 --:--:-- --:--:-- --:--:-- 25.1M "reviews.default:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_pending_requests::1024 "reviews.default", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_requests::1024 "reviews.default.svc:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_retries::3 "reviews.default.svc", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_connections::1024 "reviews.default.svc.cluster:9080", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_pending_requests::1024 "reviews.default.svc.cluster", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_requests::1024 "reviews.default.svc.cluster.local:9080", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_retries::3 "reviews.default.svc.cluster.local", out.reviews.default.svc.cluster.local|http|version=v1::added_via_api::true "10.104.14.93:9080", out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_active::0 "10.104.14.93" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_connect_fail::0 ], out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_total::0 "routes": [ out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_active::0 { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_error::0 "match": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_success::0 "prefix": "/" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_timeout::0 }, out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_total::0 "route": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::health_flags::healthy "cluster": "out.reviews.default.svc.cluster.local|http|version=v1", out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::weight::1 "timeout": "0s" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::region:: }, out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::zone:: "decorator": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::sub_zone:: "operation": "reviews-default" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::canary::false } out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::success_rate::-1 } ] } www.163yun.com