网易云 刘超 - 《基于Service+Mesh的海量容器管理平台实践》

网易云对内对外支撑的重点在于基于Kubernetes的容器平台,已经在线生产管理运行超过1000天,集群规模最大达30000台,支撑了很多内部重点项目,减轻了业务方瞬时流量增加所带来的运维成本,比较早期的业务平台的微服务化采取的是Dubbo,后来新的业务使用SpringCloud,云平台组基于Kubernetes推出了Service Mesh服务治理平台,将微服务部分工作下沉到平台层完成,并在计费中心,云管平台,互金项目中进行落地,本次分享在大规模Kubernetes容器平台中落地Service Mesh的实践。
展开查看详情

1.基于Service Mesh的 海量容器管理平台实践

2.关于我 • 刘超 • 网易云 解决方案总架构师 • 10余年云计算领域研发及架构经验,先后在EMC,CCTV证券资 讯频道,HP,华为,网易从事云计算和大数据架构工作 • 毕业于上海交通大学。 • 曾出版《Lucene应用开发揭秘》 • 多次作为邀请讲师参加Dockone容器技术大会,Segmentfault 开发者大会,InfoQ全球架构师峰会(明星讲师),CSDN SDCC大 会,51CTO WOTA大会等 • 知名技术博主,博客可搜索popsuper1982,多篇文章推荐至全 球最大IT社区CSDN首页及《程序员》杂志 • 在工作中积累了大量运营商系统,互联网金融系统,电商系统等 容器化和微服务化经验 www.163yun.com

3.目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com

4.企业上云三大架构 快速迭代 高并发 开发 应用架构 CAPEX 大数据分析 运维 IT架构 数据架构 运营 OPEX 数字化运营 www.163yun.com

5.容器技术的三种视角 微服务的交付形式Kubernetes 应用架构 IT架构 数据架构 轻量级的IT运维模式Swarm 资源利用率高的任务执行模式Mesos www.163yun.com

6.Kubernetes + Docker 是 Dev 和 Ops 融合的一个桥梁。 开发,构建,测试 服务发现 配置中心 Dev 熔断降级 Kubernetes + Docker Dockerfile Ops 镜像环境交付 提供资源,部署,运维 www.163yun.com

7.Kubernetes更加适合微服务和DevOps的设计 微服务设计 Kubernetes功能 设计要点一:API 网关 Ingress 设计要点二:无状态化,区分有状态的和无状态的应用。 无状态对应Deployment,有状态对应 StatefulSet 设计要点三:数据库的横向扩展。 headless service指向PaaS服务,或者StatefulSet部署 设计要点四:缓存 headless service指向PaaS服务,或者StatefulSet部署 设计要点五:服务拆分和服务发现 Service 设计要点六:服务编排与弹性伸缩 Deployment的Replicas 设计要点七:统一配置中心 ConfigMap 设计要点八:统一的日志中心 DaemonSet部署日志Agent 设计要点九:熔断,限流,降级 Service Mesh 设计要点十:全方位的监控 Cadvisor,DaemonSet部署监控Agent www.163yun.com

8.目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com

9.Service Mesh的范例Istio www.163yun.com

10.一切从envoy开始 • 轻量级的proxy • 静态配置,热加载,热重启 • 动态配置,拉取模式 Listener LDS Routes RDS Clusters CDS Endpoints EDS www.163yun.com

11.Envoy之静态配置 admin: access_log_path: /tmp/admin_access.log address: socket_address: { address: 127.0.0.1, port_value: 9901 } static_resources: listeners: - name: listener_0 address: Listener socket_address: { address: 127.0.0.1, port_value: 10000 } filter_chains: - filters: - name: envoy.http_connection_manager config: Routes stat_prefix: ingress_http codec_type: AUTO route_config: name: local_route virtual_hosts: - name: local_service Clusters domains: ["*"] routes: Endpoints - match: { prefix: "/" } route: { cluster: some_service } http_filters: - name: envoy.router clusters: - name: some_service connect_timeout: 0.25s type: STATIC lb_policy: ROUND_ROBIN hosts: [{ socket_address: { address: 127.0.0.2, port_value: 1234 }}] www.163yun.com

12.Envoy之动态配置 admin: access_log_path: /tmp/admin_access.log address: socket_address: { address: 127.0.0.1, port_value: 9901 } dynamic_resources: lds_config: api_config_source: api_type: GRPC cluster_names: [xds_cluster] cds_config: api_config_source: api_type: GRPC cluster_names: [xds_cluster] static_resources: clusters: - name: xds_cluster connect_timeout: 0.25s type: STATIC lb_policy: ROUND_ROBIN http2_protocol_options: {} hosts: [{ socket_address: { address: 127.0.0.3, port_value: 5678 }}] www.163yun.com

13.加入控制面Pilot www.163yun.com

14.Pilot路由策略 www.163yun.com

15.Pilot的负载均衡 www.163yun.com

16.目录 从Docker到K8s到Service Mesh Service Mesh 介绍 深入解析 Service Mesh 网易云实践 www.163yun.com

17.这个简单例子背后发生的事情 www.163yun.com

18.Productpage是一个python程序 @app.route('/api/v1/products') def productsRoute(): return json.dumps(getProducts()), 200, {'Content-Type': 'application/json'} details = { @app.route('/api/v1/products/<product_id>') "name" : "http://details:9080", def productRoute(product_id): "endpoint" : "details", headers = getForwardHeaders(request) "children" : [] status, details = getProductDetails(product_id, headers) } return json.dumps(details), status, {'Content-Type': 'application/json'} ratings = { "name" : "http://ratings:9080", @app.route('/api/v1/products/<product_id>/reviews') "endpoint" : "ratings", def reviewsRoute(product_id): "children" : [] headers = getForwardHeaders(request) } status, reviews = getProductReviews(product_id, headers) return json.dumps(reviews), status, {'Content-Type': 'application/json'} reviews = { "name" : "http://reviews:9080", "endpoint" : "reviews", @app.route('/api/v1/products/<product_id>/ratings') "children" : [ratings] def ratingsRoute(product_id): } headers = getForwardHeaders(request) status, ratings = getProductRatings(product_id, headers) return json.dumps(ratings), status, {'Content-Type': 'application/json'} www.163yun.com

19.使用Kubernetes编排productpage apiVersion: v1 apiVersion: extensions/v1beta1 kind: Service kind: Deployment metadata: metadata: name: productpage name: productpage-v1 labels: spec: app: productpage replicas: 1 spec: template: ports: metadata: - port: 9080 labels: name: http app: productpage selector: version: v1 app: productpage spec: --- containers: - name: productpage image: istio/examples-bookinfo-productpage- v1:1.5.0 imagePullPolicy: IfNotPresent ports: - containerPort: 9080 --- www.163yun.com

20.使用Kubernetes编排reviews apiVersion: v1 kind: Service metadata: name: reviews labels: app: reviews spec: ports: - port: 9080 name: http selector: app: reviews --- apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1 kind: Deployment kind: Deployment kind: Deployment metadata: metadata: metadata: name: reviews-v1 name: reviews-v2 name: reviews-v3 spec: spec: spec: replicas: 1 replicas: 1 replicas: 1 template: template: template: metadata: metadata: metadata: labels: labels: labels: app: reviews app: reviews app: reviews version: v1 version: v2 version: v3 spec: spec: spec: containers: containers: containers: - name: reviews - name: reviews - name: reviews image: istio/examples-bookinfo-reviews-v1:1.5.0 image: istio/examples-bookinfo-reviews-v2:1.5.0 image: istio/examples-bookinfo-reviews-v3:1.5.0 imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent ports: ports: ports: - containerPort: 9080 - containerPort: 9080 - containerPort: 9080 --- --- --- www.163yun.com

21.嵌入proxy_init作为InitContainer Init Containers: istio-init: Image: docker.io/istio/proxy_init:0.7.1 Port: <none> Host Port: <none> Args: -p 15001 -u 1337 Environment: <none> Mounts: <none> enable-core-dump: Image: alpine Port: <none> Host Port: <none> Command: /bin/sh Args: -c sysctl -w kernel.core_pattern=/etc/istio/proxy/core.%e.%p.%t && ulimit -c unlimited Environment: <none> Mounts: <none> www.163yun.com

22.proxy_init设置iptables规则 /usr/local/bin/prepare_proxy.sh -p PORT -u UID iptables -t nat -A ISTIO_OUTPUT -m owner -- 15001 uid-owner ${ENVOY_UID} -j RETURN iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port ${ENVOY_PORT} 9080 iptables -t nat -A PREROUTING -j ISTIO_REDIRECT iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT iptables -t nat -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_REDIRECT www.163yun.com

23.嵌入proxy容器作为sidecar istio-proxy: Image: docker.io/istio/proxy_debug:0.7.1 Port: <none> Host Port: <none> Args: proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:8080 --discoveryRefreshDelay www.163yun.com 1s

24.SideCar内启动的进程 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND istio-p+ 1 0.0 0.2 32656 19108 ? Ssl 08:55 0:00 /usr/local/bin/pilot-agent proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage --drainDuration 45s -- parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:8080 --discoveryRefreshDelay 1s -- zipkinAddress zipkin.istio-system:9411 --connectTimeout 10s --statsdUdpAddress istio-mixer.istio-system:9125 -- proxyAdminPort 15000 --controlPlaneAuthPolicy NONE istio-p+ 11 3.9 0.5 127352 41068 ? Sl 08:55 13:19 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster productpage --service-node sidecar~192.168.1.114~productpage-v1-8666ffbd7c-mss5p.default~default.svc.cluster.local --max-obj-name-len 189 -l info --v2-config-only www.163yun.com

25.Envoy进程的配置 "admin": { "dynamic_resources": { "static_resources": { "access_log_path": "/dev/stdout", "lds_config": { "clusters": [ "address": { "api_config_source": { { "socket_address": { "api_type": "REST_LEGACY", "name": "rds", "address": "127.0.0.1", "refresh_delay": {"seconds": 1, "type": "STRICT_DNS", "port_value": 15000 "nanos": 0}, "connect_timeout": {"seconds": 10, } "cluster_names": [ "nanos": 0}, } "rds" "lb_policy": "ROUND_ROBIN", }, ] } "hosts": [ }, { "cds_config": { "socket_address": {"address": "istio- "api_config_source": { pilot.istio-system", "port_value": 8080} "api_type": "REST_LEGACY", } "refresh_delay": {"seconds": 1, ] "nanos": 0}, "cluster_names": [ }, "rds" ] 管理端口 } } 动态资源 静态资源 www.163yun.com

26.Pilot Agent的作用 Wrapper of envoy 当envoy的静态配置改变的时候,对envoy进行热重启后生效(例如TLS) www.163yun.com

27. apiVersion: config.istio.io/v1alpha2 Pilot的工作机制 kind: RouteRule metadata: name: reviews-default namespace: default spec: destination: name: reviews precedence: 1 route: - labels: version: v1 weight: 50 - labels: version: v3 weight: 50 www.163yun.com

28.在pilot上配置route apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: reviews-default spec: destination: name: reviews precedence: 1 route: - labels: version: v1 www.163yun.com

29.查看envoy的管理端口 curl http://127.0.0.1:15000/clusters { istio-proxy@productpage-v1-8666ffbd7c-mss5p:/$ curl http://127.0.0.1:15000/clusters | grep reviews "name": "reviews.default.svc.cluster.local|http", % Total % Received % Xferd Average Speed Time Time Time Current "domains": [ Dload Upload Total Spent Left Speed "reviews:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_connections::1024 "reviews", 100 52770 0 52770 0 0 16.1M 0 --:--:-- --:--:-- --:--:-- 25.1M "reviews.default:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_pending_requests::1024 "reviews.default", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_requests::1024 "reviews.default.svc:9080", out.reviews.default.svc.cluster.local|http|version=v1::default_priority::max_retries::3 "reviews.default.svc", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_connections::1024 "reviews.default.svc.cluster:9080", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_pending_requests::1024 "reviews.default.svc.cluster", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_requests::1024 "reviews.default.svc.cluster.local:9080", out.reviews.default.svc.cluster.local|http|version=v1::high_priority::max_retries::3 "reviews.default.svc.cluster.local", out.reviews.default.svc.cluster.local|http|version=v1::added_via_api::true "10.104.14.93:9080", out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_active::0 "10.104.14.93" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_connect_fail::0 ], out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::cx_total::0 "routes": [ out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_active::0 { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_error::0 "match": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_success::0 "prefix": "/" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_timeout::0 }, out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::rq_total::0 "route": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::health_flags::healthy "cluster": "out.reviews.default.svc.cluster.local|http|version=v1", out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::weight::1 "timeout": "0s" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::region:: }, out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::zone:: "decorator": { out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::sub_zone:: "operation": "reviews-default" out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::canary::false } out.reviews.default.svc.cluster.local|http|version=v1::192.168.2.23:9080::success_rate::-1 } ] } www.163yun.com