当分布式数据库遇到云时 — 汲取经验教训

数据库作为一项服务,是云产业中最有趣且最具挑战性的领域之一。我们正在迁移分布式数据库,如 Greenplum(传统 MPP 数据库)和 Apache HAWQ(Hadoop 原生 SQL 引擎),以实现此目标。 本次演讲的内容是关于我们汲取的经验教训——在此过程中我们发现的最佳做法和陷阱包括: - 如何容器化分布式数据库以及 Kubernetes 能够提供哪些帮助? - 如何将分布式数据库的遗留架构重新设计为云原生架构? - 如何实现智能调度程序的最佳资源利用率? - 如何通过数据局部性分配有状态工作负载并提高性能? - 如何利用 Operator 自动化分布式数据库操作? - 如何将 CNCF 项目与分布式数据库集成以提高其易用性?
展开查看详情

1.When Distributed Database Meets Cloud Lessons Learned Yanqing Weng iweng@pivotal.io © Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

2. ■ Principal Software Engineer in Pivotal ■ Apache HAWQ Committer About me ■ Apache HAWQ PMC Member

3.Agenda ■ Introduction to Distributed Database ■ Distributed Database on Cloud ■ Lessons Learned ■ Q&A Cover w/ Image

4.Distributed Database

5. ■ Apache Hadoop Native SQL, Advanced, MPP, Elastic Query Engine. Apache HAWQ ■ Apache Top Level Project in 2018.8

6.Apache HAWQ Architecture

7.Apache HAWQ Query Processing Slice: 1. a portion of the plan that segments can work on independently. 2. a query plan is sliced wherever a motion operation occurs in the plan. Motion: 1. an operation involves moving tuples between the segments during query processing. 2. three types: redistribution, broadcast, Virtual Segment: gather motion. 1. a resource unit for QD and Resource Manager 2. an execution unit 3. VSEG number determines the degree of parallelism of a query SELECT COUNT(*) FROM lineitem, part WHERE dynamically. p_partkey=l_partkey AND p_brand = 'Brand#23'

8.Virtual Segment ■ Resource allocation unit ■ Query execution unit ■ Variable virtual segment number ■ Place on any physical segment

9. ■ High Performance ■ Storage computing separation Summary ■ Fine-grained resource management ■ Elastic query execution engine ■ Stateless segment

10.Cloud Database

11. ■ Database as Service ■ Efficient Resource Management Requirement ■ Infrastructure Agnostic ■ DBA Free

12. ■ Container VS. Virtual Machine Deployment & ■ Kubernetes Operation ○ Service discovery ○ Load balancing ○ Horizontal and Vertical auto scaling ○ Rolling upgrade ○ Monitor and metrics collection ○ …...

13.Architecture

14. ● Storage Service ○ Cloud Storage, Amazon S3, Hadoop…... ○ Unified Cache Lever by Alluxio Architecture ● Computing Service ○ Shared Segment Pool ○ Global Resource Management Service ● Database Service ○ Master/Standby as Database ○ Get Segments for Query on Demand ● Control Plane ○ Operator/Controllers as DBA

15.Apache HAWQ on Kubernetes

16.Custom Resource

17. ■ HAWQ Operator ■ Resource Pool Controller Controller ■ Resource Pool AutoScaler ■ Resource Recommender ■ Query Controller

18.Apache HAWQ on Kubernetes

19.Lesson Learned

20. ■ Service Oriented Architecture ○ Monolithic → Micro Service Architecture ■ Resource Centric ○ Abstract Component as Resource ○ Service for Resource Usage ○ Controller for Resource Management

21. ■ Container != Image ■ Container != VM Containerization ■ Container = Fine-grained Resource

22. ■ Traditional Database Resource ○ Fixed resources Management ○ Balance resource usage among queries ■ Cloud Database ○ Dynamic resources ○ Maximize resource sharing ○ Maximize resource utilization for each query

23.Resource ● Database Monitoring & ○ Variant Query Workload ○ Data Size Tuning ○ …… ● Query Similarity and Classification ● Query Resource Monitoring ○ Pod Runtime Metrics ○ Application Logs ○ Kubernetes Events ○ …... ● Intelligent Resource Tuning ○ Resource Pool Definition ○ Horizontal & Vertical

24. ■ Log Collection Kubernetes ○ Fluentd Ecosystem ■ Monitoring and Metrics Collection ○ Prometheus ■ Visualization ○ Grafana ■ …...

25. ■ Management Utilities ■ Imperative VS. Declarative Others ■ Pod Priority ■ …...

26.Thank you! Questions?