Percona XtraDB Cluster的不可变数据库基础结构

1 . Immutable Database Infrastructure with PXC Satoshi Mitani | @mita2 Yahoo Japan Corporation 

2 .Agenda • Yahoo! JAPAN Introduction • Demo • What is Immutable Infrastructure • Architecture • Why Percona XtraDB Cluster • Disadvantages of our method • Q&A 2 

3 .Yahoo! JAPAN Introduction 

4 .Yahoo! JAPAN Introduction 4 

5 .Yahoo! JAPAN Introduction Monthly Page Views Daily Unique Browser 70+ Billion 90+ Million Daily Unique Browser Number of services (Only Smartphone) 100+ 60+ Million 

6 .Demo 

7 .Demo • Our steps to release new software 1. Take the node offline 2. Rebuild the node with image including new software 3. Bring the node back online 

8 .

9 .What is Immutable Infrastructure? 

10 .Legacy infrastructure (Mutable) • Accumulated changes SoftA SoftB SoftA SoftB • Long life-span v.1.0 v.2.0 v.1.0 v.2.0 • Advantage • Existing Infrastructure v.1.1 • Persistent v.2.1 v.2.1 • Disadvantage • Need to track states • Need to upgrade perfectly v.1.2 v.2.2 v.2.2 • Difficult to test all combinations Server A Server B 

11 .Immutable Infrastructure • Does not change after creation SoftA SoftB • Disposable v.1.0 v.2.0 • Replace servers to release new features • Short life-span SoftA SoftB • Advantage. v.1.1 v.2.1 • Always fresh • Less combinations SoftA SoftB • Disadvantage • Volatile v.1.2 v.2.2 

12 .Why do we need Immutable Infrastructure? • huge number of DBs • hard to track state • hard to test all combination 12 

13 .Architecture 

14 .Architecture overview Chef recipes GitHub Enterprise Image Repo IaaS API Screwdriver.cd Config Backup (CI System) Storage IaaS API my.cnf Golden etc… Image VM Image factory Databases on IaaS 

15 .Architecture – Image factory Chef • Golden Image recipes • Include all software GitHub Enterprise • PXC • Prometheus Screwdriver.cd • Fluentd (CI System) • etc.. IaaS API Golden Image VM Image factory 

16 .Architecture – Image factory 1. Update Chef recipe yum_package ['Percona-XtraDB-Cluster-' + pxc_pkg_version, Chef 'Percona-XtraDB-Cluster-shared-' + pxc_pkg_version] do recipes version [pxc_version, pxc_version] action [:install, :lock] GitHub Enterprise options '--enablerepo="percona-release"' end1 Screwdriver.cd cookbook_file (CI System) "/etc/systemd/system/mysql.service.d/override.conf" do source 'etc/systemd/system/mysqld.service.d/override.conf' IaaS API mode 00444 owner 'root' group 'root' Golden end Image VM Image factory 

17 .Architecture – Image factory Chef • chef-client local mode recipes • No workstation GitHub Enterprise • No server 2. Boot new VM 3. Run chef-client Screwdriver.cd (CI System) IaaS API $ sudo chef-client –z –r “role[some-role]” Golden Image VM Image factory 

18 . Architecture – Image factory Chef • Snapshot VM recipes GitHub Enterprise 4. Create Snapshot Screwdriver.cd (CI System) IaaS API Golden Image VM Image factory 

19 .Architecture – Image factory • Tests Chef • Based on new Golden Image recipes • Creating new Database Cluster • Monitoring Process GitHub Enterprise • Load Balancing • etc… Screwdriver.cd (CI System) • Tests are covered by our own python scripts • Fabric IaaS API Golden Image VM Image factory 5. Tests 

20 . Architecture - Database • Re-imaging clears all data • MySQL configuration • OS configuration • MySQL data Image Repo • etc.. IaaS API Config Backup • MySQL configuration files Storage • Other OS configuration files • network-scripts/if-cfg, /etc/hosts etc.. my.cnf • Generated automatically by IaaS etc… Database 

21 . Architecture - Database 6. Rebuild • Database consists of 3 nodes • Re-imaging the node one by one Image Repo my.cnf etc… • To avoid downtime IaaS API Config Backup Storage • Pass the backed up config file to rebuild OpenStack IaaS API my.cnf etc… Database 

22 .Why Percona XtraDB Cluster 

23 .Our maintenance requirements • No downtime • Anytime, without scheduling 

24 .Percona XtraDB Cluster (PXC) • MySQL compatible High-availability solution • Multi-writer • Galera replication • Automatic data recovery • State Snapshot Transfer (SST) 24 

25 .Zero-downtime maintenance • Taking node offine before re-imaging • Wait for all client connections move to others • Possible write across the nodes • PXC supports multi-writer App App App 

26 .SST - Automatic data recovery • All data cleared by re-imaging • State Snapshot Transfer • full data copy from one node to the joining node Joining node 

27 .Disadvantages of our method 

28 .SST Problem (1) 1. SST compatibility issue between 5.7.22 and before 5.7.21 • If you have TDE tables (ENCRYPTION=Y) • Need to upgrade all node before SST 2. SST failed with TDE and Compressed Table • ENCRYPTION=Y, ROW_FORMAT=COMPRESSED • Will be fixed in next Percona XtraBackup release 2.4.15 • https://jira.percona.com/browse/PXB-1867 

29 .SST Problem (2) 3. SST blocks DDL • Not a bug ! • xtrabackup runs with –lock-ddl for safety • App with frequent DDL faces this problem