Percona XtraDB Cluster的不可变数据库基础结构

在本课程中,我们将讨论如何为数据库构建不变的基础结构。
雅虎!日本是日本最大的门户网站。我们每月有大约9000万个独特的浏览器访问。
我们专注于许多星展银行的高效管理。
如果需要更改某些内容,我们将完全放弃数据库服务器并对其重新映像。
换句话说,“不变的基础设施”是指“一次性的”。
这样,数据库服务器始终保持干净和新鲜。
此外,数据库在重建过程中不会停止,因此它是一种高可用性方法。
PerconaXtradb集群具有实现该机制的重要功能。
会议摘要包括:
-雅虎!日本简介
-什么是“不变的基础设施”
-建筑
-为什么是Percona Xtradb集群
-我们方法的缺点

展开查看详情

1. Immutable Database Infrastructure with PXC Satoshi Mitani | @mita2 Yahoo Japan Corporation

2.Agenda • Yahoo! JAPAN Introduction • Demo • What is Immutable Infrastructure • Architecture • Why Percona XtraDB Cluster • Disadvantages of our method • Q&A 2

3.Yahoo! JAPAN Introduction

4.Yahoo! JAPAN Introduction 4

5.Yahoo! JAPAN Introduction Monthly Page Views Daily Unique Browser 70+ Billion 90+ Million Daily Unique Browser Number of services (Only Smartphone) 100+ 60+ Million

6.Demo

7.Demo • Our steps to release new software 1. Take the node offline 2. Rebuild the node with image including new software 3. Bring the node back online

8.

9.What is Immutable Infrastructure?

10.Legacy infrastructure (Mutable) • Accumulated changes SoftA SoftB SoftA SoftB • Long life-span v.1.0 v.2.0 v.1.0 v.2.0 • Advantage • Existing Infrastructure v.1.1 • Persistent v.2.1 v.2.1 • Disadvantage • Need to track states • Need to upgrade perfectly v.1.2 v.2.2 v.2.2 • Difficult to test all combinations Server A Server B

11.Immutable Infrastructure • Does not change after creation SoftA SoftB • Disposable v.1.0 v.2.0 • Replace servers to release new features • Short life-span SoftA SoftB • Advantage. v.1.1 v.2.1 • Always fresh • Less combinations SoftA SoftB • Disadvantage • Volatile v.1.2 v.2.2

12.Why do we need Immutable Infrastructure? • huge number of DBs • hard to track state • hard to test all combination 12

13.Architecture

14.Architecture overview Chef recipes GitHub Enterprise Image Repo IaaS API Screwdriver.cd Config Backup (CI System) Storage IaaS API my.cnf Golden etc… Image VM Image factory Databases on IaaS

15.Architecture – Image factory Chef • Golden Image recipes • Include all software GitHub Enterprise • PXC • Prometheus Screwdriver.cd • Fluentd (CI System) • etc.. IaaS API Golden Image VM Image factory

16.Architecture – Image factory 1. Update Chef recipe yum_package ['Percona-XtraDB-Cluster-' + pxc_pkg_version, Chef 'Percona-XtraDB-Cluster-shared-' + pxc_pkg_version] do recipes version [pxc_version, pxc_version] action [:install, :lock] GitHub Enterprise options '--enablerepo="percona-release"' end1 Screwdriver.cd cookbook_file (CI System) "/etc/systemd/system/mysql.service.d/override.conf" do source 'etc/systemd/system/mysqld.service.d/override.conf' IaaS API mode 00444 owner 'root' group 'root' Golden end Image VM Image factory

17.Architecture – Image factory Chef • chef-client local mode recipes • No workstation GitHub Enterprise • No server 2. Boot new VM 3. Run chef-client Screwdriver.cd (CI System) IaaS API $ sudo chef-client –z –r “role[some-role]” Golden Image VM Image factory

18. Architecture – Image factory Chef • Snapshot VM recipes GitHub Enterprise 4. Create Snapshot Screwdriver.cd (CI System) IaaS API Golden Image VM Image factory

19.Architecture – Image factory • Tests Chef • Based on new Golden Image recipes • Creating new Database Cluster • Monitoring Process GitHub Enterprise • Load Balancing • etc… Screwdriver.cd (CI System) • Tests are covered by our own python scripts • Fabric IaaS API Golden Image VM Image factory 5. Tests

20. Architecture - Database • Re-imaging clears all data • MySQL configuration • OS configuration • MySQL data Image Repo • etc.. IaaS API Config Backup • MySQL configuration files Storage • Other OS configuration files • network-scripts/if-cfg, /etc/hosts etc.. my.cnf • Generated automatically by IaaS etc… Database

21. Architecture - Database 6. Rebuild • Database consists of 3 nodes • Re-imaging the node one by one Image Repo my.cnf etc… • To avoid downtime IaaS API Config Backup Storage • Pass the backed up config file to rebuild OpenStack IaaS API my.cnf etc… Database

22.Why Percona XtraDB Cluster

23.Our maintenance requirements • No downtime • Anytime, without scheduling

24.Percona XtraDB Cluster (PXC) • MySQL compatible High-availability solution • Multi-writer • Galera replication • Automatic data recovery • State Snapshot Transfer (SST) 24

25.Zero-downtime maintenance • Taking node offine before re-imaging • Wait for all client connections move to others • Possible write across the nodes • PXC supports multi-writer App App App

26.SST - Automatic data recovery • All data cleared by re-imaging • State Snapshot Transfer • full data copy from one node to the joining node Joining node

27.Disadvantages of our method

28.SST Problem (1) 1. SST compatibility issue between 5.7.22 and before 5.7.21 • If you have TDE tables (ENCRYPTION=Y) • Need to upgrade all node before SST 2. SST failed with TDE and Compressed Table • ENCRYPTION=Y, ROW_FORMAT=COMPRESSED • Will be fixed in next Percona XtraBackup release 2.4.15 • https://jira.percona.com/browse/PXB-1867

29.SST Problem (2) 3. SST blocks DDL • Not a bug ! • xtrabackup runs with –lock-ddl for safety • App with frequent DDL faces this problem