Pinterest的HBase升级实践

1 .

2 .Removable Singularity A Story of HBase Upgrade at Pinterest Storage & Caching, Engineering @ Pinterest Tian-Ying Chang 

3 .Agenda 1 Usage of HBase in Pinterest 2 Upgrading Situation and Challenges 3 Migration Steps 4 Performance Tuning 5 Final Notes 

4 .HBase @ Pinterest §Early applications: homefeed, search & discovery §UserMetaStore: a generic store layer for all applications using user KV data §Zen: a graph service layer • Many applications can be modeled as graph • Usage of HBase flourished after Zen release §Other misc. use cases, e.g., analytics, OpenTSDB 

5 .40+ HBase clusters on 0.94 

6 .Need Upgrading §HBase 0.94 is not supported by community anymore §Newer version has better reliability, availability and performance §Easier to contribute back to community 

7 .Singularity of HBase 0.96 §RPC protocol changes §Data format changes § HDFS folder structure changes §API changes §Generally considered “impossible” to live upgrade without downtime http://blog.cloudera.com/blog/2012/06/the-singularity-hbase-compatibility-and-extensibility/ 

8 .The Dilemma §Singularity: cannot live upgrade from HBase 0.94 to later version • Need downtime §Pinterest hosts critical online real time services on HBase • Cannot afford downtime §Stuck? 

9 .Fast Forward §Successfully upgraded production clusters last year • Chose one of the most loaded clusters as pilot • No application redeploy needed on the day of switching to 1.2 cluster • Live switch with no interrupt to Pinterest site §Big performance improvement 

10 . P99 Latency of Different APIs Point of Live Switch 

11 .How did we do it? 

12 . Client ZK read/write HBase 0.94 

13 . Client ZK read/write native HBase 0.94 HBase 0.94 replication 

14 . Client ZK read/write native HBase 0.94 HBase 0.94 replication 

15 . Client ZK read/write HBase 0.94 

16 . Client ZK read/write 1. Build empty cluster HBase 0.94 HBase 0.94 

17 . Client ZK read/write 1. Build empty cluster native HBase 0.94 HBase 0.94 replication 2. Setup replication 

18 . Client ZK read/write 1. Build empty cluster 3. Export snapshot native HBase 0.94 HBase 0.94 replication 2. Setup replication 

19 . Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot native HBase 0.94 HBase 0.94 replication 2. Setup replication 

20 . Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot native HBase 0.94 HBase 0.94 replication 5. Replication drain 2. Setup replication 

21 . Client ZK read/write native HBase 0.94 HBase 0.94 replication 

22 . Client ZK read/write HBase 0.94 replication HBase 1.2 

23 . Client ZK read/write HBase 0.94 replication HBase 1.2 

24 . Client ZK read/write HBase 0.94 HBase 0.94 

25 . Client ZK read/write 1. Build empty cluster HBase 0.94 HBase 0.94 HBase 1.2 

26 . Client ZK read/write 1. Build empty cluster non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication 

27 . Client ZK read/write 1. Build empty cluster 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication 

28 . Client ZK read/write 1. Build empty cluster 4. Recover 1.2 table from 0.94 snapshot 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication 

29 . Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 5. Replication drain 2. Setup replication