Pinterest的HBase升级实践

Pinterest的HBase升级实践
展开查看详情

1.

2.Removable Singularity A Story of HBase Upgrade at Pinterest Storage & Caching, Engineering @ Pinterest Tian-Ying Chang

3.Agenda 1 Usage of HBase in Pinterest 2 Upgrading Situation and Challenges 3 Migration Steps 4 Performance Tuning 5 Final Notes

4.HBase @ Pinterest §Early applications: homefeed, search & discovery §UserMetaStore: a generic store layer for all applications using user KV data §Zen: a graph service layer • Many applications can be modeled as graph • Usage of HBase flourished after Zen release §Other misc. use cases, e.g., analytics, OpenTSDB

5.40+ HBase clusters on 0.94

6.Need Upgrading §HBase 0.94 is not supported by community anymore §Newer version has better reliability, availability and performance §Easier to contribute back to community

7.Singularity of HBase 0.96 §RPC protocol changes §Data format changes § HDFS folder structure changes §API changes §Generally considered “impossible” to live upgrade without downtime http://blog.cloudera.com/blog/2012/06/the-singularity-hbase-compatibility-and-extensibility/

8.The Dilemma §Singularity: cannot live upgrade from HBase 0.94 to later version • Need downtime §Pinterest hosts critical online real time services on HBase • Cannot afford downtime §Stuck?

9.Fast Forward §Successfully upgraded production clusters last year • Chose one of the most loaded clusters as pilot • No application redeploy needed on the day of switching to 1.2 cluster • Live switch with no interrupt to Pinterest site §Big performance improvement

10. P99 Latency of Different APIs Point of Live Switch

11.How did we do it?

12. Client ZK read/write HBase 0.94

13. Client ZK read/write native HBase 0.94 HBase 0.94 replication

14. Client ZK read/write native HBase 0.94 HBase 0.94 replication

15. Client ZK read/write HBase 0.94

16. Client ZK read/write 1. Build empty cluster HBase 0.94 HBase 0.94

17. Client ZK read/write 1. Build empty cluster native HBase 0.94 HBase 0.94 replication 2. Setup replication

18. Client ZK read/write 1. Build empty cluster 3. Export snapshot native HBase 0.94 HBase 0.94 replication 2. Setup replication

19. Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot native HBase 0.94 HBase 0.94 replication 2. Setup replication

20. Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot native HBase 0.94 HBase 0.94 replication 5. Replication drain 2. Setup replication

21. Client ZK read/write native HBase 0.94 HBase 0.94 replication

22. Client ZK read/write HBase 0.94 replication HBase 1.2

23. Client ZK read/write HBase 0.94 replication HBase 1.2

24. Client ZK read/write HBase 0.94 HBase 0.94

25. Client ZK read/write 1. Build empty cluster HBase 0.94 HBase 0.94 HBase 1.2

26. Client ZK read/write 1. Build empty cluster non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication

27. Client ZK read/write 1. Build empty cluster 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication

28. Client ZK read/write 1. Build empty cluster 4. Recover 1.2 table from 0.94 snapshot 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 2. Setup replication

29. Client ZK read/write 1. Build empty cluster 4. Recover table from snapshot 3. Export snapshot non-native HBase 0.94 HBase 0.94 HBase 1.2 replication 5. Replication drain 2. Setup replication

为了让众多HBase相关从业人员及爱好者有一个自由交流HBase相关技术的社区,阿里巴巴、小米、华为、网易、京东、滴滴、知乎等公司的HBase技术研究人员共同发起了组建中国HBase技术社区。