MySQL compatibility for HBase. 大量的用户仍然习惯使用Mysql这样的传统数据库,而AntsDB是一个mysql的底层存储可以构建在HBase之上,实现更大的容量和可靠性,作为尝鲜者,AntsDB项目的代码维护者介绍了它的内部实现和架构,特别是可以和现有的大数据处理引擎比如Spark SQL集成。

注脚

展开查看详情

1.hosted by AntsDB MySQL Compatibility for HBase August 17,2018

2. hosted by Content About Us 01 02 Scenario Architecture 03 Complementary to 04 HBase

3. hosted by 01 About Us

4. hosted by Water Guo  Founder of AntsDB  Love Hadoop and HBase but a heavy RDB user in the past  Love programming for extreme performance  Previously Software Architecture at Informatica, DataMirror, founder of BDI Systems.

5. hosted by AntsDB  Open source project at https://github.com/waterguo/antsdb  Problem to resolve: NoSQL is taking over the world but majority of the data applications are still using RDB  Database virtualization software  Attempt to bring MySQL compatibility to HBase

6. hosted by 02 Scenario

7. hosted by Virtual Telephone Number  Telecom industry  Rigid latency requirement: 20ms max  Data is growing fast  Built on top of relational database - MySQL

8. hosted by Solution Architecture User Batch Application Analytics Spark SQL AntsDB HBase

9. hosted by Lessons Learned What we like HBase What we were struggling  Linear scalability  Latency spike  Hadoop ecosystem  Transaction  Good random access  Join performance performance

10. hosted by Why don’t cache hot data  Operational data = hot data  Historical data = majority of big data  Daily hot data = 1-5%  Weekly hot data = 5-10%  Monthly hot data = 10-15%

11. hosted by Silver bullet - caching ✧ Large amount local caching using SSD – 500 GB ✧ Local data fetch -> sub-millisecond latency ✧ Distributed transaction -> local transaction ✧ Distributed joins -> local joins

12. hosted by 03 AntsDB Architecture

13. hosted by Layers MySQL Network JDBC, ODBC, PHP, Python, Native 1 SQL Parser/Executor 2 MySQL syntax, functions, data types Transaction MVCC, row locking 3 4 Caching HBase Connectivity Data in and out from HBase

14. hosted by Architecture Subtitle Text AntsDB SQL Parser Application MySQL Driver Query Executor Query Result Transaction Manager Local Transactions Cache HBase HBase Connector Query Cluster Result

15. hosted by Fight the GC - Java Unsafe - skip list, cache, WAL, row locking, cursor - Atomic operation for row locks - Small heap size - less than 4G - Escape analysis - Scan performance : 1 million rows / thread / second - Insert performance: 200K inserts / thread / second

16. hosted by Performance - YCSB write

17. hosted by Performance - TPCC

18. hosted by 04 Complementary to HBase

19. hosted by MySQL Compatibility  MySQL protocol – JDBC, ODBC, PHP, Python, Native  Core SQL syntax  MySQL data types  MySQL functions – 45  Low latency transactions  Low latency joins  Indexes - Unique, non-unique, full-text

20. hosted by MySQL Behaviour Simulation - Locking - Commit and rollback - MVCC - Quirks such as 0000-00-00 00:00:00 datetime, zerofill integers - Millisecond latency

21. hosted by Compatible applications  MySQL command line utilities  MySQL Workebench  DBeaver  SquirrelSQL  MediaWiki  SonarQube  PHPMyAdmin  BechmarkSQL – TPC-C benchmarking  YCSB

22. hosted by Next step High availability Cluster Low latency CP system

23.hosted by Thanks

user picture
为了让众多HBase相关从业人员及爱好者有一个自由交流HBase相关技术的社区,阿里巴巴、小米、华为、网易、京东、滴滴、知乎等公司的HBase技术研究人员共同发起了组建中国HBase技术社区。

相关文档