HBase at DiDi

来自滴滴的工程师唐天航为我们带来了 HBase 在滴滴的业务中的应用场景和经验。
滴滴国内的 HBase 集群有7个,海外国际化集群有4个。覆盖了滴滴全部的业务线,目前服务的项目大概有200多个,数据级是 PB 级。

滴滴使用 HBase 主要有两个场景:

  • 离线数据查询,包括 Phoenix,Kylin,openTSDB 的使用;

  • GeoMesa 系统构建的轨迹数据系统,可用于实时查询、监控和特征工程的数据挖掘。

GeoMesa 系统提供导入和导出接口,导入接口支持 Native API,MR/BulkLoad,StreamingSQL,导出接口支持 SparkCore,SparkSQL,CQL,GeoServer.这样使用 GeoMesa 可以有以下好处:

  • 开箱即用;

  • 类 SQL 文本语言支持;

  • 横向可扩张;

  • 基于 Hadoop 生态。

滴滴在实践中对 zookeeper 的改进为:分离 server 和 client 所依赖 ZK,当client 端的突发大量访问造成 zk 不可用时,不会影响到服务端。(HBASE-20159,ZOOKEEPER-832)。滴滴在 HBase/Phoenix 上的改进,主要是 Quota设置、replication 以及查询优化(HBASE-21964,HBASE-22620,PHOENIX-5242)

最后, 滴滴建立了从 Client 端到 HAProxy,然后到 Thriftserver 和 QueryServer上,之后再到 HBase 的多元用户全链路追踪,能够更加有效提升运维效率。

展开查看详情

1.

2.HBase at DiDi TianHang Tang

3. 1.Usage Scenario On HBase 2.GeoMesa Applications On HBase Content 3.Improvements Of ZK 4.Improvements Of HBase & Phoenix

4.Usage Scenario On HBase

5.

6. Clusters Chinese clusters: 7 Internationalized clusters: 4 01 TPS 1kw+/sec Buiness Line Full coverage 06 02 Overview Projects Amount 200+ 05 03 PB level 04 HBase Tables 1000+

7.

8.

9.Pain points • The feature dimension continues to change • Support feature values of different road link versions • Support for increasing time series features • Supports querying according to KEY index, and supports SCAN according to KEY range • Good scalability and query performance • Feature name, map version, time slice, three dimensions cross HBase • Dynamic columns • Multi-version • Get and Scan • Linear expansion,region balance

10.

11.GeoMesa Applications On HBase

12.GeoMesa GeoMesa Client Client HBase Client GeoMesa is an open source suite of tools that enables large-scale geospatial querying and analytics on distributed computing systems. Create Table/ Insert/ Query/Modify/… • Out of the box GeoMesa • Text language support • Horizontal expansion • Hadoop ecology Server HBase HDFS

13.

14.

15.

16.

17.

18. 2018.04 2018.08 2018.11 2019.03 2019.05 Research Precipitation StreamingSQL access ingest process MR,Spark, First User Python Spark…

19.Improvements Of ZooKeeper

20.

21.HBASE-20159 Support using separate ZK quorums for client. ZOOKEEPER-832 If the hbase client is not restarted, the client will cache the id of the old server zk. This id is larger than the client zk id, resulting in rejection of many attempts when accessing the client zk cluster.

22.

23.Improvements Of HBase & Phoenix

24.1. HBASE-21964 : unset Quota by Throttle Type 2. HBASE-22620 : when a cluster open replication,regionserver will not clean up the walLog references on zk due to no wal entry need to be replicated 3. PHOENIX-5242 : concurrent tasks 14000 to 32

25.Quota experience 1. NUMBER type Semantic : RPC 2. ThrottlingException -> DoNotRetryIOException 3. HBASE-21592 : quota.addGetResult(r) throws NPE 4. HBASE-21964

26.Full Link Monitor

27.Thanks! tangtianhang@didiglobal.com