确定删除吗？
来自腾讯的工程师程广旭为我们带来了 HBase 在腾讯的业务中的应用场景和经验。
腾讯目前有90多个 HBase 集群，最大的集群有500多台节点。腾讯内部多个业务包括腾讯视频，微信支付和腾讯云等都在使用 HBase 服务。其首先分享了使用 HBase 进行数据迁移的经验：Replication 和 ExportSnapshot.在实际使用中，业务每天的数据量都很大，这些数据需要保存的周期要么很大，要么很小。因此采取了按天分表的方式，也就是每天会创建一个新的表，对于过期的数据，直接把当天的表删掉即可。其次分享了对带宽的优化。
写入 HBase 的流量主要有五个部分:
写入
WAL
Flush
Small Compaction
Major Compaction
优化的方法有：
开启 CellBlock 的压缩。
WAL 的压缩。
增大 memstore,减少 Flush,减少 Compaction.
减少 Compaction 的线程数目。
关闭 Major Compaction.
按天建表。
最后介绍了如何共享 RestServer.当每个 HBase 集群搭建一个 RestServer 时，如果读取集群的请求很少，那么集群的 RestServer 资源比较浪费。腾讯做了一个改进，配置一个 RestServer 可以访问多个 HBase 集群，同时在 MySQL 里记了哪些表可以通过这种方式访问。
1.
2.HBase At Tencent Andrew Cheng | 程广旭 Tencent | HBase Committer
3.Content 01. HBase Service In Tencent 02. Applications 03. Practices & Optimization
4.01. HBase Service In Tencent
5.HBase Story in Tencent l Began using since 2013 l Used version l 0.94.17 -> 0.98.6 -> 1.2.5 -> 2.2.0 (ing) l Largest cluster more than 500 nodes 90+ 4000+ 10PB+ 3Tri+ Clusters Nodes Data RPD
6.Overview HBase Users come from 6 groups , more than 100+ different applications
7.Architecture Phoenix Advertiseme Deploy Center Tenpay Wepay Game … nt OpenTSDB Doss TDBank Spark Tookit RestServer S2Graph Lhotse HBase Api ThriftServer monitoring Kylin Tencent HBase Zookeeper TNM2
8.02. Applications
9.Tencent Ads – Real-Time Logjoin System Data Source Mixer Exposure Click … Transport TDBank Logical LogJoin LogJoin LogJoin LogJoin Association Table Storage Tencent HBase Flow Table Consumer Model learning Freshness Budget control Report
10.Tenpay - Transaction record Data Source MySQL Application Application Binlog Paser DBSync C++ JAVA Read Cache Hippo Read Thrift Server Write TDSort Read Write Storage Tencent HBase
11.03. Practices & Optimization
12.Practices–Data migration Business-insensitive data migration Cluster A Cluster B add_peer delete_snapshot Client switch to new cluster disable_peer enable_peer Check Data Set REPLICATION_SCOPE => '1' Set REPLICATION_SCOPE => '0' snapshot clone_snapshot ExportSnapshot
13.Practices–Table l Create table per day l Large amount of data l TTL is short l Benefit l Reduce the amount of data in compaction l Easy to delete expired data
14.Optimization - Bandwidth Input Data Input Data Input Data ① Input Data ② RS2 and RS3 Wal data ① ② ③ RS2 and RS3 Flush data Wal Flush ③ Small compact ④ ④ RS2 and RS3 Small compact RS1 RS2 RS3 Large compact ⑤ ⑤ RS2 and RS3 Large compact
15.Optimization - Bandwidth l Enable compressing of CellBlocks l Wal compressor l Increase the size of memstore l Reduce the number of threads about compaction l Turn off major compaction l create tables by day
16.Optimization - Online filtering of dirty data l A large amount of data which have the same Rowkey l How to find filter rowkeys? Input Data l ResponseTooSlow Yes l How to set filter rowkeys? Filter Yes l hbase.hregion.filter.rowkeys Filter Enable l How to refresh filter rowkeys? No l update_config No Write
17.Optimization - Prefix Bloom Filter（HBASE-20636） l ROWPREFIX_FIXED_LENGTH Create Table: l ROWPREFIX_DELIMITER uin ts action Prefix File info: Bloom Filter
18.Optimization - Prefix Bloom Filter（HBASE-20636） Write Read Input Data Get prefix key by Scan prefix_length Rowkey Get prefix key by prefix_length Computer hash {StartKey,EndKey} value Computer hash value No Same No prefix? Hit Filter StoreFile BloomFilter? Set BloomFilter Yes Prefix length Yes Yes No >= Last line? prefix_length Yes No Not Filter StoreFile Write BloomFilter information to StoreFile metadata
19.Optimization - RestServer User Nginx RestServer A RestServer B RestServer C RestServer D Cluster A Cluster B Cluster C
20.Optimization - RestServer User Nginx Mysql RestServer A RestServer B RestServer C Cluster A Cluster B Cluster C
21.Optimization - RestServer l Only maintain one configuration l use effectively resources l User-friendly access
22.HBase Community l 1 Committer, 2 Contributor l Total commits: 80+ l Feature l HBASE-20636 Introduce two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED l HBASE-19799 Add web UI to rsgroup l HBASE-20243 [Shell] Add shell command to create a new table by cloning the existent table l HBASE-19483 Add proper privilege check for rsgroup commands l ………
23.Join Us Dept. Wechat Personal Wechat
24.Thanks！
