展开查看详情
1 . hosted by
HBase Practics at Meituan
Gehua New Century Hotel Beijing，China
August 17,2018
2 . hosted by
Content
01 Multi-Tenancy
02 Object Storage
03 Large Query Isolation
3 . hosted by
01 Multi-Tenancy
RSGroup - Computing Resource Isolation
DNGroup - Storage Isolation
Replication Isolation
4 . hosted by
Multi-Tenancy
RSGroup - Compute Resource Isolation
Group_A
RegionServer
DATA_BLOCK_ENCODING
64 GB BLOOMFILTER
Table_A
RegionServer COMPRESSION
GROUP_NAME : Group_A
64 GB
Group_B
Table_B
RegionServer
128 GB
RegionServer
128 GB
5 . hosted by
Multi-Tenancy
DNGroup - Storage Isolation
Group_A
DataNode path
create time
SSD机型
owner
DataNode …
group_name : Group_A
SSD机型
Group_B
DataNode
SATA机型
DataNode
SATA机型
6 . hosted by
Multi-Tenancy
Replication Isolation
Source Cluster Target Cluster
RegionServer RegionServer
ReplicationSource ReplicationSink
Groups Need
RegionServer RegionServer Replication
ReplicationSource ReplicationSink
Other Groups
RegionServer RegionServer
ReplicationSource ReplicationSink
7 . hosted by
Multi-Tenancy
Constraint PeerId naming specification
• [GROUP]SOURCE_TARGET_INDEX
(1) [GROUP] is a keyword
(2) SOURCE is the group of source cluster
(3) TARGET is the group of target cluster
(4) INDEX used to determine the unique identification
8 . hosted by
Multi-Tenancy
Support Heterogeneous Storage
SSD SATA
SSD SSD SSD
敏感业务
Other Groups
SATA SATA SATA
⼀一般业务
9 . hosted by
Multi-Tenancy
Comparison with multi cluster deployment
Resource can be used flexibly between groups
Storage(DN) Compute(RS)
80% 80%
20% 20% 20% 20%
Machine A Machine B Machine C
Group A Group B
10 . hosted by
Multi-Tenancy
Some BUGFIX About RSGroup
• HBASE-18272 Fix issue about RSGroupBasedLoadBalancer
#roundRobinAssignment where BOGUS_SERVER_NAME is involved
in two groups
• HBASE-20791 RSGroupBasedLoadBalancer#setClusterMetrics should
pass ClusterMetrics to its internalBalancer
11 . hosted by
02 Object Storage
MOB Solution
BUGFIX
12 . hosted by
Object Storage
YARN LOG Storage
• Background
400k applications per day
200 million log files need to upload to HDFS and bring great
pressure to NN
So consider transferring these log files to HBase storage
13 . hosted by
Object Storage
MOB Solution
数据块不参与缓存
有效避免内存碎⽚片问题
Client
内存每次flush会⽣生成2个⽂文件
只有索引⽂文件通过线上来托管
memstore
flush
拆分、整理
等涉及⼤大量IO的操作
IndexFile IndexFile DataFile DataFile
Online Region HDFS DIR
14 . hosted by
Object Storage
Some Problems About MOB
TTL清理存在不同步问题
数据⽂文件清理后索引⽂文件还在
Client
启⽤用MOB特性以后，Region只
能采⽤用默认的整理策略
memstore
flush
拆分、整理
等涉及⼤大量IO的操作
IndexFile IndexFile DataFile DataFile
Online Region HDFS DIR
线下数据采⽤用单⼀一路径进⾏行存储
15 . hosted by
Object Storage
MOB - BUGFIX
Client
将索引⽂文件的清理
放在数据⽂文件清理之前进⾏行
memstore
flush
拆分、整理
等涉及⼤大量IO的操作
IndexFile IndexFile DataFile DataFile
Online Region HDFS DIR
线下⽂文件采⽤用多路径进⾏行存储
16 . hosted by
Object Storage
MOB - BUGFIX
• HBASE-19650 ExpiredMobFileCleaner has wrong logic about TTL check
• HBASE-19664 MOB should compatible with other types of Compactor
in addition to DefaultCompactor
17 . hosted by
03 Large Query Isolation
18 . hosted by
Large Query Isolation
Large Query Concepts
• Characteristic of Large Query
(1) Time delay requirements are not very sensitive
(2) may occupying Handler thread for a long time
• Large Query Types
(1) Scan has no startkey or endkey (sucn as full table scan)
(2) Client call ResultScanner.next() more than a certain threshold
(3) Client call custom coprocessor which involves large queries
19 . hosted by
Large Query Isolation
Large Query Problems - resource may be run up
Client
Handler Large Query
Handler Common Query
Handler Large Query
Call
Call
Call … Large Query
Producer Consumer
20 . hosted by
Large Query Isolation
Implementation Details
Client
Call Call Call
Call Call Call
Call Call Call
Large Query Priority general
Handler … Handler … …
21 . hosted by
Large Query Isolation
Isolation Effect - Limit the situation of resources run up
