展开查看详情
1 . hosted by
Hbase Practice At ke.com
Guoxian Zhao
2 . hosted by
Content
01 Architecture Evolution
02 Olap Scenario
03 Monitor Scenario
04 Data Mining Scenario
05 Reatime ETL Scenario
06 Hdic Scenario
3 . hosted by
01 Architecture Evolution
4 . hosted by
bigdata architecture for ke
Evolution
5 . hosted by
02 Olap Scenario
6 . hosted by
Olap Scenario
Requirement List
Billion data set High concurrency
subsecond response times sql interface
7 . hosted by
Olap Scenario
kylin
8 . hosted by
Olap Scenario
Olap data architecture map
product
Cube
Metrics api redis
query execute
redis
permission
adhoc Meta
Statistics
manager
queryengine
Kylin middleware
monitor
Sparksql
presto zk nginx
Offline
Spark cluster kylin
meta
build build query query
Alluxio
hadoop, warehouse + compute Hbase
scheduler
load
…
…
9 . hosted by
Data For Kylin
800+cube、16+business
200T storage、160000 million data、1 cube 6000 million
Query/day million， <500ms(95%), <1s (99%)
10 . hosted by
Data Product
Metrics platform
• roll-up drill
• multidimensional analysis
• self-service configuration
report
• standard indices
11 . hosted by
Performance Optimizations For Hbase
SSD-FIRST
Clien
t
OR
H
D
S
S
D
H
D
D
• Read remote ssds first
D
DN1 DN2 DN3
12 . hosted by
Performance Optimizations For Hbase
others
HDFS short-circuit
Data hedged ，multi thread read
Test stale datanodes, avoid the slow datanode
Shutdown balance
Multiwal， Improve write performance
13 . hosted by
03 Monitor Scenario
14 . hosted by
Monitor senario
grafana
TeleGraf
TSD
Task metrics HAProxy HBASE
Hadoopmetrics TSD
15 . hosted by
Monitor Scenario
16 . hosted by
04 Data Mining Scenario
17 . hosted by
Data Mining Scenario
18 . hosted by
05 RealTime ETL Scenario
19 . hosted by
RealTime ETL Scenario
Subtitle Text
RealTime DB ETL RealTime log ETL
fast
write
bigdata
storage HBASE 3 fast
read
4
change
record
20 . hosted by
06 Hdic Scence
21 . hosted by
Hdic Scenario
Subtitle Text
3
4
22 . hosted by
Data For Ke
热门城市
