华为HBase灾难恢复解决方案

1 .HBase Disaster Recovery Solution at Huawei Ashish Singhi 1 

2 .About.html • Senior Technical Leader at Huawei • Around 6 years of experience in Big Data related projects • Apache HBase Committer 2 

3 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 3 

4 .Why Disaster Recovery ? Cost of Downtime 4 

5 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 5 

6 .Backup Vs Disaster Recovery Two different problems and solutions Backup Disaster Recovery Process Archive items to Replicate to secondary cold media site Infrastructure Medium level Duplicate of active cluster (high level) Cost Affordable Expensive Restore process One to few at a One to everything time Restore time Slow Fast Production usage Common Rare 6 

7 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 7 

8 .HBase Disaster Recovery • HBase Disaster recovery is based on replication, which mirrors data across a network in real time. • The technology is used to move data from a local source location to one or more target locations. • Replication over WAN has become an ideal technology for disaster recovery to prevent data loss in the event of failure. 8 

9 .Deployment Strategies 9 

10 .Active – Standby Cluster Active Cluster Standby Cluster HBase HBase Read Replication Write /hbase/clusterStat Serves Read Serves only /hbase/clusterStat e: active and Write Read Client e: standby Client Requests Requests ZooKeeper ZooKeeper 10 

11 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 11 

12 .Replication Source Cluster Peer Cluster 1 [tableCfs - 1] Region Server Region Server Bulk load 1 Replication 1 1 Table Replication Sink Source/End Point 1 WAL Batch Batch 1 Replication Source Manager 2 Bulk load 1 Bulk load 1 Replication 31 21 1 Replication Sink Table Source/End Point 2 1 …/peers/ Batch …/rs/ Region Server …/hfile-refs/ Peer Cluster 2 [tableCfs - ] ZooKeeper 12 

13 .Sync DDL Operations • Synchronize the table properties across clusters • Any change in the source cluster, reflects immediately in the peer clusters. • Does not break the replication. • An additional option with DDL command to sync • Internally sync those changes to peer clusters. 13 

14 .Sync Security related Data • Synchronize security related HBase data across the clusters • Any update in the source cluster ACL, Quota or Visibility Labels table, reflects immediately in peer clusters. • A custom WAL entry filter is added in replication for this. • Does not break the security for HBase data access. 14 

15 .Read Only Cluster • Enable a cluster to serve only read requests • A coprocessor based solution • Standby cluster will serve all the read requests • Standby cluster will serve write requests only if the requests is coming from a, • Super user • From a list of accepted IPs 15 

16 .Cluster Recovery Active Standby Cluster Standby Active Cluster HBase HBase Read Replication Write /hbase/clusterStat Serves only Serves Read /hbase/clusterStat e: active standby Read Client and Write e: standby active Requests Client Requests ZooKeeper ZooKeeper 16 

17 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 17 

18 .Miscellaneous • Increased the default replication.source.ratio to 0.5 • Adaptive hbase.replication.rpc.timeout • Active cluster HDFS server configurations are maintained in Standby cluster ZooKeeper for bulk loaded data replication. 18 

19 .Agenda • Why Disaster Recovery ? • Backup Vs Disaster Recovery • HBase Disaster Recovery • Solution • Miscellaneous • Future Work 19 

20 .Future work • Move HBase Replication tracking from ZooKeeper to HBase table (HBASE-15867) • Copy bulk loaded data to peer with data locality • Replication data network bandwidth throttling. 20 

21 .Thank You！ mailto: ashishsinghi@apache.org Twitter: ashishsinghi89 21