1 . MOB User Guide
Data comes in many sizes, and it is convenient to save the binary data like images, documents
into the HBase. While HBase can handle binary objects with cells that are 1 byte to 10MB long,
HBase's normal read and write paths are optimized for values smaller than 100KB in size. When
HBase deals with large numbers of values > 100kb and up to ~10MB of data, it encounters
performance degradations due to write amplification caused by splits and compactions. HBase
2.0+ has added support for better managing large numbers of *Medium Objects* (MOBs) that
maintains the same high performance, strongly consistently characteristics with low operational
overhead.
To enable the feature, one must enable and configure the mob components in each region server
and enable the mob feature on particular column families during table creation or table alter. Also
in the preview version of this feature, the admin must setup periodic processes that re-optimize the
layout of mob data.
Enable and configure the mob feature on region servers
Edit hbase-site.xml, change or add the following properties:
a. Set MOB file cache properties:
<property>
<name>hbase.mob.file.cache.size</name>
<value>1000</value>
<description>
Number of opened file handlers to cache.
A larger value will benefit reads by provinding more file handlers per mob
file cache and would reduce frequent file opening and closing.
However, if this is set too high, this could lead to a "too many opened file handers"
The default value is 1000.
</description>
</property>
<property>
<name>hbase.mob.cache.evict.period</name>
<value>3600</value>
<description>
The amount of time in seconds before the mob cache evicts cached mob files.
The default value is 3600 seconds.
</description>
</property>
<property>
<name>hbase.mob.cache.evict.remain.ratio</name>
2 . <value>0.5f</value>
<description>
The ratio (between 0.0 and 1.0) of files that remains cached after an eviction
is triggered when the number of cached mob files exceeds the hbase.mob.file.cache.size.
The default value is 0.5f.
</description>
</property>
b. Configure the MobMasterObserver as the coprocessor master to archive the MOB files after
the table is deleted.
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.coprocessor.MobMasterObserver </value>
</property>
Mob management
The mob feature introduces a new read and write path to HBase and in its current incarnation
requires external tools for housekeeping and optimization. There are two tools introduced -- the
expiredMobFileCleaner for handling TTLs and time based expiry of data, and the sweep tool for
coalescing small mob files or mob files with many deletions or updates.
a. Clean the expired MOB data(expiredMobFileCleaner)
org.apache.hadoop.hbase.mob.compactions.expiredMobFileCleaner tableName familyName
Set the mob clean delay, the default is one hour.
<property>
<name>hbase.mob.cleaner.delay</name>
<value>60 * 60 * 1000</value>
</property>
b. Sweep tool
org.apache.hadoop.hbase.mob.compactions.Sweeper tableName familyName
The properties are set as follows:
<property>
<description>
If there're too many cells deleted in a mob file, it's regarded
as a invalid file and needs to be re-written/merged.
If (mobFileSize-existingCellsSize)/mobFileSize>=ratio, it's regarded
as a invalid file. The default value is 0.3f.
</description>
3 . <name>hbase.mob.compaction.invalid.file.ratio</name>
<value>0.3f</value>
</property>
<property>
<description>
If the size of a mob is less than the threshold, it's regarded as a small
file and needs to be merged. The default value is 64MB.
</description>
<name>hbase.mob.compaction.small.file.threshold</name>
<value>67108864</value>
</property>
<property>
<description>
The flush size for the memstore used by sweep job. Each sweep reducer owns such a
memstore.
The default value is 128MB.
</description>
<name>hbase.mob.compaction.memstore.flush.size</name>
<value>134217728</value>
</property>
WARN: the worst case of using sweep tool is: the compaction of MOB files succeeds but the
update of ref (PUT) fails. That means the new MOB files have been created but failed to put the
new mob file paths to HBase, so these MOB files will not be referenced by HBase.
Tips: Please check the yarn-site.xml, add the HBase install dir: $HBASE_HOME/* and HBase lib
dir: $HBASE_HOME/lib/* to yarn.application.classpath
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
$HBASE_HOME/*, $HBASE_HOME/lib/*
</value>
</property>
Enable the mob feature on user tables
a. Set the column family be MOB
4 .HColumnDescriptor hcd = new HColumnDescriptor(“f”);
hcd.setValue(MobConstants.IS_MOB, Bytes.toBytes(Boolean.TRUE));
b. Set the MOB cell size threshold, the default is 102400
HColumnDescriptor hcd;
hcd.setValue(MobConstants.MOB_THRESHOLD, Bytes.toBytes(102400L);
To a client, mob cells actions just like normal cells.
c. Put the MOB value
KeyValue kv = new KeyValue(row1, family, qf1, ts, KeyValue.Type.Put, value );
Put put = new Put(row1);
put.add(kv);
region.put(put);
d. Get the MOB value
Scan scan = new Scan();
InternalScanner scanner = (InternalScanner) region.getScanner(scan);
scanner.next(result, limit);
There is a special scanner mode users can use to read the raw values
e. Get the MOB reference
Scan scan = new Scan();
scan.setAttribute(MobConstants.MOB_SCAN_RAW, Bytes.toBytes(Boolean.TRUE));
InternalScanner scanner = (InternalScanner) region.getScanner(scan);
scanner.next(result, limit);
Run Mob Integration Test
sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestMOB
热门城市
