Bcache Stability- Improvement for Cloud Infrastructure
分享
点赞
2
收藏
1
下载 2
-
快召唤伙伴们来围观吧
-
微博
QQ
QQ空间
贴吧
-
文档嵌入链接
- 复制
-
-
微信扫一扫分享
-
已成功复制到剪贴板
圆圆
/
发布于
/
2422
人观看
Bcache was merged as a block layer cache in Linux v3.10. Although there are bug reports time to time, it has been deployed in various systems of cloud infrastructure in the past five years.
Since 2017 we have more active developers (most of them are from China) to contribute many critical stability fixes. These fixes address issues from deadlock, I/O hang, kernel oops. Especially when cache or cached devices are failed or gone, bcache device can be stopped precisely and elegantly (panic on failure is unnecessary anymore). The stability of bcache has huge improvement, and it is ready for more work loads from large scaled could infrastructure for now days.
展开查看详情
1 . Bcache Stability
Improvement for Cloud Infrastructure
Coly Li Junhui Tang
Software Engineer Software Engineer
SUSE Labs ZTE
2 .About Speakers
Coly Li
• Software Engineer, SUSE Labs
• Maintains md/dm/bcache for SUSE Linux
• Bcache maintainer upstream Linux kernel
Junhui Tang
• Software Engineer, ZTE
• Maintains bcache for ZTE storage products
• Active developer and important contributor to
bcache of upstream Linux kernel
3 .Bcache Overview
A high performance block layer cache, exists in
Linux kernel for 5+ years.
Originally developed by Kent Overstreet, and
merged into Linux v3.10.
Widely deployed in industry (virtualization, data
base, software define storage, etc) everywhere hot
data can be accelerated.
4 . Brief Design
• Storage space allocated in buckets
• Cache indexed by B+ tree, in-memory binary search
tree used within each B+ tree node
All the above charts copied from Hua Rui <huarui@szsandstone.com>, ”Practices for accelerating Ceph OSD with bcache”
5 .Improvement required by Industry
• Device failure handling
• Writeback performance
• I/O latency in worst case
• Stability, don’t lock, don’t panic
Many development and improvement take place in
the past 2 years.
6 .Device Failure Handling
• Potential data corruption if device failure is not
handled properly
– Broken cache device
– Broken backing device
• Backing device gone without I/O
• Linux v4.18 has fundamental device failure
handling for bcache
– Fix many bugs for dependences of multiple reference counters
– Retire cache set for broken cache
– Stop bcache device for broken cache
– Stop bcache device for broken backing device
7 .Writeback Performance
• Dirty blocks on cache device are mostly randomly
distributed on backing device.
• Write them back to spindle hard drive results
unhappy performance (40~400KB/s)
• Even worse on SMR (Shingled Magnetic Recording)
hard drives
• Michael Lyle makes situation better by re-ordering
write requests before issuing to backing device.
• Throughput improved 3+ times for ideal conditions.
8 .I/O latency in Worst Case
• There are multiple threads to access B+ tree in
parallel
– Writeback
– Garbage collection
– Data insert/invalidate/replacement
• Lock contention
• Dependency loop between B+ tree and journal
• Junhui Tang from ZTE contributes a lot of fixes to
make things much better
– Reduce worst latency from 120+ seconds to ~200ms
– Well done!
9 .Stability, don’t lock, don’t panic
• Deadlock on cache retire code path
• Panic of NULL pointer deference of gone struct
block_device *bdev.
• Dependence circle within multiple locks among
multiple threads
Almost all reported issues are fixed, bcache code
in Linux v4.18 has improved stability with better
overall performance.
10 .Stable for Cloud Infrastructure
Positive feedback from industry partners and
community,
• Virtualization
• Software Defined Storage
• Hyper-converged Infrastructure
Bcache is stable to I/O cache acceleration for your
large scale could infrastructure.
Full enterprise production support from SUSE Linux
Enterprise Sever 12 and 15.
11 .Credit to Bcache Developers
Incomplete contributors list since Linux v3.10:
Kent Overstreet (164) Kees Cook (5) Liang Chen (2)
Coly Li (26) Ming Lei (5) Jan Kara (2)
Tang Junhui (24) Ingo Molnar (4) Al Viro (2)
Michael Lyle (13) Mike Christie (4) Mike Snitzer (2)
Slava Pestov (12) Andy Shevchenko (3) Wei Yongjun (1)
Nicholas Swenson (10) Zheng Liu (3) Zhai Zhaoxuan (1)
Bart Van Assche (9) Dan Carpenter (3) Jianjian Huo (1)
Jens Axboe (7) Jiri Kosina (3) Gu Zheng (1)
Christoph Hellwig (7) NeilBrown (3) Guoqing Jiang (1)
Eric Wheeler (7) Surbhi Palande (2) Greg Kroah-Hartman (1)
Yijing Wang (5) Rui Hua (2) Chengguang Xu (1)
Gabriel de Perthuis (5) Michal Hocko (2)
12 .Development Roadmap
• Big endian support
• Reduce lock contention on B+ btree iteration
• User space tool enhancement
• SMR & 4K native hard drive support
We have active developers and users community,
wildly used in cloud industry. Welcome to join us
for a better block layer cache!
linux-bcache@vger.kernel.org
热门城市
