TokuDB Troubleshooting

1 .TokuDB Troubleshooting March, 22, 2017 Sveta Smirnova, George O. Lorch III, Vlad Lesin 

2 .Table of Contents •What you need to know about TokuDB? •Data Corruption and Inconsistencies •Locks •Performance Query Performance High Concurrency Performance •Instruments 2 

3 .What you need to know about TokuDB? 

4 .Full featured engine • Transactional • ACID • MVCC • Row-level locking • XA • Transaction Isolation Levels • No Foreign Keys 4 

5 .Optimized for Writes • Fast inserts • Bulk loader • Compression • Fractal Tree 5 

6 .Optimizations for Retrievals • Clustered secondary indexes • Indexes can include up to 32 columns • Indexes do not age 6 

7 .Online Operations • OPTIMIZE TABLE • Index creations • DROP INDEX is offline • Hot column operations • Addition • Deletion • Expansion • Rename 7 • Backup 

8 .Read Free Replication • Slave-only feature • Requires RBR on master • Fast slaves • Must be read-only • Tables must have UNIQUE key • You can use any engine on master 8 

9 .Powerful Diagnostic Tools • SHOW ENGINE TOKUDB STATUS • Detailed lock information • Tables in Information Schema • No Performance Schema Support • Runtime statistics in SHOW ENGINE TOKUDB STATUS • Will be added in future versions 9 

10 .Data Corruption and Inconsistencies 

11 .How TokuDB Deals with Data Corruption? • Handled automatically at startup • Fixes known inconsistencies • Real corruptions are not reparable! 11 

12 .How TokuDB Deals with Data Corruption? • Handled automatically at startup • You interfere this process only slightly • There is no option, similar to innodb force recovery • Feature Request FT-733 • tokudb strip frm data removes embedded .frm metadata from status files Should be used with great care! 11 

13 .How TokuDB Deals with Data Corruption? • Handled automatically at startup • You interfere this process only slightly • Backup restoration required when corruption cannot be cured automatically 11 

14 .How TokuDB Deals with Data Corruption? • Handled automatically at startup • You interfere this process only slightly • Backup restoration required when corruption cannot be cured automatically • Disable write cache for safety 11 

15 .How TokuDB Deals with Data Corruption? • Handled automatically at startup • You interfere this process only slightly • Backup restoration required when corruption cannot be cured automatically • Disable write cache for safety • Verify your backups! 11 

16 .Corruptions vs Inonsistencies • What are Symptoms of Real Corruption? • Assertions about bad checksums • Assertions while unpaking row data • Not recoverable 12 

17 .Corruptions vs Inonsistencies • What are Symptoms of Real Corruption? • Assertions about bad checksums • Assertions while unpaking row data • Not recoverable • What are Symptoms of Inconsistency? • Different query results for Primary vs secondary key access? 12 

18 .Automatic Inconsistencies Recovery • Are inconsistency limited to know key range? • Recoverable 13 

19 .Automatic Inconsistencies Recovery • Are inconsistency limited to know key range? • Is it not possible to limit inconsistency to known range? • Not recoverable • Use dump-reload 13 

20 .Automatic Inconsistencies Recovery • Are inconsistency limited to know key range? • Is it not possible to limit inconsistency to known range? • There can be some data ”loss” or ”gain” • If Primary key misses data • If Primary key still contains ”deleted” rows 13 

21 .File Formats • TokuDB file formats are not the same across MySQL variants • Percona Server 5.6 and 5.7 • MariaDB 10.x • Builds, released by Tokutek On Oracle MySQL On MariaDB 14 

22 .File Formats • TokuDB file formats are not the same across MySQL variants • Percona Server 5.6 and 5.7 • MariaDB 10.x • Builds, released by Tokutek On Oracle MySQL On MariaDB • Migrating will require logical dump and reload • Upgrade from Percona Server 5.6 to Percona Server 5.7 can be in place 14 

23 .TokuDB Hotbackup • Can cause InnoDB corruptions with innodb use native aio=1 15 

24 .TokuDB Hotbackup • Can cause InnoDB corruptions with innodb use native aio=1 • Cannot be used to create new slave • Work in progress to solve this 15 

25 .Locks 

26 .Locks in TokuDB • Row-level • No Auto Increment lock tuning • Table locks for table definition modifications 17 

27 .Which Issues Will We Discuss? • Lock wait timeout • session1> START TRANSACTION; 18 

28 .Which Issues Will We Discuss? • Lock wait timeout • session1> START TRANSACTION; • session1> UPDATE t SET f=’foo’ WHERE id=12345; 18 

29 .Which Issues Will We Discuss? • Lock wait timeout • session1> START TRANSACTION; • session1> UPDATE t SET f=’foo’ WHERE id=12345; • session2> UPDATE t SET f=’bar’ WHERE id=12345; 18