MongoDB Backup and Recovery Field Guide



1.MongoDB Backup & Recovery Field Guide Tim Vaillancourt Percona Speaker Name

2.`whoami` { name: “tim”, lastname: “vaillancourt”, employer: “percona”, techs: [ “mongodb”, “mysql”, “cassandra”, “redis”, “rabbitmq”, “solr”, “mesos” “kafka”, “couch*”, “python”, “golang” ] }

3.Agenda ● Backups ○ Logical ○ Binary ○ Architecture ○ Security ● Consistent Backups ● Percona-Lab/mongodb_consistent_backup ● Recovery ○ Logical ○ Binary

4. MongoDB Backups “An admin is only worth the backups they keep” ~ Unknown

5.Logical Backups ● Storage-engine agnostic ● Logical representation ○ Is not machine bytes ○ Is not architecture specific / dependent ● Simple backup procedure ● Low disk consumption ● Common logical backup tools ○ mongodump - MongoDB ○ mysqldump - MySQL ○ ...

6.Logical Backups: mongodump ● ‘mongodump’ tool from mongo-tools project ● Supports ○ Multi-threaded dumping in 3.2+ ○ Optional inline gzip compression of data ○ Optional dumping of oplog for single-node consistency ○ Replica set awareness via --readPreference= flag ■ Ie: primary, primaryPreferred, secondary, secondaryPreferred, nearest

7.Logical Backups: mongodump ● Process ○ Tool issues .find() query with $snapshot cursor ○ Stores BSON data in a file per collection ○ Stores BSON oplog data in “oplog.bson”

8.Logical Backups: mongodump ● Useful for... ○ upgrades of very old systems, eg: 2.6 -> 3.4 upgrade ○ protection from binary-level/storage-engine corruption ○ export/import to different CPU architecture

9.Logical Backups: mongodump ● Limitations ○ Index metadata only in backup ■ Indexes are rebuilt entirely, in serial!! ■ Often indexing process takes longer than restoring the data! ■ Expect hours or days of restore time ○ Not Sharding aware ■ Sharded backups are not Point-in-Time consistent ■ Must use mongos, very inefficient

10.Logical Backups: mongodump ● Limitations ○ Fetch from storage-engine, serialization, networking, etc is very inefficient ○ Oplogs fetched in batch at end / oplog must be as long as the backup run-time ○ Wire Protocol Compression (added in 3.4+, default in 3.6) not supported yet: 668 (Please vote/watch Issue!)

11.Binary Backups ● Backup data is the storage-engine files ● Options ○ Cold Backup ○ LVM Snapshot ○ Hot Backup ■ Percona Server for MongoDB (FREE!) ■ MongoDB Enterprise Hot Backup ($$$) ■ NOTE: MMAPv1 not supported

12.Binary Backups ● Benefits ○ Faster backup time ■ Backups can move at the speed of the disk/host ○ Faster time to restore ■ Indexes are backed up entirely ■ No time spent rebuilding indexes (!!!)

13.Binary Backups ● Limitations ○ Increased backup storage requirements ○ CPU Architecture limitations (64-bit vs 32-bit) ○ Cascading corruption ○ Batteries not included ■ Not Sharding aware ■ Not Replica Set aware ■ Oplog is not captured separately

14.Binary Backups ● Process ○ Cold Backup ■ Stop a mongod SECONDARY, copy/archive dbPath ○ LVM Snapshot ■ Optionally call ‘db.fsyncLock()’ (not required in 3.2+ with Journaling) ■ Create LVM snapshot of the dbPath ■ Copy/Archive dbPath ■ Remove LVM snapshot (as quickly as possible!) ■ NOTE: LVM snapshots can cause up to 30%* write latency impact to disk (due to COW)

15.Binary Backups ● Process ○ Hot Backup (PSMDB or MongoDB Enterprise) ■ Pay $$$ for MongoDB Enterprise or download PSMDB for free(!) ■ db.adminCommand({ createBackup: 1, backupDir: "/data/mongodb/backup" }) ■ Copy/archive the output path ■ Delete the backup output path ■ NOTE: RocksDB-based createBackup creates filesystem hardlinks whenever possible! ■ NOTE: Delete RocksDB backupDir as soon as possible to reduce bloom filter overhead!

16.Backup Security ● Authorization ○ “backup” built-in role ○ Client Source IP restriction (new in 3.6!) ○ x509 Client Certificate vs Passwords ● Transport ○ Use SSL/TLS with MongoDB ■ “preferSSL” mode allows secure and plain ○ Upload backups with secure connection

17.Backup Security ● Storage ○ Who can access the backups? ○ File System access ○ Encryption

18.Backup Architecture ● Risks ○ Dynamic nature of Replica Set ○ Impact of backup on live nodes ● Example: Cheap Disaster-Recovery ○ Place a ‘hidden: true’ SECONDARY in another location ○ Optionally use cloud object store (AWS S3, Google GS, etc)

19.Backup Architecture ● Example: Replica Set Tags ○ “tags” allow fine-grained server selection with key/value pairs ○ Use key/value pair to fence various application workflows ○ Example: ■ { “role”: “backup” } == Backup Node ■ { “role”: “application” } == App Node

20.Backup Style ● Full ○ Take a full copy of the database data ○ Simple ○ Very costly to store frequent backups ● Incremental ○ Logical (oplog) ○ Binary ● Hybrid ○ Full every N days ○ Incremental every 12-24 hours

21.Backup Tips ● Store backups in structured paths ○ <name>-YYYYMMDD_HHMM ● Also backup the ○ MongoDB configuration ○ (When secure) MongoDB Internal Authentication key

22.Consistent Backups

23.Shards and Consistency ● Problem ○ Backup tools are replset consistent but NOT sharding consistent ○ Shards can complete backup at different times ○ ..but a consistent backup means all shards are at the same point in time!

24.Shards and Consistency ● Solution: a process that ■ Understands the sharded cluster topology ■ Is able to watch the oplogs of all shards ■ Handles the backups of all shards ■ Ensures the oplogs are aligned for all shards for consistency ■ ...


26.MCB: History ● Python project by Percona-Lab for consistent backups ● URL: _backup ● Best-effort support, not a “Percona Product” ● Created to solve limitations in MongoDB backup tools: ○ Replica Set and Sharded Cluster awareness ○ Cluster-wide Point-in-time consistency ○ In-line Oplog backup (vs post-backup) ○ Notifications of success / failure

27.MCB: Features ● Features ○ Auto replica-set and sharded-cluster discovery ○ Cluster-consistent live oplog backup ○ Remote Upload (AWS S3, Google Cloud Storage and Rsync) ○ Archiving (Tar or ZBackup deduplication and optional AES-at-rest) ○ CentOS/RHEL7 RPMs and Docker-based releases (.deb soon!) ○ Single Python PEX binary ○ Multithreaded / Concurrent and auto-scales to available CPUs

28.MCB: Features ● Low-Impact ○ Uses Secondary nodes only ○ Considers (Scoring) ■ Replication Lag ■ Replication Priority ■ Replication Health / State ■ Hidden-Secondary State (preferred by tool) ■ Fails if chosen Secondary becomes Primary (on purpose)

29.MCB: Future ● Future ○ End of life of Python-based tool ○ Productization ○ Incremental Backups ○ Binary-level Backups (Hot Backup, Cold Backup, LVM, Cloud-based, etc) ○ Restore Functionality ○ Instrumentation / Metrics