1 . CalvinFS: Consistent WAN Replica5on and Scalable Metdata Management for Distributed File Systems Slide credits: Thomas Kao 1 

2 . Background •  Scalable solu5ons provided for data storage, why not file systems? 2 

3 . Mo5va5on •  OLen boMlenecked by the metadata management layer •  Availability suscep5ble to data center outages •  S5ll provides expected file system seman5cs 3 

4 . Key Contribu5ons •  Distributed database system for scalable metadata management •  Strongly consistent geo-replica5on of file system state 4 

5 . Calvin: Log •  Many front end servers •  Asynchronously-replicated distributed block store •  Small number of “meta-data” log servers •  Transac5on requests are replicated and appended, in order, by the “meta log” 5 

6 . Calvin: Storage Layer •  Knowledge of physical data store organiza5on and actual transac5on seman5cs •  Read/write primi5ves that execute on one node •  Placement manager •  Mul5version key-value store at each node, plus consistent hashing mechanism 6 

7 . Calvin: Scheduler •  Drives local transac5on execu5on •  Fully examines transac5on before execu5on •  Determinis5c locking •  Transac5on protocol: Execute Serve Collect Perform all transac5on remote remote read local reads to reads results comple5on •  No distributed commit protocol 7 

8 . CalvinFS Architecture •  Design Principles: •  Components –  Main-memory metadata •  Block store store •  Calvin database –  Poten5ally many small files •  Client library –  Scalable read/write throughput –  Tolerate slow writes –  Linearizable and snapshot reads –  Hash-par55oned metadata –  Op5mize for single-file opera5ons 8 

9 . CalvinFS Block Store •  Variable-size immutable blocks –  1 byte to 10 megabytes •  Block storage and placement –  Unique ID –  Block “buckets” –  Global Paxos-replicated config file –  Compacts small blocks 9 

10 . CalvinFS Metadata Management •  Key-value store –  Key: absolute path of file/directory –  Value: entry type, permissions, contents 10 

11 . Metadata Storage Layer •  Six transac5on types: –  Read(path) –  Create{File, Dir}(path) –  Resize(path, size) –  Write(path, file_offset, source, source_offset, num_bytes) –  Delete(path) –  Edit permissions(path, permissions) 11 

12 . Recursive Opera5ons on Directories •  Use OLLP •  Analyze phase –  Determines affected entries and read/write set •  Run phase –  Check that read/write set has not grown 12 

13 .Performance: File Counts and Memory Usage •  10 million files of varying size per machine •  Far less memory used per machine •  Handles many more files than HDFS 13 

14 .Performance: Throughput Linear Sub-linear scalability scalability Linear scalability 14 

15 .Performance: Latency Write/append latency dominated by WAN replica5on 15 

16 . Performance: Fault Tolerance •  Able to tolerate outages with liMle to no hit to availability 16 

17 . Discussion Pros Cons •  Fast metadata •  File crea5on is distributed management transac5on, doesn’t scale •  Deployments are well scalable on large •  Metadata opera5ons have to clusters recursively modify all entries •  Huge storage in affected subtree capabili5es •  High throughput •  File-fragmenta5on addressed of reads and using mechanism that updates en5rely rewrites files •  Resistant to datacenter outages 17 

18 . Discussion Ques5ons •  Unlimited number of files? •  What about larger files? 18