Speedy Transactions in Multicore In-Memory Databases

1 . Speedy Transactions in Multicore In-Memory Databases Stephen Tu, Wenting Zheng, Eddie Kohler† , Barbara Liskov, and Samuel Madden MIT CSAIL and † Harvard University Abstract nization scale with the data, allowing larger databases to Silo is a new in-memory database that achieves excel- support more concurrency. lent performance and scalability on modern multicore Silo uses a Masstree-inspired tree structure for its un- machines. Silo was designed from the ground up to use derlying indexes. Masstree [23] is a fast concurrent B- system memory and caches efficiently. For instance, it tree-like structure optimized for multicore performance. avoids all centralized contention points, including that of But Masstree only supports non-serializable, single-key centralized transaction ID assignment. Silo’s key contri- transactions, whereas any real database must support bution is a commit protocol based on optimistic concur- transactions that affect multiple keys and occur in some rency control that provides serializability while avoid- serial order. Our core result, the Silo commit protocol, is ing all shared-memory writes for records that were only a minimal-contention serializable commit protocol that read. Though this might seem to complicate the en- provides these properties. forcement of a serial order, correct logging and recov- Silo uses a variant of optimistic concurrency control ery is provided by linking periodically-updated epochs (OCC) [18]. An OCC transaction tracks the records it with the commit protocol. Silo provides the same guar- reads and writes in thread-local storage. At commit time, antees as any serializable database without unnecessary after validating that no concurrent transaction’s writes scalability bottlenecks or much additional latency. Silo overlapped with its read set, the transaction installs all achieves almost 700,000 transactions per second on a written records at once. If validation fails, the transaction standard TPC-C workload mix on a 32-core machine, as aborts. This approach has several benefits for scalability. well as near-linear scalability. Considered per core, this OCC writes to shared memory only at commit time, af- is several times higher than previously reported results. ter the transaction’s compute phase has completed; this short write period reduces contention. And thanks to the 1 Introduction validation step, read-set records need not be locked. This matters because the memory writes required for read Thanks to drastic increases in main memory sizes and locks can induce contention [11]. processor core counts for server-class machines, modern Previous OCC implementations are not free of scal- high-end servers can have several terabytes of RAM and ing bottlenecks, however, with a key reason being the re- 80 or more cores. When used effectively, this is enough quirement for tracking “anti-dependencies” (write-after- processing power and memory to handle data sets and read conflicts). Consider a transaction t1 that reads a computations that used to be spread across many disks record from the database, and a concurrent transaction and machines. However, harnassing this power is tricky; t2 that overwrites the value t1 saw. A serializable sys- even single points of contention, like compare-and- tem must order t1 before t2 even after a potential crash swaps on a shared-memory word, can limit scalability. and recovery from persistent logs. To achieve this order- This paper presents Silo, a new main-memory ing, most systems require that t1 communicate with t2 , database that achieves excellent performance on multi- such as by posting its read sets to shared memory or via core machines. We designed Silo from the ground up a centrally-assigned, monotonically-increasing transac- to use system memory and caches efficiently. We avoid tion ID [18, 19]. Some non-serializable systems can all centralized contention points and make all synchro- avoid this communication, but they suffer from anoma- Permission to make digital or hard copies of part or all of this work for lies like snapshot isolation’s “write skew” [2]. personal or classroom use is granted without fee provided that copies Silo provides serializability while avoiding all shared- are not made or distributed for profit or commercial advantage and that memory writes for read transactions. The commit proto- copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other col was carefully designed using memory fences to scal- uses, contact the Owner/Author. ably produce results consistent with a serial order. This leaves the problem of correct recovery, which we solve Copyright is held by the Owner/Author(s). using a form of epoch-based group commit. Time is di- SOSP’13, Nov. 3–6, 2013, Farmington, Pennsylvania, USA. ACM 978-1-4503-2388-8/13/11. vided into a series of short epochs. Even though transac- http://dx.doi.org/10.1145/2517349.2522713 tion results always agree with a serial order, the system 

2 .does not explicitly know the serial order except across 2 Related work epoch boundaries: if t1 ’s epoch is before t2 ’s, then t1 pre- A number of recent systems have proposed storage cedes t2 in the serial order. The system logs transactions abstractions for main-memory and multicore systems. in units of whole epochs and releases results to clients These can be broadly classified according to whether or at epoch boundaries. As a result, Silo provides the same not they provide transactional support. guarantees as any serializable database without unneces- sary scalability bottlenecks or much additional latency. 2.1 Non-transactional systems Epochs have other benefits as well; for example, we use The non-transactional system most related to Silo them to provide database snapshots that long-lived read- is Masstree [23], an extremely scalable and high- only transactions can use to reduce aborts. throughput main-memory B+ -tree. Masstree uses tech- niques such as version validation instead of read locks On a single 32-core machine, Silo achieves roughly and efficient fine-grained locking algorithms. It builds 700,000 transactions per second on the standard TPC- on several prior trees, including OLFIT [4], Bronson et C benchmark for online transaction processing (OLTP). al. [3], and Blink -trees [20], and adds new techniques, in- This is about 22,000 transactions per second per core. cluding a trie-like structure that speeds up key compar- Per-core transaction throughput at 32 cores is 91% isons. Silo’s underlying concurrent B+ -tree implemen- of that at 8 cores, a small drop indicating that Silo tation was inspired by Masstree. scales well. For context, the database literature for high- PALM [31] is another high-throughput B+ -tree struc- performance in-memory OLTP systems cites per-core ture designed for multicore systems. It uses a batch- TPC-C throughput at least several times lower than ing technique, extensive prefetching, and intra-core Silo’s [16, 25, 27, 28], and benchmarks on our hardware SIMD parallelism to provide extremely high throughput. with a commercial main-memory database system per- PALM’s techniques could potentially speed up Silo’s form at most 3,000 transactions per second per core. tree operations. An important Silo design choice is its shared-memory The Bw-tree [21] is a high-throughput multiversion store: any database worker can potentially access the tree structure optimized for multicore flash storage. New whole database. Several recent main-memory database versions of data are installed using delta records and designs instead use data partitioning, in which database compare-and-swaps on a mapping table; there are no workers effectively own subsets of the data [25, 32]. Par- locks or overwrites (we found both helpful for perfor- titioning can shrink table sizes and avoids the expense of mance). Silo’s data structures are designed for main managing fine-grained locks, but works best when the memory, whereas many of the Bw-tree’s structures are query load matches the partitioning. To understand the designed for flash. Like Silo, the Bw-tree uses RCU- tradeoffs, we built and evaluated a partitioned variant of style epochs for garbage collection; Silo’s epochs also Silo. Partitioning performs better for some workloads, support scalable serializable logging and snapshots. but a shared-memory design wins when cross-partition LLAMA [22] adds support for transactional logging, but transactions are frequent or partitions are overloaded. its logger is centralized. 2.2 Transactional systems Silo assumes a one-shot request model in which all parameters for each client request are available at Silo uses optimistic concurrency control [10, 18], which the start, and requests always complete without further has several advantages on multicore machines, including client interaction. This model is well-suited for OLTP a relatively short period of contention. However, OCC workloads. If high-latency client communication were and its variants (e.g., [1, 12, 34]) often induce contention part of a transaction, the likelihood of abort due to con- in the commit phase, such as centralized transaction ID current updates would grow. assignment or communication among all concurrently executing transactions. Our performance is higher than a full system would Larson et al. [19] recently revisited the performance observe since our clients do not currently use the net- of locking and OCC-based multi-version concurrency work. (In Masstree, network communication reduced control (MVCC) systems versus that of traditional throughput by 23% [23]; we would expect a similar single-copy locking systems in the context of multi- reduction for key-value workloads, less for workloads core main-memory databases. Their OCC implementa- with more computation-heavy transactions.) Neverthe- tion exploits MVCC to avoid installing writes until com- less, our experiments show that Silo has very high per- mit time, and avoids many of the centralized critical sec- formance, that transactions can be serialized without tions present in classic OCC. These techniques form the contention, and that cache-friendly design principles basis for concurrency control in Hekaton [8], the main- work well for shared-memory serializable databases. memory component of SQL Server. However, their de- 

3 .sign lacks many of the multicore-specific optimizations In Silo, we eliminate the bottleneck of a centralized of Silo. For example, it has a global critical section when lock manager by co-locating locks with each record. assigning timestamps, and reads must perform non-local VLL [29] also adopts this approach, but is not focused memory writes to update other transactions’ dependency on optimizing for multicore performance. sets. It performs about 50% worse on simple key-value Shore-MT [15], Jung et al. [17], and Horikawa [14] workloads than a single-copy locking system even un- take traditional disk- and 2PL-based relational database der low levels of contention, whereas Silo’s OCC-based systems and improve multicore scalability by removing implementation is within a few percent of a key-value centralized locking and latching bottlenecks. However, system for small key-value workloads. 2PL’s long locking periods and requirement for read Several recent transactional systems for multicores locking introduce inherent scalability concerns on multi- have proposed partitioning as the primary mechanism core architectures. for scalability. DORA [25] is a locking-based system Porobic et al. [28] perform a detailed performance that partitions data and locks among cores, eliminating analysis of shared-nothing versus shared-everything long chains of lock waits on a centralized lock manager OLTP on a single multicore machine, and conclude that and increasing cache affinity. Though this does improve shared-nothing configurations are preferable due to the scalability, overall the performance gains are modest— effects of non-uniform memory accesses (NUMA). Our about 20% in most cases—compared to a locking sys- results argue otherwise. tem. Additionally, in some cases, this partitioning can Snapshot transactions like Silo’s have been imple- cause the system to perform worse than a conventional mented before, including in distributed transactional system when transactions touch many partitions. systems [7]. Our implementation offers very recent PLP [26] is follow-on work to DORA. In PLP, the snapshots and tightly integrates with our epoch system. database is physically partitioned among many trees such that only a single thread manages a tree. The par- 2.3 Transactional memory titioning scheme is flexible, and thus requires maintain- Silo’s goals (fast transactions on an in-memory ing a centralized routing table. As in DORA, running a database) resemble those of a software transactional transaction requires decomposing it into a graph of ac- memory system (fast transactions on arbitrary memory). tions that each run against a single partition; this necessi- Recent STM systems, including TL2 [9], are based on tates the use of rendezvous points, which are additional optimistic concurrency control and maintain read and sources of contention. The authors only demonstrate a write sets similar to those in OCC databases. Some STM modest improvement over a two-phase locking (2PL) implementation techniques, such as read validation and implementation. collocation of lock bits and versions, resemble Silo’s. H-Store [32] and its commercial successor VoltDB (Many of these ideas predate STM.) Other STM im- employ an extreme form of partitioning, treating each plementation techniques are quite different, and overall partition as a separate logical database even when par- Silo has more in common with databases than STMs. titions are collocated on the same physical node. Trans- Our durability concerns are irrelevant in STMs; Silo’s actions local to a single partition run without locking at database structures are designed for efficient locking and all, and multi-partition transactions are executed via the concurrency, whereas STM deals with an arbitrary mem- use of whole-partition locks. This makes single-partition ory model. transactions extremely fast, but creates additional scala- STM would not be an appropriate implementation bility problems for multi-partition transactions. We com- technique for Silo. Our transactions access many shared pare Silo to a partitioned approach and confirm the intu- memory words with no transactionally relevant mean- ition that this partitioning scheme is effective with few ing, such as nodes in the interior of the tree. Unlike STM multi-partition transactions, but does not scale well in systems, Silo can distinguish relevant from irrelevant the presence of many such transactions. modifications. Ignoring irrelevant modifications is crit- Multimed [30] runs OLTP on a single multicore ma- ical for avoiding unnecessary aborts. Finally, we know chine by running multiple database instances on separate of no STM that beats efficient locking-based code. cores in a replicated setup. A single master applies all writes and assigns a total order to all updates, which are 3 Architecture then applied asynchronously at the read-only replicas. Multimed only enforces snapshot isolation, a weaker no- Silo is a relational database that provides tables of typed, tion of consistency than serializability. Read scalability named records. Its clients issue one-shot requests: all pa- is achieved at the expense of keeping multiple copies of rameters are available when a request begins, and the re- the data, and write-heavy workloads eventually bottle- quest does not interact with its caller until it completes. neck on the master. One-shot requests are powerful; each consists of one 

4 .or more serializable database transactions that may in- Client Client Client clude arbitrary application logic. We write one-shot re- quests in C++, with statements to read and manipulate the Silo database directly embedded. Of course a one- Request Response shot request could also be written in SQL (although we Transaction Logic: T = FIND(A.x,>10) have not implemented that). Supporting only one-shot foreach t in T: Worker 1 … Worker n thread 1 thread n requests allows us to avoid the potential for stalls that is UPDATE(t.x,10) present when requests involve client interaction. Silo tables are implemented as collections of index Primary Secondary Tree for Trees trees, including one primary tree and zero or more sec- Table A for Table A ondary trees per table (see Figure 1). Each record in a table is stored in a separately-allocated chunk of mem- ory pointed to by the table’s primary tree. To access a … key1,dat1 key2,dat2 record by primary key, Silo walks the primary tree us- Records in leaves key7 key2 … ing that key. Primary keys must be unique, so if a table has no natural unique primary key, Silo will invent one. Secondary leaves contain primary key values In secondary indexes, the index key maps to a secondary record that contains the relevant record’s primary key(s). Thus, looking up a record by a secondary index requires Figure 1: The architecture of Silo. two tree accesses. All keys are treated as strings. This organization is typical of databases. 4 Design Each index tree is stored in an ordered key-value This section describes how we execute transactions in structure based on Masstree [23]. Masstree read opera- Silo. Our key organizing principle is to eliminate un- tions never write to shared memory; instead, readers co- necessary contention by reducing writes to shared mem- ordinate with writers using version numbers and fence- ory. Our variant of OCC achieves serializability, even based synchronization. Compared to other concurrent after recovery, using periodically-updated epochs; epoch B-trees [4, 31], Masstree adopts some aspects of tries, boundaries form natural serialization points. Epochs also which optimizes key comparisons. Each Masstree leaf help make garbage collection efficient and enable snap- contains information about a range of keys, but for keys shot transactions. Several design choices, such as trans- in that range, the leaf may point either directly to a action ID design, record overwriting, and range query record or to a lower-level tree where the search can be support, simplify and speed up transaction execution continued. Although our implementation uses tree struc- further, and the decentralized durability subsystem also tures, our commit protocol is easily adaptable to other avoids contention. In the rest of this section we describe index structures such as hash tables. the fundamentals of our design (epochs, transaction IDs, and record layout), then present the commit protocol and Each one-shot request is dispatched to a single explain how database operations are implemented. Later database worker thread, which carries out the request to subsections describe garbage collection, snapshot trans- completion (commit or abort) without blocking. We run action execution, and durability support. one worker thread per physical core of the server ma- chine, and have designed Silo to scale well on modern 4.1 Epochs multicore machines with tens of cores. Because trees are Silo is based on time periods called epochs, which are stored in (shared) main memory, in Silo any worker can used to ensure serializable recovery, to remove garbage access the entire database. (e.g., due to deletes), and to provide read-only snapshots. Although the primary copy of data in Silo is in main Each epoch has an epoch number. memory, transactions are made durable via logging to A global epoch number E is visible to all threads. A stable storage. Results are not returned to users until they designated thread periodically advances E; other threads are durable. access E while committing transactions. E should ad- vance frequently, since the epoch period affects transac- Read-only transactions may, at the client’s discretion, tion latency, but epoch change should be rare compared run on a recent consistent snapshot of the database in- to transaction duration so that the value of E is gener- stead of its current state. These snapshot transactions ally cached. Our implementation updates E once every return slightly stale results, but never abort due to con- 40 ms; shorter epochs would also work. No locking is current modification. required to handle E. 

5 . Each worker w also maintains a local epoch num- that protects memory structures. The latest-version bit ber ew . This can lag behind E while the worker com- is 1 when a record holds the latest data for the corre- putes, and is used to determine when it is safe to collect sponding key. When a record is superseded (for instance, garbage. Silo requires that E and ew never diverge too when an obsolete record is saved temporarily for snap- far: E − ew ≤ 1 for all w. This is not usually a prob- shot transactions), the bit is turned off. Finally, the ab- lem since transactions are short, but if a worker does fall sent bit marks the record as equivalent to a nonexistent behind, the epoch-advancing thread delays its epoch up- key; such records are created in our implementations of date. Workers running very long transactions should pe- insert and remove. riodically refresh their ew values to ensure the system We use “TID” to refer to a pure transaction ID and makes progress. “TID word” for a TID plus status bits, including the lock. Snapshot transactions use additional epoch variables described below. 4.3 Data layout A record in Silo contains the following information: 4.2 Transaction IDs • A TID word as above. Silo concurrency control centers on transaction IDs, or • A previous-version pointer. The pointer is null if TIDs, which identify transactions and record versions, there is no previous version. The previous version is serve as locks, and detect conflicts. Each record contains used to support snapshot transactions (§4.9). the TID of the transaction that most recently modified it. TIDs are 64-bit integers. The high bits of each TID • The record data. When possible, the actual record contain an epoch number, which equals the global epoch data is stored in the same cache line as the record at the corresponding transaction’s commit time. The header, avoiding an additional memory fetch to ac- middle bits distinguish transactions within the same cess field values. epoch. The lower three bits are the status bits described Committed transactions usually modify record data below. We ignore wraparound, which is rare. in place. This speeds up performance for short writes, Unlike many systems, Silo assigns TIDs in a decen- mainly by reducing the memory allocation overhead for tralized fashion. A worker chooses a transaction’s TID record objects. However, readers must then use a ver- only after verifying that the transaction can commit. sion validation protocol to ensure that they have read a At that point, it calculates the smallest number that is consistent version of each record’s data (§4.5). (a) larger than the TID of any record read or written Excluding data, records are 32 bytes on our system. by the transaction, (b) larger than the worker’s most re- cently chosen TID, and (c) in the current global epoch. 4.4 Commit protocol The result is written into each record modified by the We now describe how we run transactions. We first dis- transaction. cuss transactions that only read and update existing keys. The TID order often reflects the serial order, but not We describe how to handle inserts and deletes in §4.5 always. Consider transactions t1 and t2 where t1 hap- and range queries in §4.6. pened first in the serial order. If t1 wrote a tuple that t2 As a worker runs a transaction, it maintains a read-set observed (by reading it or overwriting it), then t2 ’s TID that identifies all records that were read, along with the must be greater than t1 ’s, thanks to (a) above. However, TID of each record at the time it was accessed. For mod- TIDs do not reflect anti-dependencies (write-after-read ified records, it maintains a write-set that stores the new conflicts). If t1 merely observed a tuple that t2 later over- state of the record (but not the previous TID). Records wrote, t1 ’s TID might be either less than or greater than that were both read and modified occur in both the read- t2 ’s! Nevertheless, the TIDs chosen by a worker increase set and the write-set. In normal operation, all records in monotonically and agree with the serial order, the TIDs the write-set also occur in the read-set. assigned to a particular record increase monotonically On transaction completion, a worker attempts to com- and agree with the serial order, and the ordering of TIDs mit using the following protocol (see Figure 2). with different epochs agrees with the serial order. In Phase 1, the worker examines all records in the The lower three bits of each TID word are status bits transaction’s write-set and locks each record by acquir- that are logically separate from the TID itself. Includ- ing the record’s lock bit. To avoid deadlocks, workers ing these bits with the TIDs simplifies concurrency con- lock records in a global order. Any deterministic global trol; for example, Silo can update a record’s version order is fine; Silo uses the pointer addresses of records. and unlock the record in one atomic step. The bits are After all write locks are acquired, the worker takes a lock bit, a latest-version bit, and an absent bit. The a snapshot of the global epoch number using a single lock bit protects record memory from concurrent up- memory access. Fences are required to ensure that this dates; in database terms it is a latch, a short-term lock read goes to main memory (rather than an out-of-date 

6 . Data: read set R, write set W , node set N, Thread 1 Thread 2 global epoch number E t1 ← read(x) t2 ← read(y) // Phase 1 write(y ← t1 + 1) write(x ← t2 + 1) for record, new-value in sorted(W ) do Figure 3: A read-write conflict between transactions. lock(record); compiler-fence(); e ← E; // serialization point of read records, (2) it treats locked records as dirty and compiler-fence(); aborts on encountering them, and (3) the fences that close Phase 1 ensure that TID validation sees all con- // Phase 2 current updates. We argue its serializability by reduction for record, read-tid in R do to strict two-phase locking (S2PL): if our OCC valida- if record.tid = read-tid or not record.latest tion protocol commits, then so would S2PL. To simplify or (record.locked and record ∈ W ) the explanation, assume that the write-set is a subset of then abort(); the read-set. S2PL acquires read locks on all records for node, version in N do read (which, here, includes all records written) and re- if node.version = version then abort(); leases them only after commit. Consider a record in the commit-tid ← generate-tid(R, W , e); read-set. We verify in Phase 2 that the record’s TID has // Phase 3 not changed since its original access, and that the record for record, new-value in W do is not locked by any other transaction. This means that write(record, new-value, commit-tid); S2PL would have been able to obtain a read lock on that unlock(record); record and hold it up to the commit point. For updated Figure 2: Commit protocol run at the end of every records, S2PL can be modeled as upgrading shared read transaction. locks to exclusive write locks at commit time. Our pro- tocol obtains exclusive locks for all written records in Phase 1, then in Phase 2 verifies that this is equivalent to cache), happens logically after all previous memory ac- a read lock that was held since the first access and then cesses, and happens logically before all following mem- upgraded. The fences that close Phase 1 ensure that the ory accesses. On x86 and other TSO (total store order) version checks in Phase 2 access records’ most recent machines, however, these fences are compiler fences that TID words. do not affect compiled instructions; they just prevent the The protocol also ensures that epoch boundaries agree compiler from moving code aggressively. The snapshot with the serial order. Specifically, committed transac- of the global epoch number is the serialization point for tions in earlier epochs never transitively depend on trans- transactions that commit. actions in later epochs. This holds because the memory In Phase 2, the worker examines all the records in the fences ensure that all workers load the latest version of transaction’s read-set (which may contain some records E before read validation and after write locking. Plac- that were both read and written). If some record either ing the load before read validation ensures that com- has a different TID than that observed during execution, mitted transactions’ read- and node-sets never contain is no longer the latest version for its key, or is locked by data from later epochs; placing it after write locking en- a different transaction, the transaction releases its locks sures that all transactions in later epochs would observe and aborts. at least the lock bits acquired in Phase 1. Thus, epochs If the TIDs of all read records are unchanged, the obey both dependencies and anti-dependencies. transaction is allowed to commit, because we know that all its reads are consistent. The worker uses the snapshot For example, consider two records x and y that both of the global epoch number taken in Phase 1 to assign start out with value zero. Given the two transactions the transaction a TID as described in §4.2. shown in Figure 3, the state x = y = 1 is not a valid seri- Finally, in Phase 3, the worker writes its modified alizable outcome. We illustrate why this final state is im- records to the tree and updates their TIDs to the transac- possible in Silo. For it to occur, thread 1 must read x = 0, tion ID computed in the previous phase. Each lock can thread 2 must read y = 0, and both threads must com- be released immediately after its record is written. Silo mit. So assume that thread 1 reads x = 0 and commits, must ensure that the new TID is visible as soon as the which means that thread 1’s Phase 2 validation verified lock is released; this is simple since the TID and lock that x was unlocked and unchanged. Since thread 2 locks share a word and can be written atomically. x as part of its Phase 1, we know then that thread 2’s se- rialization point follows thread 1’s. Thus, thread 2 will Serializability This protocol is serializable because observe either thread 1’s lock on y (which was acquired (1) it locks all written records before validating the TIDs before thread 1’s serialization point) or a new version 

7 .number for y. It will either set y ← 2 or abort. constructed in the absent state and with TID 0, a map- ping from k → r is added to the tree (using an insert- 4.5 Database operations if-absent primitive), and r is added to both the read-set This section provides more detail about how we support and the write-set as if a regular put occurred. The insert- different database operations. if-absent primitive ensures that there is never more than one record for a given key; Phase 2’s read-set validation Reads and writes When a transaction commits, we ensures that no other transactions have superseded the overwrite modified records if possible since this im- placeholder. proves performance. If it is not possible (e.g., because A transaction that does an insert commits in the usual the new record is larger), we create new storage to con- way. If the commit succeeds, the new record is overwrit- tain the new data, mark the old record as no longer being ten with its proper value and the transaction’s TID. If most recent, and modify the tree to point to the new ver- the commit fails, the commit protocol registers the ab- sion. Modification in place means that concurrent read- sent record for future garbage collection. ers might not see consistent record data. The version val- idation protocol described below is used to detect this 4.6 Range queries and phantoms problem. Silo supports range queries that allow a transaction to To modify record data during Phase 3 of the commit access a range of the keys in a table. Range queries are protocol, a worker while holding the lock (a) updates the complicated by the phantom problem [10]: if we scanned record, (b) performs a memory fence, and (c) stores the a particular range but only kept track of the records that TID and releases the lock. The consistency requirement were present during the scan, membership in the range is that if a concurrent reader sees a released lock, it must could change without being detected by the protocol, vi- see both the new data and the new TID. Step (b) ensures olating serializability. that the new data is visible first (on TSO machines this The typical solution to this problem in database sys- is a compiler fence); Step (c) exposes the new TID and tems is next-key locking [24]. Next-key locking, how- released lock atomically since the lock is located in the ever, requires locking for reads, which goes against TID word. Silo’s design philosophy. To access record data during transaction execution Silo deals with this issue by taking advantage of the (outside the commit protocol), a worker (a) reads the underlying B+ -tree’s version number on each leaf node. TID word, spinning until the lock is clear, (b) checks The underlying B+ -tree guarantees that structural modi- whether the record is the latest version, (c) reads the fications to a tree node result in a version number change data, (d) performs a memory fence, and (e) checks the for all nodes involved. A scan on the interval [a, b) TID word again. If the record is not the latest version in therefore works as follows: in addition to registering all step (b) or the TID word changes between steps (a) and records in the interval in the read-set, we also maintain (e), the worker must retry or abort. an additional set, called the node-set. We add the leaf nodes that overlap with the key space [a, b) to the node- Deletes Snapshot transactions require that deleted set along with the version numbers examined during the records stay in the tree: linked versions relevant for snap- scan. Phase 2 checks that the version numbers of all tree shots must remain accessible. A delete operation there- nodes in the node-set have not changed, which ensures fore marks its record as “absent” using the absent bit and that no new keys have been added or removed within the registers the record for later garbage collection. Clients ranges examined. treat absent records like missing keys, but internally, ab- Phantoms are also possible due to lookups and deletes sent records are treated like present records that must that fail because there is no entry for the key in the tree; be validated on read. Since most absent records are reg- in either case the transaction can commit only if the key istered for future garbage collection, Silo writes do not is still not in the tree at commit time. In this case the overwrite absent record data. node that would contain the missing key is added to the node-set. Inserts Phase 2 handles write-write conflicts by re- The careful reader will have noticed an issue with quiring transactions to first lock records. However, when insert operations, which can trigger structural modi- the record does not exist, there is nothing to lock. To fications, and node-set tracking. Indeed, we need to avoid this problem, we insert a new record for the insert distinguish between structural modifications caused by request before starting the commit protocol. concurrent transactions (which must cause aborts) and An insert operation on key k works as follows. If k al- modifications caused by the current transaction (which ready maps to an non-absent record, the insert fails and should not). We fix this problem as follows. Recall that the transaction is aborted. Otherwise, a new record r is the tree modifications for an insertion actually occur be- 

8 .fore commit time. For each tree node n that is affected by point in the serial order, and none from transactions af- the insert (there can be more than one due to splits), let ter that point. Snapshots are used for running snapshot vold be its version number before the insert, and vnew its transactions (and could also be helpful for checkpoint- version number afterwards. The insertion then updates ing). Managing snapshots has two tricky parts: ensuring the node-set: if n is in the node-set with version num- that the snapshot is consistent and complete, and ensur- ber vold , it is changed to vnew , otherwise the transaction ing that its memory is eventually reclaimed. is aborted. This means only other concurrent modifica- We provide consistent snapshots using snapshot tions to n will cause an abort. epochs. Snapshot epoch boundaries align with epoch boundaries, and thus are consistent points in the serial 4.7 Secondary indexes order. However, snapshot epochs advance more slowly To Silo’s commit protocol, secondary indexes are sim- than epochs. (We want to avoid frequent snapshot cre- ply additional tables that map secondary keys to records ation because snapshots are not free.) The snapshot containing primary keys. When a modification affects a epoch snap(e) for epoch e equals k · e/k ; currently secondary index, the index is explicitly changed via ex- k = 25, so a new snapshot is taken about once a second. tra accesses in the code for the transaction that did the The epoch-advancing thread periodically computes a modification. These modifications will cause aborts in global snapshot epoch SE ← snap(E − k). Every worker the usual way: a transaction that used a secondary index w also maintains a local snapshot epoch sew ; at the start will abort if the record it accessed there has changed. of each transaction, it sets sew ← SE. Snapshot trans- 4.8 Garbage collection actions use this value to find the right record versions. Silo transactions generate two sources of garbage that For record r, the relevant version is the most recent one must be reaped: B+ -tree nodes and database records. with epoch ≤ sew . When a snapshot transaction com- Rather than reference count these objects (which would pletes it commits without checking; it never aborts, since require that all accesses write to shared memory), Silo the snapshot is consistent and is never modified. leverages its epochs into an epoch-based reclamation Read/write transactions must not delete or overwrite scheme à la read-copy update (RCU) [11, 13]. record versions relevant for a snapshot. Consider a trans- When a worker generates garbage—for example, by action committing in epoch E. When modifying or delet- deleting a record or by removing a B+ -tree node—it reg- ing a record r in Phase 3, the transaction compares isters the garbage object and its reclamation epoch in a snap(epoch(r.tid)) and snap(E). If these values are the per-core list for that object type. The reclamation epoch same, then it is safe to overwrite the record; the new is the epoch after which no thread could possibly access version would supersede the old one in any snapshot. the object; once it is reached, we can free the object. If the values differ, the old version must be preserved for This could happen either in a separate background task current or future snapshot transactions, so the read/write or in the workers themselves. We do it in the workers transaction installs a new record whose previous-version between requests, which reduces the number of helper pointer links to the old one. Snapshot transactions will threads and can avoid unnecessary data movement be- find the old version via the link. (When possible, we tween cores. actually copy the old version into new memory linked Silo’s snapshot transactions mean that different kinds from the existing record, so as to avoid dirtying tree- of objects have different reclamation policies, though all node cache lines, but the semantics are the same.) are similar. The simplest policy is for B+ -tree nodes. Re- A record version should be freed once no snap- call that every worker w maintains a local epoch number shot transaction will access it. This is tied to snap- ew that is set to E just before the start of each transaction. shot epochs. When a transaction committing in epoch A tree node freed during a transaction is given reclama- E allocates memory for a snapshot version, it regis- tion epoch ew . Thanks to the way epochs advance, no ters that memory for reclamation with epoch snap(E). thread will ever access tree nodes freed during an epoch The epoch-advancing thread periodically computes a e ≤ min ew −1. The epoch-advancing thread periodically snapshot reclamation epoch as min sew − 1. Snapshot checks all ew values and sets a global tree reclamation versions with equal or smaller reclamation epochs can epoch to min ew − 1. Garbage tree nodes with smaller or safely be freed. The reclamation process need not ad- equal reclamation epochs can safely be freed. just any previous-version pointers: the dangling pointer to the old version will never be traversed, since any fu- 4.9 Snapshot transactions ture snapshot transaction will prefer the newer version. We support running read-only transactions on recent- Deletions, however, require special handling. Deleted past snapshots by retaining additional versions for records should be unhooked from the tree, but this can- records. These versions form a consistent snapshot that not happen right away: snapshot transactions must be contains all modifications of transactions up to some able to find the linked older versions. The commit pro- 

9 .cess therefore creates an “absent” record whose status worker publishes its buffer to its corresponding logger bits mark the value as deleted; the space for record data using a per-worker queue, and then publishes its last instead stores the relevant key. This absent record is reg- committed TID by writing to a global variable ctidw . istered for cleanup at reclamation epoch snap(E). When Loggers operate in a continuous loop. At the start of the snapshot reclamation epoch reaches this value, the each iteration, the logger calculates t = min ctidw for its cleanup procedure modifies the tree to remove the ref- workers. From this TID, it computes a local durable erence to the record (which requires the key), and then epoch d = epoch(t) − 1. All its workers have published registers the absent record for reclamation based on the all their transactions with epochs ≤ d, so as far as this tree reclamation epoch (the record cannot be freed right logger is concerned, epochs ≤ d are complete. It appends away since a concurrent transaction might be accessing all the buffers plus a final record containing d to the end it). However, if the absent record was itself superseded of a log file and waits for these writes to complete. (It by a later insert, the tree should not be modified. The never needs to examine the actual log buffer memory.) cleanup procedure checks whether the absent record is It then publishes d to a per-logger global variable dl and still the latest version; if not, it simply ignores the record. returns the buffers back to the workers for recycling. No further action is required, because such a record was A thread periodically computes and publishes a global marked for future reclamation by the inserting transac- durable epoch D = min dl . All transactions with epochs tion. ≤ D are known to have been durably logged. Workers can thus respond to clients whose transactions occurred 4.10 Durability in epochs ≤ D. Durability is a transitive property: a transaction is Silo uses record-level redo logging exclusively, not durable if all its modifications are recorded on durable undo logging or operation logging. Undo logging is not storage and all transactions serialized before it are necessary for our system since we log after transactions durable. Thus, durability requires the system to recover commit; logging at record level simplifies recovery. the effects of a set of transactions that form a prefix of To recover, Silo would read the most recent dl for each the serial order. In Silo, epochs provide the key to re- logger, compute D = min dl , and then replay the logs, covering this prefix, just as they do for snapshot trans- ignoring entries for transactions whose TIDs are from actions. Since epoch boundaries are consistent with the epochs after D. Log records for the same record must be serial order, Silo treats whole epochs as the durability applied in TID order to ensure that the result equals the commit units. The results of transactions in epoch e are latest version, but replaying can otherwise be performed not released to clients until all transactions with epochs concurrently. ≤ e have been stored durably. A full system would recover from a combination of Durable storage could be obtained through replica- logs and checkpoints to support log truncation. Check- tion, but our current implementation uses logging and lo- points could take advantage of snapshots to avoid inter- cal storage (disks). All transactions in epoch e are logged fering with read/write transactions. Though checkpoint- together. After a failure, the system examines the logs ing would reduce worker performance somewhat, it need and finds the durable epoch D, which is the latest epoch not happen frequently. We evaluate common-case log- whose transactions were all successfully logged. It then ging, but have not yet implemented full checkpointing recovers all transactions with epochs ≤ D, and no more. or recovery. Not recovering more is crucial: although epochs as a whole are serially consistent, the serial order within an 5 Evaluation epoch is not recoverable from the information we log, In this section, we evaluate the effectiveness of the tech- so replaying a subset of an epoch’s transactions might niques in Silo, confirming the following performance produce an inconsistent state. This also means the epoch hypotheses: period directly affects the average case latency required • The cost of Silo’s read/write set tracking for a simple for a transaction to commit. key-value workload is low (§5.2). Logging in Silo is handled by a small number of log- • Silo scales as more cores become available, even ger threads, each of which is responsible for a disjoint when transactions are made persistent (§5.3). subset of the workers. Each logger writes to a log file on a separate disk. • Silo’s performance is more robust to workload When a worker commits a transaction, it creates a log changes compared to a partitioned data store, specfi- record consisting of the transaction’s TID and the ta- cially as we increase the level of cross-partition con- ble/key/value information for all modified records. This tention and the skew of the workload (§5.4). log record is stored in a local memory buffer in disk for- • Large read-only transactions benefit substantially mat. When the buffer fills or a new epoch begins, the from Silo’s snapshot transactions (§5.5). This mech- 

10 . anism incurs only a small space overhead, even for a 18M Throughput (txns/sec) 16M write heavy workload (§5.6). 14M Key-Value MemSilo 12M MemSilo+GlobalTID We also show in §5.7 the relative importance of several 10M of Silo’s implementation techniques. 8M 6M 4M 2M 5.1 Experimental setup 0 1 8 16 24 32 All our experiments were run on a single machine with Worker threads four 8-core Intel Xeon E7-4830 processors clocked at Figure 4: Overhead of MemSilo versus Key-Value 2.1GHz, yielding a total of 32 physical cores. Each when running a variant of the YCSB benchmark. core has a private 32KB L1 cache and a private 256KB L2 cache. The eight cores on a single processor share a 24MB L3 cache. We disabled hyperthreading on all 5.2 Overhead of small transactions CPUs; we found slightly worse results with hyperthread- This section shows that Silo’s read/write set tracking has ing enabled. The machine has 256GB of DRAM with low overhead by comparing to the underlying key-value 64GB of DRAM attached to each socket, and runs 64- store, which does not perform any tracking. bit Linux 3.2.0. We evaluate two systems. The first system, Key-Value, In all graphs, each point reported is the median of is simply the concurrent B+ -tree underneath Silo. That three consecutive runs, with the minimum and maxi- is, Key-Value provides only single-key gets and puts. The mum values shown as error bars. In our experiments, we second system is MemSilo. We ran a variant of YCSB follow the direction of Masstree and size both internal workload mix A; YCSB [6] is Yahoo’s popular key- nodes and leaf nodes of our B+ -tree to be roughly four value benchmark. Our variant differs in the following cache lines (a cache line is 64 bytes on our machine), and ways: (a) we fix the read/write ratio to 80/20 (instead of use software prefetching when reading B+ -tree nodes. 50/50), (b) we change write operations to read-modify- writes, which in MemSilo happen in a single transaction, We pay careful attention to memory allocation and and (c) we shrink the size of records to 100 bytes (in- thread affinity in our experiments. We use a custom stead of 1000 bytes). These modifications prevent unin- memory allocator for both B+ -tree nodes and records. teresting overheads that affect both systems from hiding Our allocator takes advantage of 2MB “superpages” (a overheads that affect only MemSilo. More specifically, feature supported in recent Linux kernels) to reduce TLB (a) prevents the memory allocator for new records from pressure. We pin threads to cores so that the memory becoming the primary bottleneck, (b) creates a transac- allocated by a particular thread resides on that thread’s tion that actually generates read-write conflicts, which NUMA node. Before we run an experiment, we make stresses MemSilo’s protocol, and (c) prevents memcpy sure to pre-fault our memory pools so that the scalabil- from becoming the primary bottleneck. Both transac- ity bottlenecks of Linux’s virtual memory system [5] are tions sample keys uniformly from the key space. We fix not an issue in our benchmarks. the tree size to contain 160M keys, and vary the number For experiments which run with persistence enabled, of workers performing transactions against the tree. we use four logger threads and assign each logger a file Figure 4 shows the results of running the YCSB residing on a separate device. Three loggers are assigned benchmark on both systems. The overhead of MemSilo different Fusion IO ioDrive2 devices, and the fourth is compared to Key-Value is negligible; Key-Value outper- assigned to six 7200RPM SATA drives in a RAID-5 con- forms MemSilo by a maximum of 1.07×. figuration. Collectively, these devices provide enough bandwidth that writing to disk is not a bottleneck. Globally generated TIDs Figure 4 also quantifies Our experiments do not use networked clients. We the benefits of designing Silo to avoid a single glob- would expect networking to reduce our throughput ally unique TID assigned at commit time. Mem- somewhat; in Masstree, networked clients reduced Silo+GlobalTID follows an identical commit protocol throughput by 23% [23]. In our experiments, each thread as Silo, except it generates TIDs from a single shared combines a database worker with a workload generator. counter. This is similar to the critical section found These threads run within the same process, and share in Larson et al. [19] Here we see a scalability col- Silo trees in the same address space. We run each exper- lapse after 24 workers; Key-Value outperforms Mem- iment for 60 seconds. When durability is not an issue, Silo+GlobalTID by 1.80× at 32 workers. This demon- our experiments use MemSilo, which is Silo with log- strates the necessity of avoiding even a single global ging disabled. atomic instruction during the commit phase. 

11 .Throughput (txns/sec) 0.9M 800 0.8M 700 0.7M MemSilo Silo 600 Latency (ms) Silo Silo+tmpfs 0.6M 500 0.5M 400 0.4M 0.3M 300 0.2M 200 0.1M 100 0 0 1 8 16 24 32 1 8 16 24 32 Worker threads Worker threads Figure 5: Throughput of Silo when running the TPC- Figure 7: Silo transaction latency for TPC-C with C benchmark, including persistence. logging to either durable storage or an in-memory file system. Throughput/core (txns/sec/core) 30K 8 workers. This drop happens because of several fac- 25K tors, including increasing database size and sharing of 20K resources such as the L3 cache (particularly important 15K from one to 8 threads) as well as actual contention. Silo 10K performs slightly slower, and its scalability is degraded MemSilo by logging past 28 workers, since worker threads and the 5K Silo four logger threads start to contend for cores. 0 1 8 16 24 32 We also ran experiments to separate the impact of Worker threads Silo’s logging subsystem from that of our durable stor- age devices. With Silo’s loggers writing to an in-memory Figure 6: Per-core throughput of Silo when running file system instead of stable storage (Silo+tmpfs), we ob- the TPC-C benchmark. served at most a 1.03× gain in throughput over Silo. This argues that most of the loss in throughput when en- 5.3 Scalability and persistence abling durability is due to the overhead of transferring Our next experiment shows that Silo scales effectively log records from worker threads to logger threads, rather as we increase the number of workers. We use the popu- than the overhead of writing to physical hardware (given lar TPC-C benchmark [33], which models a retail oper- sufficient hardware bandwidth). ation and is a common benchmark for OLTP workloads. Figure 7 shows transaction latency for this experiment Transactions in TPC-C involve customers assigned to (the latency of MemSilo is negligible and therefore not a set of districts within a local warehouse, placing or- shown). For both Silo+tmpfs and Silo, we see a spike in ders in those districts. Most orders can be fulfilled en- latency around 28 workers due to worker/logger thread tirely from the local warehouse, but a small fraction re- contention. The effect is more pronounced with Silo, il- quest a product from a remote warehouse. To run TPC-C lustrating the fact that latency is more sensitive to real on Silo, we assign all clients with the same local ware- hardware than throughput is. house to the same thread. This models client affinity, and Overall, we see that logging does not significantly de- means memory accessed by a transaction usually resides grade the throughput of Silo and incurs only a modest on the same NUMA node as the worker. We size the increase in latency. MemSilo outperforms Silo by only a database such that the number of warehouses equals the maximum of 1.16× at 32 workers. number of workers, so the database grows as more work- ers are introduced (in other words, we fix the contention 5.4 Comparison with Partitioned-Store ratio of the workload). We do not model client “think” This section describes our evaluation of Silo versus a time, and we run the standard workload mix involving all statically partitioned data store, which is a common con- five transactions. To show how durability affects scala- figuration for running OLTP on a single shared-memory bility, we run both MemSilo and Silo. node [25, 30, 32]. Most of the time Silo is preferable. Figures 5 and 6 show the throughput of running TPC- Tables in TPC-C are typically partitioned on C as we increase the number of workers (and thus ware- warehouse-id, such that each partition is responsible for houses) in both MemSilo and Silo. Scaling is close to lin- the districts, customers, and stock levels for a particu- ear for MemSilo up to 32 workers. The per-core through- lar warehouse. This is a natural partitioning, because as put of MemSilo at 32 workers is 81% of the throughput mentioned in §5.3, each transaction in TPC-C is cen- at one worker, and 98% of the per-core throughput at tered around a single local warehouse. 

12 .Throughput (txns/sec) 1.4M 0.8M Throughput (txns/sec) 1.2M 0.7M MemSilo+FastIds 1M 0.6M MemSilo 0.8M 0.5M Partitioned-Store 0.4M 0.6M 0.3M Partitioned-Store 0.4M MemSilo+Split 0.2M 0.2M MemSilo 0.1M 0 0 0 10 20 30 40 50 60 1 8 16 24 32 % cross-partition transactions Worker threads Figure 8: Performance of Partitioned-Store versus Figure 9: Performance of Partitioned-Store versus MemSilo as the percentage of cross-partition trans- MemSilo as the number of workers processing the actions is varied. same size database is varied. Cross-partition transactions We first fix the database size and number of workers, and vary the percentage of The design of Partitioned-Store is motivated by H- cross-partition transactions. Store/VoltDB [32]. We physically partition the data by warehouse so that each partition has a separate set of As in our previous experiments on TPC-C, Silo exe- B+ -trees for each table, and we associate each parti- cutes transactions by associating each worker with a lo- tion with a single worker. The workers are all in the cal warehouse and issuing queries against the shared B+ - same process to avoid IPC overhead. We then associate trees. We focus on the most frequent transaction in TPC- a global partition lock with each partition. Every trans- C, the new-order transaction. Even though the new-order action first obtains all partition locks (in sorted order). transaction is bound to a local warehouse, each transac- Once all locks are obtained, the transaction can pro- tion has some probability, which we vary, of writing to ceed as in a single-threaded data store, without the need records in non-local warehouses (these records are in the for additional validation. We assume that we have per- stock table). Our benchmark explores the tradeoffs be- fect knowledge of the partition locks needed by each tween Partitioned-Store and various versions of Silo as transaction, so once the locks are obtained the trans- we increase the probability of a cross-partition transac- action is guaranteed to commit and never has to ac- tion from zero to over 60 percent. quire new locks. We implement these partition locks us- Figure 8 shows the throughput of running all new- ing spinlocks, and take extra precaution to allocate the order transactions on various versions of Silo and memory for the locks on separate cache lines to pre- Partitioned-Store as we increase the number of cross- vent false sharing. When single-partition transactions partition transactions. Both the number of warehouses are common, these spinlocks are cached by their local and the number of workers are fixed at 28. The parameter threads and obtaining them is inexpensive. We believe varied is the probability that a single item is drawn from this scheme performs at least as well on a multicore ma- a remote warehouse. On the x-axis we plot the proba- chine as the cross-partition locking schemes described bility that any given transaction will touch at least one for H-Store. Partitioned-Store does not support snap- remote warehouse (each new-order transaction includes shot transactions, so it incurs no overhead for maintain- between 5 to 15 items, inclusive) and thus require grab- ing multiple record versions. Partitioned-Store uses the bing more than one partition lock in Partitioned-Store. same B+ -trees that Key-Value and Silo use, except we The curves in Figure 8 show that Partitioned-Store is remove the concurrency control mechanisms in place in clearly the optimal solution for perfectly partitionable the B+ -tree. Additionally, we remove the concurrency workloads. The advantages are two-fold: no concurrency control for record values (which Key-Value needed to control mechanisms are required, and there is better ensure atomic reads/writes to a single key). Partitioned- cache locality due to the partitioned trees being smaller. Store does not implement durability. Partitioned-Store outperforms MemSilo by 1.54× at no cross-partition transactions. However, performance suf- We evaluate the costs and benefits of partitioning by fers as soon as cross-partition transactions are intro- comparing a partitioned version of Silo (Partitioned- duced. At roughly 20%, the throughput of Partitioned- Store) with Silo itself (MemSilo). We run two separate Store drops below MemSilo and gets worse with increas- experiments: the first varies the percentage of cross- ing contention, whereas MemSilo maintains relatively partition transactions in the workload, and the sec- steady throughput for the entire experiment. At roughly ond varies the number of concurrent workers executing 60% cross-partition transactions, MemSilo outperforms transactions against a fixed-size database. Partitioned-Store by 2.98×. The results here can be un- 

13 .derstood as the tradeoff between coarse and fine-grained Transactions/sec Aborts/sec locking. While Silo’s OCC protocol initially pays a non- MemSilo 200,252 2,570 trivial overhead for tracking record-level changes in low MemSilo+NoSS 168,062 15,756 contention regimes, this work pays off as the contention Figure 10: Evaluation of Silo’s snapshot transactions increases. on a modified TPC-C benchmark. To understand the overhead in more detail, we intro- duce MemSilo+Split, which extends MemSilo by phys- loads where the frequency of large read-only transac- ically splitting its tables the same way as Partitioned- tions is low, such as the standard TPC-C workload, the Store does. This yields a 13% gain over MemSilo. The benefits of snapshots do not outweigh their overhead. remaining difference is due to Partitioned-Store requir- However, for other workloads involving many large ing no fine-grained concurrency control. read-only transactions over frequently updated records, Skewed workloads This next experiment shows the such as the one evaluated in this section, we show that impact of workload skew, or “hotspots,” on performance snapshots provide performance gains. in Partitioned-Store and Silo. We run a 100% new-order We again use the TPC-C benchmark, changing the workload mix, fixing the database size to four ware- setup as follows. We fix the number of warehouses to 8 houses in a single partition. We then vary the number and the number of workers to 16 (each warehouse is as- of available workers, which simulates increased work- signed to two workers). We run a transaction mix of 50% load skew (more workers processing transactions over a new-order and 50% stock-level; this is the largest of two fixed size database). Figure 9 shows the results of this read-only queries from TPC-C. On average, stock-level experiment. touches several hundred records in tables frequently up- For Partitioned-Store, the throughput stays constant dated by the new-order transaction, performing a nested- as we increase workers, because multiple workers can- loop join between the order-line table and the stock ta- not execute in parallel on a single partition (they serial- ble. ize around the partition lock). For MemSilo, throughput We measure the throughput of Silo under this load increases, but not linearly, because of actual workload in two scenarios: one where we use Silo’s snapshot contention when running 100% new-order transactions. transactions to execute the stock-level transaction at Specifically, a counter record per unique (warehouse-id, roughly one second in the past (MemSilo), and one district-id) pair is used to generate new-order IDs. As where we execute stock-level as a regular transaction we add more workers, we increase the number of read- in the present (MemSilo+NoSS). In both scenarios, the write conflicts over this shared counter, which increases new-order transaction executes as usual. the number of aborted transactions. Note that this is not Figure 10 shows the results of this experiment. Mem- unique to OCC; in 2PL, conflicting workers would seri- Silo outperforms MemSilo+NoSS by 1.19×. This is due alize around write locks held on the shared counter. In to the larger number of aborts that occur when a large Figure 9, we see that MemSilo outperforms Partitioned- read-only transaction executes with concurrent modifi- Store by a maximum of 8.20× at 24 workers. cations (recall that snapshot transactions never abort). Because the contention in this workload is a property 5.6 Space overhead of snapshots of the workload itself and not so much the system, it is interesting to see how MemSilo responds if the con- This section shows that the space overhead of main- tention is removed. We substantially reduce this con- taining multiple versions of records to support snapshot tention in MemSilo+FastIds by generating IDs outside transactions is low, even for update heavy workloads. the new-order transaction. The new-order client request We run a variant of YCSB where every transaction is a runs two transactions, where the first generates a unique read-modify-write operation on a single record. We pick ID and the second uses that ID. (This sacrifices the in- this operation because each transaction has a very high variant that the new-order ID space is contiguous, since probability of generating a new version of a record (since counters do not roll back on abort.) The result is through- we size the tree to contain 160M keys). This stresses put scaling nicely until 28 workers, when the next our garbage collector (and memory allocator), because contention bottleneck in the workload appears. Over- at every snapshot epoch boundary a large number of old all MemSilo+FastIds outperforms Partitioned-Store by records need to be garbage collected. As in §5.2, we a maximum of 17.21× at 32 workers. use 100 byte records and uniformly sample from the key space. We run MemSilo with 28 workers. 5.5 Effectiveness of snapshot transactions In this experiment, the database records initially oc- In this section we show that Silo’s snapshot transactions cupied 19.5GB of memory. Throughout the entire run, are effective for certain challenging workloads. Main- this increased by only a maximum of 672.3MB, repre- taining snapshots is not free (see §5.7); for some work- senting a 3.4% increase in memory for snapshots. This 

14 . The takeaways from Figure 11 are that spending extra Regular CPU cycles to reduce the amount of bytes written to disk Throughput (txns/sec) Persistence 800K 1.79 1.84 1.62 1.00 0.98 surprisingly does not pay off for this TPC-C workload, 0.90 0.85 and that the overhead of copying record modifications 600K 1.21 1.00 into logger buffers and then into persistent storage is low. 400K 200K 6 Conclusions 0 We have presented Silo, a new OCC-based serializable database storage engine designed for excellent perfor- Si +A +O ator +N rite +N aps M +S +F +C m em m ul cs om llo ve oS oG hot pl al lR mance at scale on large multicore machines. Silo’s con- Si rw c n e C pr lR ec lo es e s s currency control protocol is optimized for multicore per- s s Figure 11: A factor analysis for Silo. Changes are formance, avoiding global critical sections and non-local added left to right for each group, and are cumula- memory writes for read operations. Its use of epochs fa- tive. cilitates serializable recovery, garbage collection, and ef- ficient read-only snapshot transactions. Our results show shows that, despite having to keep old versions around, Silo’s near-linear scalability on YCSB-A and TPC-C, space overhead is reasonable and garbage collection is very high transaction throughput on TPC-C, and low very effective at reaping old versions. overheads relative to a non-transactional system. To- gether, these results show that transactional consistency 5.7 Factor analysis and scalability can coexist at high performance levels in To understand in greater detail the overheads and bene- shared-memory databases. fits of various aspects of Silo, we show a factor analysis Given the encouraging performance of Silo, we are in- in Figure 11 that highlights the key factors for perfor- vestigating a number of areas for future work, designed mance. The workload here is the standard TPC-C mix to further improve its performance and augment its us- running with 28 warehouses and 28 workers. We per- ability. These include further exploiting “soft partition- form analysis in two groups of cumulative changes: the ing” techniques to improve performance on highly parti- Regular group deals with changes that do not affect the tionable workloads; fully implementing checkpointing, persistence subsystem, and the Persistence group deals recovery, and replication; investigating how to balance only with changes in the persistence layer. load across workers most effectively, perhaps in a way In the Regular group, Simple refers to Silo running that exploits core-to-data affinity; and integrating into a with no NUMA aware memory allocator and allocat- full-featured SQL engine. ing a new record for each write. +Allocator adds the NUMA aware allocator described in §5.1. +Overwrites Acknowledgements allows Silo to write record modifications in place when We thank our shepherd Timothy Roscoe and the anony- possible (and is equivalent to MemSilo as presented in mous reviewers for their feedback. Eugene Wu provided Figure 5). +NoSnapshots disables keeping multiple ver- helpful comments on an earlier draft of the paper. We are sions of records for snapshots, and does not provide also very grateful to Yandong Mao, who generously ran any snapshot query mechanism. +NoGC disables the benchmarking experiments on other systems. This work garbage collector described in §4.8. was supported by NSF Grants 1065219 and 0704424. The two major takeaways from Figure 11 are that per- Eddie Kohler’s work was partially supported by a Sloan forming in-place updates is an important optimization Research Fellowship and a Microsoft Research New for Silo and that the overhead of maintaining snapshots Faculty Fellowship. plus garbage collection is low. For persistence analysis, we break down the follow- References ing factors: MemSilo refers to running without logging. [1] D. Agrawal, A. J. Bernstein, P. Gupta, and S. Sengupta. +SmallRecs enables logging, but only writes log records Distributed optimistic concurrency control with reduced that are eight bytes long (containing a TID only, omitting rollback. Distributed Computing, 2(1), 1987. record modifications); this illustrates an upper bound on performance for any logging scheme. +FullRecs is Silo [2] H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O’Neil, running with persistence enabled (and is equivalent to and P. O’Neil. A critique of ANSI SQL isolation levels. Silo in Figure 5). +Compress uses LZ4 compression1 to In SIGMOD, 2005. compress log records before writing to disk. [3] N. G. Bronson, J. Casper, H. Chafi, and K. Olukotun. A 1 http://code.google.com/p/lz4/ practical concurrent binary search tree. In PPoPP, 2010. 

15 . [4] S. K. Cha, S. Hwang, K. Kim, and K. Kwon. Cache- [19] P.-Å. Larson, S. Blanas, C. Diaconu, C. Freedman, J. M. conscious concurrency control of main-memory indexes Patel, and M. Zwilling. High-performance concurrency on shared-memory multiprocessor systems. In VLDB, control mechanisms for main-memory databases. Proc. 2001. VLDB Endow., 5(4), 2011. [5] A. T. Clements, M. F. Kaashoek, and N. Zeldovich. [20] P. L. Lehman and S. B. Yao. Efficient locking for con- RadixVM: Scalable address spaces for multithreaded ap- current operations on B-trees. ACM TODS, 6(4), 1981. plications. In Eurosys, 2013. [21] J. Levandoski, D. Lomet, and S. Sengupta. The Bw-tree: [6] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, A B-tree for new hardware. In ICDE, 2013. and R. Sears. Benchmarking cloud serving systems with YCSB. In SoCC, 2010. [22] J. Levandoski, D. Lomet, and S. Sengupta. LLAMA: A cache/storage subsystem for modern hardware. Proc. [7] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, VLDB Endow., 6(10), 2013. J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, [23] Y. Mao, E. Kohler, and R. Morris. Cache craftiness for A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quin- fast multicore key-value storage. In Eurosys, 2012. lan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, [24] C. Mohan. ARIES/KVL: A key-value locking method for R. Wang, and D. Woodford. Spanner: Google’s globally- concurrency control of multiaction transactions operating distributed database. In OSDI, 2012. on B-tree indexes. In VDLB, 1990. [8] C. Diaconu, C. Freedman, E. Ismert, P.-A. Larson, P. Mit- [25] I. Pandis, R. Johnson, N. Hardavellas, and A. Ailamaki. tal, R. Stonecipher, N. Verma, and M. Zwilling. Hekaton: Data-oriented transaction execution. Proc. VLDB En- SQL Server’s memory-optimized OLTP engine. In SIG- dow., 3(1-2), 2010. MOD, 2013. [26] I. Pandis, P. Tözün, R. Johnson, and A. Ailamaki. PLP: [9] D. Dice, O. Shalev, and N. Shavit. Transactional locking page latch-free shared-everything OLTP. Proc. VLDB II. In DISC, 2006. Endow., 4(10), 2011. [10] K. P. Eswaran, J. N. Gray, R. A. Lorie, and I. L. Traiger. [27] A. Pavlo, C. Curino, and S. Zdonik. Skew-aware au- The notions of consistency and predicate locks in a tomatic database partitioning in shared-nothing, parallel database system. Commun. ACM, 19(11), 1976. OLTP systems. In SIGMOD, 2012. [11] K. Fraser. Practical lock freedom. PhD thesis, Cam- [28] D. Porobic, I. Pandis, M. Branco, P. Tözün, and A. Aila- bridge University Computer Laboratory, 2003. URL maki. OLTP on hardware islands. Proc. VLDB Endow., http://www.cl.cam.ac.uk/users/kaf24/lockfree.html. 5(11), 2012. [12] T. Härder. Observations on optimistic concurrency con- [29] K. Ren, A. Thomson, and D. J. Abadi. Lightweight lock- trol schemes. Inf. Syst., 9(2), 1984. ing for main memory database systems. Proc. VLDB En- [13] T. E. Hart, P. E. McKenney, A. D. Brown, and J. Walpole. dow., 6(2), 2012. Performance of memory reclamation for lockless syn- [30] T.-I. Salomie, I. E. Subasu, J. Giceva, and G. Alonso. chronization. J. Parallel Distrib. Comput., 67(12), 2007. Database engines on multicores, why parallelize when [14] T. Horikawa. Latch-free data structures for DBMS: de- you can distribute? In Eurosys, 2011. sign, implementation, and evaluation. In SIGMOD, 2013. [31] J. Sewall, J. Chhugani, C. Kim, N. Satish, and P. Dubey. [15] R. Johnson, I. Pandis, N. Hardavellas, A. Ailamaki, and PALM: Parallel architecture-friendly latch-free modifica- B. Falsafi. Shore-MT: a scalable storage manager for the tions to B+ trees on many-core processors. Proc. VLDB multicore era. In EDBT, 2009. Endow., 4(11), 2011. [16] E. P. Jones, D. J. Abadi, and S. Madden. Low over- [32] M. Stonebraker, S. Madden, D. J. Abadi, S. Harizopou- head concurrency control for partitioned main memory los, N. Hachem, and P. Helland. The end of an archi- databases. In SIGMOD, 2010. tectural era: (it’s time for a complete rewrite). In VLDB, 2007. [17] H. Jung, H. Han, A. D. Fekete, G. Heiser, and H. Y. Yeom. A scalable lock manager for multicores. In SIG- [33] The Transaction Processing Council. TPC-C Benchmark MOD, 2013. (Revision 5.9.0). http://www.tpc.org/tpcc/, June 2007. [18] H. T. Kung and J. T. Robinson. On optimistic methods [34] R. Unland. Optimistic concurrency control revisited. for concurrency control. ACM TODS, 6(2), 1981. Technical report, University of Münster, Department of Information Systems, 1994.