ARIES：使用预写日志记录支持精细粒度锁定和部分回滚的事务恢复方法

1 .ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging C. MOHAN IBM Almaden Research Center and DON HADERLE IBM Santa Teresa Laboratory and BRUCE LINDSAY, HAMID PIRAHESH and PETER SCHWARZ IBM Almaden Research Center In this paper we present a simple and efficient method, called ARIES ( Algorithm for Recouery and Isolation Exploiting Semantics), which supports partial rollbacks of transactions, fine- granularity (e. g., record) locking and recovery using write-ahead logging (WAL). We introduce the paradigm of repeating history to redo all missing updates before performing the rollbacks of the loser transactions during restart after a system failure. ARIES uses a log sequence number in each page to correlate the state of a page with respect to logged updates of that page. All updates of a transaction are logged, including those performed during rollbacks. By appropriate chaining of the log records written during rollbacks to those written during forward progress, a bounded amount of logging is ensured during rollbacks even in the face of repeated failures during restart or of nested rollbacks We deal with a variety of features that are very Important in building and operating an industrial-strength transaction processing system ARIES supports fuzzy checkpoints, selective and deferred restart, fuzzy image copies, media recovery, and high concurrency lock modes (e. g., increment /decrement) which exploit the semantics of the opera- tions and require the ability to perform operation logging. ARIES is flexible with respect to the kinds of buffer management policies that can be implemented. It supports objects of varying length efficiently. By enabling parallelism during restart, page-oriented redo, and logical undo, it enhances concurrency and performance. We show why some of the System R paradigms for logging and recovery, which were based on the shadow page technique, need to be changed in the context of WAL. We compare ARIES to the WAL-based recovery methods of Authors’ addresses: C Mohan, Data Base Technology Institute, IBM Almaden Research Center, San Jose, CA 95120; D. Haderle, Data Base Technology Institute, IBM Santa Teresa Labora- tory, San Jose, CA 95150; B. Lindsay, H. Pirahesh, and P. Schwarz, IBM Almaden Research Center, San Jose, CA 95120. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. @ 1992 0362-5915/92/0300-0094 $1.50 ACM Transactions on Database Systems, Vol 17, No. 1, March 1992, Pages 94-162 

2 . ARIES: A Transaction Recovery Method . 95 DB2TM, IMS, and TandemTM systems. ARIES is applicable not only to database management systems but also to persistent object-oriented languages, recoverable file systems and transaction-based operating systems. ARIES has been implemented, to varying degrees, in IBM’s OS/2TM Extended Edition Database Manager, DB2, Workstation Data Save Facility/VM, Starburst and QuickSilver, and in the University of Wisconsin’s EXODUS and Gamma database machine. Categories and Subject Descriptors: D.4.5 [Operating Systems]: Reliability–backup proce- dures, checkpoint/ restart, fault tolerance; E.5. [Data]: Files– backup/ recouery; H.2.2 [Database Management]: Physical Design–reco~ery and restart; H.2.4 [Database Management]: Sys- tems—concurrency, transaction processing; H.2.7 [Database Management]: Database Adminis- tration—logging and recovery General Terms: Algorithms, Designj Performance, Reliability Additional Key Words and Phrases: Buffer management, latching, locking, space management, write-ahead logging 1. INTRODUCTION In this section, first we introduce some basic concepts relating to recov- ery, concurrency control, and buffer management, and then we outline the organization of the rest of the paper. 1.1 Logging, Failures, and Recovery Methods The transaction concept, which is well understood by now, has been around for a long time. It encapsulates the ACID (Atomicity, Consistency, Isolation and Durability) properties [361. The application of the transaction concept is not limited to the database area [6, 17, 22, 23, 30, 39, 40, 51, 74, 88, 90, 1011. Guaranteeing the atomicity and durability of transactions, in the face of concurrent execution of multiple transactions and various failures, is a very important problem in transaction processing. While many methods have been developed in the past to deal with this problem, the assumptions, performance characteristics, and the complexity and ad hoc nature of such methods have not always been acceptable. Solutions to this problem may be judged using several metrics: degree of concurrency supported within a page and across pages, complexity of the resulting logic, space overhead on non- volatile storage and in memory for data and the log, overhead in terms of the number of synchronous and asynchronous 1/0s required during restart recov- ery and normal processing, kinds of functionality supported (partial transac- tion rollbacks, etc.), amount of processing performed during restart recovery, degree of concurrent processing supported during restart recovery, extent of system-induced transaction rollbacks caused by deadlocks, restrictions placed ‘M AS/400, DB2, IBM, and 0S/2 are trademarks of the International Business Machines Corp. Encompass, NonStop SQL and Tandem are trademarks of Tandem Computers, Inc. DEC, VAX DBMS, VAX and Rdb/VMS are trademarks of Digital Equipment Corp. Informix is a registered trademark of Informix Software, Inc. ACM Transactions on Database Systems, Vol. 17, No 1, March 1992. 

3 .96 . C. Mohan et al on stored data (e. g., requiring unique keys for all records, restricting maxi- mum size of objects to the page size, etc.), ability to support novel lock modes which allow the concurrent execution, based on commutativity and other properties [2, 26, 38, 45, 88, 891, of operations like increment/decrement on the same data by different transactions, and so on. In this paper we introduce a new recovery method, called ARL?LSl (Algorithm for Recovery and Isolation Exploiting Semantics), which fares very well with respect to all these metrics. It also provides a great deal of flexibility to take advantage of some special characteristics of a class of applications for better performance (e. g., the kinds of applications that IMS Fast Path [28, 421 supports efficiently). To meet transaction and data recovery guarantees, ARIES records in a log the progress of a transaction, and its actions which cause changes to recover- able data objects. The log becomes the source for ensuring either that the transaction’s committed actions are reflected in the database despite various types of failures, or that its uncommitted actions are undone (i.e., rolled back). When the logged actions reflect data object content, then those log records also become the source for reconstruction of damaged or lost data (i.e., media recovery). Conceptually, the log can be thought of as an ever growing sequential file. In the actual implementation, multiple physical files may be used in a serial fashion to ease the job of archiving log records [151. Every log record is assigned a unique log sequence number (LSN) when that record is appended to the log. The LSNS are assigned in ascending sequence. Typically, they are the logical addresses of the corresponding log records. At times, version numbers or timestamps are also used as LSNS [6’71. If more than one log is used for storing the log records relating to different pieces of data, then a form of two-phase commit protocol (e. g., the current industry- standard Presumed Abort protocol [63, 641) must be used. The nonvolatile version of the log is stored on what is generally called stable storage. Stable storage means nonvolatile storage which remains intact and available across system failures. Disk is an example of nonvolatile storage and its stability is generally improved by maintaining synchronously two identical copies of the log on different devices. We would expect the online log records stored on direct access storage devices to be archived to a cheaper and slower medium like tape at regular intervals. The archived log records may be discarded once the appropriate image copies (archive dumps) of the database have been produced and those log records are no longer needed for media recovery. Whenever log records are written, they are placed first only in the volatile storage (i.e., virtual storage) buffers of the log file. Only at certain times (e.g., at commit time) are the log records up to a certain point (LSN) written, in log page sequence, to stable storage. This is called forcing the log up to that LSN. Besides forces caused by transaction and buffer manager activi - 1 The choice of the name ARIES, besides its use as an acronym that describes certain features of our recovery method, is also supposed to convey the relationship of our work to the Starburst project at IBM, since Aries is the name of a constellation. ACM TransactIons on Database Systems, Vol. 17, No 1, March 1992 

4 . ARIES: A Transaction Recovery Method . 97 ties, a system process may, in the background, periodically force the log buffers as they fill up. For ease of exposition, we assume that each log record describes the update performed to only a single page. This is not a requirement of ARIES. In fact, in the Starburst [87] implementation of ARIES, sometimes a single log record might be written to describe updates to two pages. The undo (respectively, redo) portion of a log record provides information on how to undo (respec- tively, redo) changes performed by the transaction. A log record which contains both the undo and the redo information is called an undo-redo log record. Sometimes, a log record may be written to contain only the redo information or only the undo information. Such a record is called a redo-only log record or an undo-only log record, respectively. Depending on the action that is performed, the undo-redo information may be recorded physically (e.g., before the update and after the update images or values of specific fields within the object) or operationally (e.g., add 5 to field 3 of record 15, subtract 3 from field 4 of record 10). Operation logging permits the use of high concurrency lock modes, which exploit the semantics of the operations performed on the data. For example, with certain operations, the same field of a record could have uncommitted updates of many transactions. These permit more concurrency than what is permitted by the strict executions property of the model of [3], which essentially says that modified objects must be locked exclusively (X mode) for commit duration. ARIES uses the widely accepted write ahead logging (WAL) protocol. Some of the commercial and prototype systems based on WAL are IBM’s AS/400TM [9, 211, CMU’S Camelot [23, 901, IBM’s DB2TM [1, 10,11,12,13,14,15,19, 35, 961, Unisys’s DMS/1100 [271, Tandem’s EncompassTM [4, 371, IBM’s IMS [42, 43, 53, 76, 80, 941, Informix’s Informix-Turbo m [161, Honeywell’s MRDS [911, Tandem’s NonStop SQL ‘M [95], MCC’S ORION [29], IBM’s 0S/2 Extended EditionTM Database Manager [71, IBM’s QuickSilver [40], IBM’s Starburst [871, SYNAPSE [781, IBM’s System/38 [99], and DEC’S VAX DBMSTM and VAX Rdb/VMSTM [811. In WAL-based systems, an updated page is written back to the same nonvolatile storage location from where it was read. That is, in-place updating is performed on nonvolatile storage. Contrast this with what happens in the shadow page technique which is used in systems such as System R [311 and SQL/DS [51 and which is illustrated in Figure 1. There the updated version of the page is written to a different location on nonvolatile storage and the previous version of the page is used for performing database recovery if the system were to fail before the next checkpoint. The WAL protocol asserts that the log records representing changes to some data must already be on stable storage before the changed data is allowed to replace the previous version of that data on nonvolatile storage. That is, the system is not allowed to write an updated page to the nonvolatile storage version of the database until at least the undo portions of the log records which describe the updates to the page have been written to stable storage. To enable the enforcement of this protocol, systems using the WAL method of recovery store in every page the LSN of the log record that describes the most recent update performed on that page. The reader is ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

5 .98 . C Mohan et al. Page Map ~ Fig. 1. Shadow page technique. Logical page LPI IS read from physical page PI and after modlflcat!on IS wr!tten to physical page PI’ P1’ IS the current vers!on and PI IS the shadow version During a checkpoint, the shadow version IS d]scarded and the current version becomes the shadow verson also On a failure, data base recovety IS performed us!ng the log and the shadow version of the data base referred to [31, 971 for discussions about why the WAL technique is consid- ered to be better than the shadow page technique. [16, 781 discuss methods in which shadowing is performed using a separate log. While these avoid some of the problems of the original shadow page approach, they still retain some of the important drawbacks and they introduce some new ones. Similar comments apply to the methods suggested in [82, 881. Later, in Section 10, we show why some of the recovery paradigms of System R, which were based on the shadow page technique, are inappropriate in the WAL context, when we need support for high levels of concurrency and various other features that are described in Section 2. Transaction status is also stored in the log and no transaction can be considered complete until its committed status and all its log data are safely recorded on stable storage by forcing the log up to the transaction’s commit log record’s LSN. This allows a restart recovery procedure to recover any transactions that completed successfully but whose updated pages were not physically written to nonvolatile storage before the failure of the system. This means that a transaction is not permitted to complete its commit processing (see [63, 64]) until the redo portions of all log records of that transaction have been written to stable storage. We deal with three types of failures: transaction or process, system, and media or device. When a transaction or process failure occurs, typically the transaction would be in such a state that its updates would have to be undone. It is possible that the transaction had corrupted some pages in the buffer pool if it was in the middle of performing some updates when the process disappeared. When a system failure occurs, typically the virtual storage contents would be lost and the transaction system would have to be restarted and recovery performed using the nonvolatile storage versions of the database and the log. When a media or device failure occurs, typically the contents of that media would be lost and the lost data would have to be recovered using an image copy (archive dump) version of the lost data and the log. Forward processing refers to the updates performed when the system is in normal (i. e., not restart recovery) processing and the transaction is updating ACM TransactIons on Database Systems, Vol 17, No. 1, March 1992. 

6 . ARIES: A Transaction Recovery Method . 99 the database because of the data manipulation (e.g., SQL) calls issued by the user or the application program. That is, the transaction is not rolling back and using the log to generate the (undo) update calls. Partial rollback refers to the ability to set up savepoints during the execution of a transaction and later in the transaction request the rolling back of the changes performed by the transaction since the establishment of a previous savepoint [1, 31]. This is to be contrasted with total rollback in which all updates of the transaction are undone and the transaction is terminated. Whether or not the savepoint concept is exposed at the application level is immaterial to us since this paper deals only with database recovery. A nested rollback is said to have taken place if a partial rollback were to be later followed by a total rollback or another partial rollback whose point of termination is an earlier point in the transaction than the point of termination of the first rollback. Normal undo refers to total or partial transaction rollback when the system is in normal operation. A normal undo may be caused by a transaction request to rollback or it may be system initiated because of deadlocks or errors (e. g., integrity constraint violations). Restart undo refers to transaction rollback during restart recovery after a system failure. To make partial or total rollback efficient and also to make debugging easier, all the log records written by a transaction are linked via the PreuLSN field of the log records in reverse chronological order. That is, the most recently written log record of the transaction would point to the previous most recent log record written by that transaction, if there is such a log record.2 In many WAL-based systems, the updates performed during a rollback are logged using what are called compensation log records (CLRS) [151. Whether a CLR’S update is undone, should that CLR be encountered during a rollback, depends on the particular system. As we will see later, in ARIES, a CLR’S update is never undone and hence CLRS are viewed as redo-only log records. Page-oriented redo is said to occur if the log record whose update is being redone describes which page of the database was originally modified during normal processing and if the same page is modified during the redo process- ing. No internal descriptors of tables or indexes need to be accessed to redo the update. That is, no other page of the database needs to be examined. This is to be contrasted with logical redo which is required in System R, SQL/DS and AS/400 for indexes [21, 621. In those systems, since index changes are not logged separately but are redone using the log records for the data pages, performing a redo requires accessing several descriptors and pages of the database. The index tree would have to be retraversed to determine the page(s) to be modified and, sometimes, the index page(s) modified because of this redo operation may be different from the index page(s) originally modified during normal processing. Being able to perform page-oriented redo allows the system to provide recovery independence amongst objects. That is, the recovery of one page’s contents does not require accesses to any other 2 The AS/400, Encompass and NonStop SQL do not explicitly link all the log records written by a transaction. This makes undo inefficient since a sequential backward scan of the log must be performed to retrieve all the desired log records of a transaction. ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992 

7 .100 . C. Mohan et al (data or catalog) pages of the database. As we will describe later, this makes media recovery very simple. In a similar fashion, we can define page-oriented undo and logical undo. Being able to perform logical undos allows the system to provide higher levels of concurrency than what would be possible if the system were to be restricted only to page-oriented undos. This is because the former, with appropriate concurrency control protocols, would permit uncommitted updates of one transaction to be moved to a different page by another transaction. If one were restricted to only page-oriented undos, then the latter transaction would have had to wait for the former to commit. Page-oriented redo and page-oriented undo permit faster recovery since pages of the database other than the pages mentioned in the log records are not accessed. In the interest of efficiency, ARIES supports page-oriented redo and its supports, in the interest of high concurrency, logical undos. In [62], we introduce the ARIES/IM method for concurrency control and recovery in B ‘-tree indexes and show the advantages of being able to perform logical undos by comparing ARIES/IM with other index methods. 1.2 Latches and Locks Normally latches and locks are used to control access to shared information. Locking has been discussed to a great extent in the literature. Latches, on the other hand, have not been discussed that much. Latches are like semaphores. Usually, latches are used to guarantee physical consistency of data, while locks are used to assure logical consistency of data. We need to worry about physical consistency since we need to support a multiprocessor environment. Latches are usually held for a much shorter period than are locks. Also, the deadlock detector is not informed about latch waits. Latches are requested in such a manner so as to avoid deadlocks involving latches alone, or involving latches and locks. Acquiring and releasing a latch is much cheaper than acquiring and releasing a lock. In the no-conflict case, the overhead amounts to 10s of instructions for the former versus 100s of instructions for the latter. Latches are cheaper because the latch control information is always in virtual mem- ory in a fixed place, and direct addressability to the latch information is possible given the latch name. As the protocols presented later in this paper and those in [57, 621 show, each transaction holds at most two or three latches simultaneously. As a result, the latch request blocks can be perman- ently allocated to each transaction and initialized with transaction ID, etc. right at the start of that transaction. On the other hand, typically, storage for individual locks has to be acquired, formatted and released dynamically, causing more instructions to be executed to acquire and release locks. This is advisable because, in most systems, the number of lockable objects is many orders of magnitude greater than the number of latchable objects. Typically, all information relating to locks currently held or requested by all the transactions is stored in a single, central hash table. Addressability to a particular lock’s information is gained by first hashing the lock name to get the address of the hash anchor and then, possibly, following a chain of pointers. Usually, in the process of trying to locate the lock control block, ACM Transactions on Database Systems, Vol 17, No 1, March 1992 

8 . ARIES: A Transaction Recovery Method . 101 because multiple transactions may be simultaneously reading and modifying the contents of the lock table, one or more latches will be acquired and released—one latch on the hash anchor and, possibly, one on the specific lock’s chain of holders and waiters. Locks may be obtained in different modes such as S (Shared), X (exclusive), IX (Intention exclusive), IS (Intention Shared) and SIX (Shared Intention exclusive), and at different granularities such as record (tuple), table (rela- tion), and file (tablespace) [321. The S and X locks are the most common ones. S provides the read privilege and X provides the read and write privileges. Locks on a given object can be held simultaneously by different transactions only if those locks’ modes are compatible. The compatibility relationships amongst the above modes of locking are shown in Figure 2. A check mark (’<) indicates that the corresponding modes are compatible. With hierarchi- cal locking, the intention locks (IX, IS, and SIX) are generally obtained on the higher levels of the hierarchy (e.g., table), and the S and X locks are obtained on the lower levels (e. g., record). The nonintention mode locks (S and X), when obtained on an object at a certain level of the hierarchy, implicitly grant locks of the corresponding mode on the lower level objects of that higher level object. The intention mode locks, on the other hand, only give the privilege of requesting the corresponding intention or nonintention mode locks on the lower level objects. For example, SIX on a table implicitly grants S on all the records of that table, and it allows X to be requested explicitly on the records. Additional, semantically rich lock modes have been defined in the literature [2, 38, 45, 551 and ARIES can accommodate them. Lock requests may be made with the conditional or the unconditional option. A conditional request means that the requestor is not willing to wait if, when the request is processed, the lock is not grantable immediately. An unconditional request means that the requestor is willing to wait until the lock becomes grantable. Locks may be held for different durations. An unconditional request for an instant duration lock means that the lock is not to be actually granted, but the lock manager has to delay returning the lock call with the success status until the lock becomes grantable. Manual duration locks are released some time after they are acquired and, typically, long before transaction termination. Commit duration locks are released only when the transaction terminates, i.e., after commit or rollback is completed. The above discussions concerning conditional requests, different modes, and durations, except for commit duration, apply to latches also. 1.3 Fine-Granularity Locking Fine-granularity (e.g., record) locking has been supported by nonrelational database systems (e.g., IMS [53, 76, 801) for a long time. Surprisingly, only a few of the commercially available relational systems provide fine-granularity locking, even though IBM’s System R [321, S/38 [991 and SQL/DS [51, and Tandem’s Encompass [37] supported record and/or key locking from the beginning. 3 Although many interesting problems relating to providing 3 Encompass and S/38 had only X locks for records and no locks were acquired automatically by these systems for reads. ACM Transactions on Database SyStanS, Vol. 17, No 1, March 1992 

9 .102 . C. Mohan et al. Fig. 2. matrix Lock mode comparability m lx Slx + 4 .’ fine-granularity locking in the context of WAL remain to be solved, the research community has not been paying enough attention to this area [3, 75, 88]. Some of the System R solutions worked only because of the use of the shadow page recovery technique in combination with locking (see Section 10). Supporting fine-granularity locking and variable length records in a flexible fashion requires addressing some interesting storage management issues which have never really been discussed in the database literature. Unfortunately, some of the interesting techniques that were developed for System R and which are now part of SQL/DS did not get documented in the literature. At the expense of making this paper long, we will be discussing here some of those problems and their solutions. As supporting high concurrency gains importance (see [79] for the descrip- tion of an application requiring very high concurrency) and as object-oriented systems gain in popularity, it becomes necessary to invent concurrency control and recovery methods that take advantage of the semantics of the operations on the data [2, 26, 38, 88, 891, and that support fine-granularity locking efficiently. Object-oriented systems may tend to encourage users to define a large number of small objects and users may expect object instances to be the appropriate granularity of locking. In the object-oriented logical view of the database, the concept of a page, with its physical orientation as the container of objects, becomes unnatural to think about as the unit of locking during object accesses and modifications. Also, object-oriented system users may tend to have many terminal interactions during the course of a transaction, thereby increasing the lock hold times. If the unit of locking were to be a page, lock wait times and deadlock possibilities will be aggra- vated. Other discussions concerning transaction management in an object- oriented environment can be found in [22, 29]. As more and more customers adopt relational systems for production applications, it becomes ever more important to handle hot-spots [28, 34, 68, 77, 79, 83] and storage management without requiring too much tuning by the system users or administrators. Since relational systems have been welcomed to a great extent because of their ease of use, it is important that we pay greater attention to this area than what has been done in the context of the nonrelational systems. Apart from the need for high concurrency for user data, the ease with which online data definition operations can be performed in relational systems by even ordinary users requires the support for high concurrency of access to, at least, the catalog data. Since a leaf page in an index typically describes data in hundreds of data pages, page-level locking of index data is just not acceptable. A flexible recovery method that ACM TransactIons on Database Systems, Vol 17, No. 1, March 1992. 

10 . ARIES: A Transaction Recovery Method . 103 allows the support of high levels of concurrency during index accesses is needed. The above facts argue for supporting semantically rich modes of locking such as increment/decrement which allow multiple transactions to concur- rently modify even the same piece of data. In funds-transfer applications, increment and decrement operations are frequently performed on the branch and teller balances by numerous transactions. If those transactions are forced to use only X locks, then they will be serialized, even though their operations commute. 1.4 Buffer Management The buffer manager (BM) is the component of the transaction system that manages the buffer pool and does 1/0s to read/write pages from/to the nonvolatile storage version of the database. The fix primitive of the BM may be used to request the buffer address of a logical page in the database. If the requested page is not in the buffer pool, BM allocates a buffer slot and reads the p~ge in. There may be instances (e. g., during a B ‘-tree page split, when the new page is allocated) where the current contents of a page on nonvolatile storage are not of interest. In such a case, the fix– new primitive may be used to make the BM allocate a ji-ee slot and return the address of that slot, if BM does not find the page in the buffer pool. The fix-new invoker will then format the page as desired. Once a page is fixed in the buffer pool, the corresponding buffer slot is not available for page replacement until the unfix primitive is issued by the data manipulative component. Actually, for each page, BM keeps a fix count which is incremented by one during every fix operation and which is decremented by one during every unfix operation. A page in the buffer pool is said to be dirty if the buffer version of the page has some updates which are not yet reflected in the nonvolatile storage version of the same page. The fix primitive is also used to communicate the intention to modify the page. Dirty pages can be written back to nonvolatile storage when no fix with the modification intention is held, thus allowing read accesses to the page while it is being written out. [96] discusses the role of BM in writing in the background, on a continuous basis, dirty pages to nonvolatile storage to reduce the amount of redo work that would be needed if a system failure were to occur and also to keep a certain percentage of the buffer pool pages in the nondirty state so that they may be replaced with other pages without synchronous write 1/0s having to be performed at the time of replacement. While performing those writes, BM ensures that the WAL protocol is obeyed. As a consequence, BM may have to force the log up to the LSN of the dirty page before writing the page to nonvolatile storage. Given the large buffer pools that are common today, we would expect a force of this nature to be very rare and most log forces to occur because of transactions committing or entering the prepare state. BM also implements the support for latching pages. To provide direct addressability to page latches and to reduce the storage associated with those latches, the latch on a logical page is actually the latch on the corresponding buffer slot. This means that a logical page can be latched only after it is fixed ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

11 .104 . C. Mohan et al in the buffer pool and the latch has to be released before the page is unfixed. These are highly acceptable conditions. The latch control information is stored in the buffer control block (BCB) for the corresponding buffer slot. The BCB also contains the identity of the logical page, what the fix count is, the dirty status of the page, etc. Buffer management policies differ among the many systems in existence (see Section 11, “Other WAL-Based Methods”). If a page modified by a transaction is allowed to be written to the permanent database on nonvolatile storage before that transaction commits, then the steal policy is said to be followed by the buffer manager (see [361 for such terminologies). Otherwise, a no-steal policy is said to be in effect. Steal implies that during normal or restart rollback, some undo work might have to be performed on the non- volatile storage version of the database. If a transaction is not allowed to commit until all pages modified by it are written to the permanent version of the database, then a force policy is said to be in effect. Otherwise, a no-force policy is said to be in effect. With a force policy, during restart recovery, no redo work will be necessary for committed transactions. Deferred updating is said to occur if, even in the virtual storage database buffers, the updates are not performed in-place when the transaction issues the corresponding database calls. The updates are kept in a pending list elsewhere and are performed in-place, using the pending list information, only after it is deter- mined that the transaction is definitely committing. If the transaction needs to be rolled back, then the pending list is discarded or ignored. The deferred updating policy has implications on whether a transaction can “see” its own updates or not, and on whether partial rollbacks are possible or not. For more discussions concerning buffer management, see [8, 15, 24, 961. 1.5 Organization The rest of the paper is organized as follows. After stating our goals in Section 2 and giving an overview of the new recovery method ARIES in Section 3, we present, in Section 4, the important data structures used by ARIES during normal and restart recovery processing. Next, in Section 5, the protocols followed during normal processing are presented followed, in Section 6, by the description of the processing performed during restart recovery. The latter section also presents ways to exploit parallelism during recovery and methods for performing recovery selectively or postponing the recovery of some of the data. Then, in Section 7, algorithms are described for taking checkpoints during the different log passes of restart recovery to reduce the impact of failures during recovery. This is followed, in Section 8, by the description of how fuzzy image copying and media recovery are supported. Section 9 introduces the significant notion of nested top actions and presents a method for implementing them efficiently. Section 10 describes and cri- tiques some of the existing recovery paradigms which originated in the context of the shadow page technique and System R. We discuss the problems caused by using those paradigms in the WAL context. Section 11 describes in detail the characteristics of many of the WAL-based recovery methods in use in different systems such as IMS, DB2, Encompass and NonStop SQL. ACM Transactions on Database Systems, Vol 17, No. 1, March 1992 

12 . ARIES: A Transaction Recovery Method . 105 Section 12 outlines the many different properties of ARIES. We conclude by summarizing, in Section 13, the features of ARIES which provide flexibility and efficiency, and by describing the extensions and the current status of the implementations of ARIES. Besides presenting a new recovery method, by way of motivation for our work, we also describe some previously unpublished aspects of recovery in System R. For comparison purposes, we also do a survey of the recovery methods used by other WAL-based systems and collect information appearing in several publications, many of which are not widely available. One of our aims in this paper is to show the intricate and unobvious interactions resulting from the different choices made for the recovery technique, the granularity of locking and the storage management scheme. One cannot make arbitrarily independent choices for these and still expect the combina- tion to function together correctly and efficiently. This point needs to be emphasized as it is not always dealt with adequately in most papers and books on concurrency control and recovery. In this paper, we have tried to cover, as much as possible, all the interesting recovery-related problems that one encounters in building and operating an industrial-strength transaction processing system. 2. GOALS This section lists the goals of our work and outlines the difficulties involved in designing a recovery method that supports the features that we aimed for. The goals relate to the metrics for comparison of recovery methods that we discussed earlier, in Section 1.1. Simplicity. Concurrency and recovery are complex subjects to think about and program for, compared with other aspects of data management. The algorithms are bound to be error-prone, if they are complex. Hence, we strived for a simple, yet powerful and flexible, algorithm. Although this paper is long because of its comprehensive discussion of numerous problems that are mostly ignored in the literature, the main algorithm itself is quite simple. Hopefully, the overview presented in Section 3 gives the reader that feeling. Operation logging. The recovery method had to permit operation logging (and value logging) so that semantically rich lock modes could be supported. This would let one transaction modify the same data that was modified earlier by another transaction which has not yet committed, when the two transaction:’ actions are semantically compatible (e.g., increment/decrement operations; see [2, 26, 45, 881). As should be clear, recovery methods which always perform value or state logging (i. e., logging before-images and after- images of modified data), cannot support operation logging. This includes systems that do very physical —byte-oriented— logging of all changes to a page [6, 76, 811. The difficulty in supporting operation logging is that we need to track precisely, using a concept like the LSN, the exact state of a page with respect to logged actions relating to that page. An undo or a redo of an update should not be performed without being sure that the original update ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992 

13 .106 . C. Mohan et al is present or is not present, respectively. This also means that, if one or more transactions that had previously modified a page start rolling back, then we need to know precisely how the page has been affected during the rollbacks and how much of each of the rollbacks had been accomplished so far. This requires that updates performed during rollbacks also be logged via the so-called compensation log records (CLRS). The LSN concept lets us avoid attempting to redo an operation when the operation’s effect is already present in the page. It also lets us avoid attempting to undo an operation when the operation’s effect is not present in the page. Operation logging lets us perform, if found desirable, logical logging, which means that not every- thing that was changed on a page needs to be logged explicitly, thereby saving log space. For example, changes of control information, like the amount of free space on the page, need not be logged. The redo and the undo operations can be performed logically. For a good discussion of operation and value logging, see [881. Flexible storage management. Efficient support for the storage and manip- ulation of varying length data is important. In contrast to systems like IMS, the intent here is to be able to avoid the need for off-line reorganization of the data to garbage collect any space that might have been freed up because of deletions and updates that caused data shrinkage. It is desir- able that the recovery method and the concurrency control method be such that the logging and locking is logical in nature so that movements of the data within a page for garbage collection reasons do not cause the moved data to be locked or the movements to be logged. For an index, this also means that one transaction must be able to split a leaf page even if that page currently has some uncommitted data inserted by another transac- tion. This may lead to problems in performing page-oriented undos using the log; logical undos may be necessary. Further, we would like to be able to let a transaction that has freed up some space be able to use, if necessary, that space during its later insert activity [50]. System R, for example, does not permit this in data pages. Partial rollbacks. It was essential that the new recovery method sup- port the concept of savepoints and rollbacks to savepoints (i.e., partial rollbacks). This is crucial for handling, in a user-friendly fashion (i. e., without requiring a total rollback of the transaction), integrity constraint violations (see [1, 311), and problems arising from using obsolete cached information (see [49]). Flexible buffer management. The recovery method should make the least number of restrictive assumptions about the buffer management policies (steal, force, etc.) in effect. At the same time, the method must be able to take advantage of the characteristics of any specific policy that is in effect (e.g., with a force policy there is no need to perform any redos for committed transactions.) This flexibility could result in increased concurrency, decreased 1/0s and efficient usage of buffer storage. Depending on the policies, the work that needs to be performed during restart recovery after a system ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992 

14 . ARIES: A Transaction Recovery Method . 107 failure or during media recovery maybe more or less complex. Even with large main memories, it must be noted that a steal policy is still very desirable. This is because, with a no-steal policy, a page may never get written to nonvolatile storage if the page always contains uncommitted updates due to fine-~anularity locking and overlapping transactions’ updates to that page. The situation would be further aggravated if there are long- running transactions. Under those conditions, either the system would have to frequently reduce concurrency by quiescing all activities on the page (i.e., by locking all the objects on the page) and then writing the page to non- volatile storage, or by doing nothing special and then paying a huge restart redo recovery cost if the system were to fail. Also, a no-steal policy incurs additional bookkeeping overhead to track whether a page contains any uncommitted updates. We believe that, given our goal of supporting semanti- cally rich lock modes, partial rollbacks and varying length objects efficiently, in the general case, we need to perform undo logging and in-place updating. Hence, methods like the transaction workspace model of AIM [46] are not general enough for our purposes. Other problems relating to no-steal are discussed in Section 11 with reference to IMS Fast Path. Recovery independence. It should be possible to image copy (archive dump), and perform media recovery or restart recovery at different granularities, rather than only at the entire database level. The recovery of one object should not force the concurrent or lock-step recovery of another object. Contrast this with what happens in the shadow page technique as imple- mented in System R, where index and space management information are recovered lock-step with user and catalog table (relation) data by starting from an internally consistent state of the whole database and redoing changes to all the related objects of the database simultaneously, as in normal processing. Recovery independence means that, during the restart recovery of some object, catalog information in the database cannot be accessed for descriptors of that object and its related objects, since that information itself may be undergoing recovery in parallel with the object being recovered and the two may be out of synchronization [141. During restart recovery, it should be possible to do selective recovery and defer recovery of some objects to a later point in time to speed up restart and also to accommodate some offline devices. Page-oriented recovery means that even if one page in the database is corrupted because of a process failure or a media problem, it should be possible to recover that page alone. To be able to do this efficiently, we need to log every page’s change individually, even if the object being updated spans multiple pages and the update affects more than one page. This, in conjunction with the writing of CLRS for updates performed during rollbacks, will make media recovery very simple (see Section 8). This will also permit image copying of different objects to be performed independently and at different frequencies. Logical undo. This relates to the ability, during undo, to affect a page that is different from the one modified during forward processing, as is ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

15 .108 . C. Mohan et al. needed in the earlier-mentioned context of the split by one transaction of an index page containing uncommitted data of another transaction. Being able to perform logical undos allows higher levels of concurrency to be supported, especially in search structures [57, 59, 621. If logging is not performed during rollback processing, logical undos would be very difficult to support, if we also desired recovery independence and page-oriented recovery. System R and SQL/DS support logical undos, but at the expense of recovery independence. Parallelism and fast recovery. With multiprocessors becoming very com- mon and greater data availability becoming increasingly important, the recovery method has to be able to exploit parallelism during the different stages of restart recovery and during media recovery. It is also important that the recovery method be such that recovery can be very fast, if in fact a hot-standby approach is going to be used (a la IBM’s IMS/VS XRF [431 and Tandem’s NonStop [4, 371). This means that redo processing and, whenever possible, undo processing should be page-oriented (cf. always logical redos and undos in System R and SQL/DS for indexes and space management). It should also be possible to let the backup system start processing new transac- tions, even before the undo processing for the interrupted transactions com- pletes. This is necessary because undo processing may take a long time if there were long update transactions. Minimal overhead. Our goal is to have good performance both during normal and restart recovery processing. The overhead (log data volume, storage consumption, etc.) imposed by the recovery method in virtual and nonvolatile storages for accomplishing the above goals should be minimal. Contrast this with the space overhead caused by the shadow page technique. This goal also implied that we should minimize the number of pages that are modified (dirtied) during restart. The idea is to reduce the number of pages that have to be written back to nonvolatile storage and also to reduce CPU overhead. This rules out methods which, during restart recovery, first undo some committed changes that had already reached the nonvolatile storage before the failure and then redo them (see, e.g., [16, 21, 72, 78, 881). It also rules out methods in which updates that are not present in a page on nonvolatile storage are undone unnecessarily (see, e.g., [41, 71, 881). The method should not cause deadlocks involving transactions that are already rolling back. Further, the writing of CLRS should not result in an unbounded number of log records having to be written for a transaction because of the undoing of CLRS, if there were nested rollbacks or repeated system failures during rollbacks. It should also be possible to take checkpoints and image copies without quiescing significant activities in the system. The impact of these operations on other activities should be minimal. To contrast, check- pointing and image copying in System R cause major perturbations in the rest of the system [31]. As the reader will have realized by now, some of these goals are contradic- tory. Based on our knowledge of different developers’ existing systems’ features, experiences with IBM’s existing transaction systems and contacts ACM Transactions on Database Systems, Vol 17, No 1, March 1992 

16 . ARIES: A TransactIon Recovery Method . 109 with customers, we made the necessary tradeoffs. We were keen on learning from the past successes and mistakes involving many prototypes and products. 3. OVERVIEW OF ARIES The aim of this section is to provide a brief overview of the new recovery method ARIES, which satisfies quite reasonably the goals that we set forth in Section 2. Issues like deferred and selective restart, parallelism during restart recovery, and so on will be discussed in the later sections of the paper. ARIES guarantees the atomicity and durability properties of transactions in the fact of process, transaction, system and media failures. For this purpose, ARIES keeps track of the changes made to the database by using a log and it does write-ahead logging (WAL). Besides logging, on a per- affected-page basis, update activities performed during forward processing of transactions, ARIES also logs, typically using compensation log records (CLRS), updates performed during partial or total rollbacks of transactions during both normal and restart processing. Figure 3 gives an example of a partial rollback in which a transaction, after performing three updates, rolls back two of them and then starts going forward again. Because of the undo of the two updates, two CLRS are written. In ARIES, CLRS have the property that they are redo-only log records. By appropriate chaining of the CLRS to log records written during forward processing, a bounded amount of logging is ensured during rollbacks, even in the face of repeated failures during restart or of nested rollbacks. This is to be contrasted with what happens in IMS, which may undo the same non-CLR multiple times, and in AS/400, DB2 and NonStop SQL, which, besides undoing the same non-CLR multiple times, may also undo CLRS one or more times (see Figure 4). These have caused severe problems in real-life customer situations. In ARIES, as Figure 5 shows, when the undo of a log record causes a CLR to be written, the CLR, besides containing a description of the compensating action for redo purposes, is made to contain the UndoNxtLSN pointer which points to the predecessor of the just undone log record. The predecessor information is readily available since every log record, including a CLR, contains the PreuLSN pointer which points to the most recent preceding log record written by the same transaction. The UndoNxtLSN pointer allows us to determine precisely how much of the transaction has not been undone so far. In Figure 5, log record 3’, which is the CLR for log record 3, points to log record 2, which is the predecessor of log record 3. Thus, during rollback, the UndoNxtLSN field of the most recently written CLR keeps track of the progress of rollback. It tells the system from whereto continue the rollback of the transaction, if a system failure were to interrupt the completion of the rollback or if a nested rollback were to be performed. It lets the system bypass those log records that had already been undone. Since CLRS are available to describe what actions are actually ~erformed during the undo of an original action, the undo action need not be, in terms of which page(s) is affected, the exact inverse of the original action. That is, logical undo which allows very high concurrency to be supported is made possible. For example, ACM Transactions on Database Systems, Vol 17, No. 1, March 1992. 

17 .110 . C. Mohan et al. w Fig. 3. Partial rollback example. 12 33’2’4 !3j Log > After performing 3 actions, the transaction performs a patilal rollback by undoing actions 3 and 2, wrlt!ng the compensation log records 3 and 2, and then starts go[ng forward aga!n and performs act~ons 4 and 5 I Before Failure 1 Log , During Restart DB2, s/38, 2’” 3“ 3’ ~ 1; Encompass --------------------------- > AS/400 3’ 2’ 1’ lMS ) I’ is the CLR for I and I“ is the CLR for I’ Fig. 4 Problem of compensating compensations or duplicate compensations, or both a key inserted on page 10 of a B ‘-tree by one transaction may be moved to page 20 by another transaction before the key insertion is committed. Later, if the first transaction were to roll back, then the key will be located on page 20 by retraversing the tree and deleted from there. A CLR will be written to describe the key deletion on page 20. This permits page-oriented redo which is very efficient. [59, 621 describe ARIES/LHS and ARIES/IM which exploit this logical undo feature. ARIES uses a single LSN on each page to track the page’s state. Whenever a page is updated and a log record is written, the LSN of the log record is placed in the page-LSN field of the updated page. This tagging of the page with the LSN allows ARIES to precisely track, for restart- and media- recovery purposes, the state of the page with respect to logged updates for that page. It allows ARIES to support novel lock modes! using which, before an update performed on a record’s field by one transaction is committed, another transaction may be permitted to modify the same data for specified operations. Periodically during normal processing, ARIES takes checkpoints. The checkpoint log records identify the transactions that are active, their states, and the LSNS of their most recently written log records, and also the modified data (dirty data) that is in the buffer pool. The latter information is needed to determine from where the redo pass of restart recovery should begin its processing. ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

18 . ARIES: A Transaction Recovery Method . 111 Before Failure 12 3 3’ 2’ 1! Log F i- ) ‘,; -. ?% / / \\ -=---- / -% / ------ --- During Restart ,, I ----------------------------------------------+1 I’ is the Compensation Log Record for I I’ points to the predecessor, if any, of I Fig. 5. ARIES’ technique for avoiding compensating compensation and duplicate compensations. During restart recovery (see Figure 6), ARIES first scans the log, starting from the first record of the last checkpoint, up to the end of the log. During this analysis pass, information about dirty pages and transactions that were in progress at the time of the checkpoint is brought up to date as of the end of the log. The analysis pass uses the dirty pages information to determine the starting point ( li!edoLSIV) for the log scan of the immediately following redo pass. The analysis pass also determines the list of transactions that are to be rolled back in the undo pass. For each in-progress transaction, the LSN of the most recently written log record will also be determined. Then, during the redo pass, ARIES repeats history, with respect to those updates logged on stable storage, but whose effects on the database pages did not get reflected on nonvolatile storage before the failure of the system. This is done for the updates of all transactions, including the updates of those transactions that had neither committed nor reached the in-doubt state of two-phase commit by the time of the system failure (i.e., even the missing updates of the so-called loser transactions are redone). This essentially reestablishes the state of the database as of the time of the system failure. A log record’s update is redone if the affected page’s page-LSN is less than the log record’s LSN. No logging is performed when updates are redone. The redo pass obtains the locks needed to protect the uncommitted updates of those distributed transac- tions that will remain in the in-doubt (prepared) state [63, 64] at the end of restart recovery. The next log pass is the undo pass during which all loser transactions’ updates are rolled back, in reverse chronological order, in a single sweep of the log. This is done by continually taking the maximum of the LSNS of the next log record to be processed for each of the yet-to-be-completely-undone loser transactions, until no transaction remains to be undone. Unlike during the redo pass, performing undos is not a conditional operation during the undo pass (and during normal undo). That is, ARIES does not compare the page.LSN of the affected page to the LSN of the log record to decide ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

19 .112 . C. Mohan et al Log m r’ @ Checkpoint Follure i DB2 Analysis System R I / Undo Losers *————— ——. “––-––––––X* Redo Nonlosers —— — ————,& Redo Nonlosers (FP Updates) & Analysis . ------ IMS ..:-------- Undo Losers (NonFP Updates) 1 ------- ARIES Redo ALL Undo Losers .-:”--------- I Fig. 6, Restart processing in different methods. whether or not to undo the update. When a non-CLR is encountered for a transaction during the undo pass, if it is an undo-redo or undo-only log record, then its update is undone. In any case, the next record to process for that transaction is determined by looking at the PrevLSN of that non-CLR. Since CLRS are never undone (i.e., CLRS are not compensated– see Figure 5), when a CLR is encountered during undo, it is used just to determine the next log record to process by looking at the UndoNxtLSN field of the CLR. For those transactions which were already rolling back at the time of the system failure, ARIES will rollback only those actions that had not already been undone. This is possible since history is repeated for such transactions and since the last CLR written for each transaction points (directly or indirectly) to the next non-CLR record that is to be undone, The net result is that, if only page-oriented undos are involved or logical undos generate only CLRS, then, for rolled back transactions, the number of CLRS written will be exactly equal to the number of undoable) log records written during forward processing of those transactions. This will be the case even if there are repeated failures during restart or if there are nested rollbacks. 4. DATA STRUCTURES This section describes the major data structures that are used by ARIES. 4.1 Log Records Below, we describe the important fields that may be present in different types of log records. ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992, 

20 . ARIES: A Transaction Recovery Method . 113 LSN. Address of the first byte of the log record in the ever-growing log address space. This is a monotonically increasing value. This is shown here as a field only to make it easier to describe ARIES. The LSN need not actually be stored in the record. Type. Indicates whether this is a compensation record (’compensation’), a regular update record (’update’), a commit protocol-related record (e. g., ‘pre- pare’), or a nontransaction-related record (e.g., ‘OSfile_return’). TransID. Identifier of the transaction, if any, that wrote the log record. PrevLSN. LSN of the preceding log record written by the same transac- tion. This field has a value of zero in nontransaction-related records and in the first log record of a transaction, thus avoiding the need for an explicit begin transaction log record. PageID. Present only in records of type ‘update’ or ‘compensation’. The identifier of the page to which the updates of this record were applied. This PageID will normally consist of two parts: an objectID (e.g., tablespaceID), and a page number within that object. ARIES can deal with a log record that contains updates for multiple pages. For ease of exposition, we assume that only one page is involved. UndoNxtLSN. Present only in CLRS. It is the LSN of the next log record of this transaction that is to be processed during rollback. That is, UndoNxtLSN is the value of PrevLSN of the log record that the current log record is compensating. If there are no more log records to be undone, then this field contains a zero. Data. This is the redo and/or undo data that describes the update that was performed. CLRS contain only redo information since they are never undone. Updates can be logged in a logical fashion. Changes to some fields (e.g., amount of free space) of that page need not be logged since they can be easily derived. The undo information and the redo information for the entire object need not be logged. It suffices if the changed fields alone are logged. For increment or decrement types of operations, before and after-images of the field are not needed. Information about the type of operation and the decrement or increment amount is enough. The information here would also be used to determine the appropriate action routine to be used to perform the redo and/or undo of this log record. 4.2 Page Structure One of the fields in every page of the database is the page-LSN field. It contains the LSN of the log record that describes the latest update to the page. This record may be a regular update record or a CLR. ARIES expects the buffer manager to enforce the WAL protocol. Except for this, ARIES does not place any restrictions on the buffer page replacement policy. The steal buffer management policy may be used. In-place updating is performed on nonvolatile storage. Updates are applied immediately and directly to the ACM Transactions on Database Systems, Vol. 17, No, 1, March 1992. 

21 .114 . C. Mohan et al. buffer version of the page containing the object. That is, no deferred updating as in INGRES [861 is performed. If it is found desirable, deferred updat- ing and, consequently, deferred logging can be implemented. ARIES is flexible enough not to preclude those policies from being implemented. 4.3 Transaction Table A table called the transaction table is used during restart recovery to track the state of active transactions. The table is initialized during the analysis pass from the most recent checkpoint’s record(s) and is modified during the analysis of the log records written after the beginning of that checkpoint. During the undo pass, the entries of the table are also modified. If a checkpoint is taken during restart recovery, then the contents of the table will be included in the checkpoint record(s). The same table is also used during normal processing by the transaction manager. A description of the important fields of the transaction table follows: TransID. Transaction ID. State. Commit state of the transaction: prepared (’P’ –also called in-doubt) or unprepared (’U’). LastLSN. The LSN of the latest log record written by the transaction. UndoNxtLSN. The LSN of the next record to be processed during roll- back. If the most recent log record written or seen for this transaction is an undoable non-CLR log record, then this field’s value will be set to LastLSN. If that most recent log record is a CLR, then this field’s value is set to the UndoNxtLSN value from that CLR. 4.4 Dirty_ Pages Table A table called the dirty .pages table is used to represent information about dirty buffer pages during normal processing. This table is also used during restart recovery. The actual implementation of this table may be done using hashing or via the deferred-writes queue mechanism of [961. Each entry in the table consists of two fields: PageID and RecLSN (recovery LSN). During normal processing, when a nondirty page is being fixed in the buffers with the intention to modify, the buffer manager records in the buffer pool (BP) dirty .pages table, as RecLSN, the current end-of-log LSN, which will be the LSN of the next log record to be written. The value of RecLSN indicates from what point in the log there may be updates which are, possibly, not yet in the nonvolatile storage version of the page. Whenever pages are written back to nonvolatile storage, the corresponding entries in the BP dirty _pages table are removed. The contents of this table are included in the checkpoint record(s) that is written during normal processing. The restart dirty –pages table is initialized from the latest checkpoint’s record(s) and is modified during the analysis of the other records during the analysis pass. The ACM Transactions on Database Systems, Vol 17, No 1, March 1992 

22 . ARIES: A Transaction Recovery Method . 115 minimum RecLSN value in the table gives the starting point for the redo pass during restart recovery. 5. NORMAL PROCESSING This section discusses the actions that are performed as part of normal transaction processing. Section 6 discusses the actions that are performed as part of recovering from a system failure. 5.1 Updates During normal processing, transactions may be in forward processing, partial rollback or total rollback. The rollbacks may be system- or application-ini- tiated. The causes of rollbacks may be deadlocks, error conditions, integrity constraint violations, unexpected database state, etc. If the granularity of locking is a record, then, when an update is to be performed on a record in a page, after the record is locked, that page is fixed in the buffer and latched in the X mode, the update is performed, a log record is appended to the log, the LSN of the log record is placed in the page .LSN field of the page and in the transaction table, and the page is unlatched and unfixed. The page latch is held during the call to the logger. This is done to ensure that the order of logging of updates of a page is the same as the order in which those updates are performed on the page. This is very important if some of the redo information is going to be logged physically (e.g., the amount of free space in the page) and repetition of history has to be guaranteed for the physical redo to work correctly. The page latch must be held during read and update operations to ensure physical consistency of the page contents. This is necessary because inserters and updaters of records might move records around within a page to do garbage collection. When such garbage collection is going on, no other transaction should be allowed to look at the page since they might get confused. Readers of pages latch in the S mode and modifiers latch in the X mode. The data page latch is not held while any necessary index operations are performed. At most two page latches are held simultaneously (also see [57, 621). This means that two transactions, T1 and T2, that are modi- fying different pieces of data may modify a particular data page in one order (Tl, T2) and a particular index page in another order (T2, T1).4 This scenario is impossible in System R and SQL/DS since in those systems, locks, instead of latches are used for providing physical consistency. Typically, all the (physical) page locks are released only at the end of the RSS (data manager) call. A single RSS call deals with modifying the data and all relevant indexes. This may involve waiting for many 1/0s and locks. This means that deadlocks involving (physical) page locks alone or (physical) page locks and 4 The situation gets very complicated if operations like increment/decrement are supported with high concurrency lock modes and indexes are allowed to be defined on fields on which such operations are supported. We are currently studying those situations. ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

23 .116 . C. Mohan et al (logical) record/key locks are possible. They have been a major problem in System R and SQL/DS. Figure 7 depicts a situation at the time of a system failure which followed the commit of two transactions. The dotted lines show how up to date the states of pages PI and P2 are on nonvolatile storage with respect to logged updates of those pages. During restart recovery, it must be realized that the most recent log record written for PI, which was written by a transaction which later committed, needs to be redone, and that there is nothing to be redone for P2. This situation points to the need for having the LSN to relate the state of a page on nonvolatile storage to a particular position in the log and the need for knowing where restart redo pass should begin by noting some information in the checkpoint record (see Section 5.4). For the example scenario, the restart redo log scan should begin at least from the log record representing the most recent update of PI by T2, since that update needs to be redone. It is not assumed that a single log record can always accommodate all the information needed to redo or undo the update operation. There may be instances when more than one record needs to be written for this purpose. For example, one record may be written with the undo information and another one with the redo information. In such cases, (1) the undo-only log record should be written before the redo-only log record is written, and (2) it is the LSN of the redo-only log record that should be placed in the page.LSN field. The first condition is enforced to make sure that we do not have a situation in which the redo-only record and not the undo-only record gets written to stable storage before a failure, and that during restart recovery, the redo of that redo-only log record is performed (because of the repeating history feature) only to realize later that there isn’t an undo-only record to undo the effect of that operation. Given that the undo-only record is written before the redo-only record, the second condition ensures that we do not have a situation in which even though the page in nonvolatile storage already contains the update of the redo-only record, that same update gets redone unnecessarily during restart recovery because the page contained the L SN of the undo-only record instead of that of the redo-only record. This unnecessary redo could cause integrity problems if operation logging is being performed. There may be some log records written during forward processing that cannot or should not be undone (prepare, free space inventory update, etc. records). These are identified as redo-only log records. See Section 10.3 for a discussion of this kind of situation for free space inventory updates. Sometimes, the identity of the (data) record to be modified or read may not be known before a (data) page is examined. For example, during an insert, the record ID is not determined until the page is examined to find an empty slot. In such cases, the record lock must be obtained after the page is latched. To avoid waiting for a lock while holding a latch, which could lead to an undetected deadlock, the lock is requested conditionally, and if it is not granted, then the latch is released and the lock is requested unconditionally. Once the unconditionally requested lock is granted, the page is latched again, and any previously verified conditions are rechecked. This rechecking is ACM Transactions on Database Systems, Vol 17, No. 1, March 1992. 

24 . ARIES: A Transaction Recovery Method . 117 j;:’ /’ ,’ /’ LZN’”S /’ #“ PI El / ‘“O P ‘! ‘! ‘:\,; Log w PI pi PI Commit P2 Commit o T1 Failure @ Checkpoint / a T2 Fig. 7. Database state as a failure. required because, after the page was unlatched, the conditions could have changed. The page_LSN value at the time of unlatching could be remem- bered to detect quickly, on rematching, if any changes could have possibly occurred. If the conditions are still found to be satisfied for performing the update, it is performed as described above. Otherwise, corrective actions are taken. If the conditionally requested lock is granted immediately, then the update can proceed as before. If the granularity of locking is a page or something coarser than a page, then there is no need to latch the page since the lock on the page will be sufficient to isolate the executing transaction. Except for this change, the actions taken are the same as in the record-locking case. But, if the system is to support unlocked or dirty reads, then, even with page locking, a transac- tion that is updating a page should be made to hold the X latch on the page so that readers who are not acquiring locks are assured physical consistency if they hold an S latch while reading the page. Unlocked reads may also be performed by the image copy utility in the interest of causing the least amount of interference to normal transaction processing. Applicability of ARIES is not restricted to only those systems in which locking is used as the concurrency control mechanism. Even other concur- rency control schemes that are similar to locking, like the ones in [2], could be used with ARIES. 5.2 Total or Partial Rollbacks To provide flexibility in limiting the extent of transaction rollbacks, the notion of a sauepoint is supported [1, 31]. At any point during the execution of a transaction, a savepoint can be established. Any number of savepoints could be outstanding at a point in time. Typically, in a system like I)B2, a savepoint is established before every SQL data manipulation command that might perform updates to the data. This is needed to support SQL statement- level atomicity. After executing for a while, the transaction or the system can request the undoing of all the updates performed after the establishment of a still outstanding savepoint. After such a partial rollback, the transaction can ACM Transactions on Database Systems, Vol 17, No. 1, March 1992. 

25 .118 . C. Mohan et al. continue execution and start going forward again (see Figure 3). A particu- lar savepoint is no longer outstanding if a rollback has been performed to that savepoint or to a preceding one. When a savepoint is established, the LSN of the latest log record written by the transaction, called SaueLSN, is remembered in virtual storage. If the savepoint is being established at the beginning of the transaction (i.e., when it has not yet written a log record) SaveLSN is set to zero. When the transaction desires to roll back to a savepoint, it supplies the remembered SaveLSN. If the savepoint concept were to be exposed at the user level, then we would expect the system not to expose the SaveLSNs to the user but use some symbolic values or sequence numbers and do the mapping to LSNS internally, as is done in IMS [42] and INGRES [181. Figure 8 describes the routine ROLLBACK which is used for rolling back to a savepoint. The input to the routine is the SaveLSN and the TransID. No locks are acquired during rollback, even though a latch is acquired during undo activity on a page. Since we have always ensured that latches do not get involved in deadlocks, a rolling back transaction cannot get involved in a deadlock, as in System R and R* [31, 641 and in the algorithms of [1001. During the rollback, the log records are undone in reverse chronological order and, for each log record that is undone, a CLR is written. For ease of exposition, assume that all the information about the undo action will fit in a single CLR. It is easy to extend ARIES to the case where multiple CLRS need to be written. It is possible that, when a logical undo is performed, some non-CLRs are sometimes written, as described in [59, 62]. As mentioned before, when a CLR is written, its UndoNxtLSN field is made to contain the PrevLSN value in the log record whose undo caused this CLR to be written. Since CLRS will never be undone, they don’t have to contain undo informa- tion (e.g., before-images). Redo-only log records are ignored during rollback. When a non-CLR is encountered, after it is processed, the next record to process is determined by looking up its PrevLSN field. When a CLR is encountered during rollback, the UndoNxtLSN field of that record is looked up to determine the next log record to be processed. Thus, the UndoNxtLSN pointer helps us skip over already undone log records. This means that if a nested rollback were to occur, then, because of the UndoNxtLSN in CLRS, during the second rollback none of the log records that were undone during the first rollback would be processed again. Even though Figures 4, 5, and 13 describe partial rollback scenarios in conjunction with restart undos in the various recovery methods, it should be easy to see how nested rollbacks are handled efficiently by ARIES. Being able to describe, via CLRS, the actions performed during undo gives us the flexibility of not having to force the undo actions to be the exact inverses of the original actions. In particular, the undo action could affect a page which was not involved in the original action. Such logical undo situations are possible in, for example, index management [621 and space management (see Section 10.3). ARIES’ guarantee of a bounded amount of logging during undo allows us to deal safely with small computer systems situations in which a circular online ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992 

26 . ARIES: A Transaction Recovery Method . 119 \\\ \ *** u ,0 m w dFm m ~ 0 <0 m v c al m s- L .. Q .. .. z -J x m“. nc.1 ..! WE > 0 %’ : 0 CIA. . . . !’. : !! .. ‘n Fl ..! I 5 n“ -_l . al WI-’-l w M.-s z mztn CL. ulc & -am UWL aJ-.J l..- Crfu u! It 0 ..2 .-l =% ql- !! ;E %’2 al- ACM Transactions on Database Systems, Vol. 17, No 1, March 1992. 

27 .120 . C. Mohan et al log might be used and log space is at a premium. Knowing the bound, we can keep in reserve enough log space to be able to roll back all currently running transactions under critical conditions (e. g., log space shortage). The imple- mentation of ARIES in the 0S/2 Extended Edition Database Manager takes advantage of this. When a transaction rolls back, the locks obtained after the establishment of the savepoint which is the target of the rollback may be released after the partial or total rollback is completed. In fact, systems like DB2 do not and cannot release any of the locks after a partial rollback because, after such a lock release, a later rollback may still cause the same updates to be undone again, thereby causing data inconsistencies. System R does release locks after a partial rollback completes. But, because ARIES never undoes CLRS nor ever undoes a particular non-CLR more than once, because of the chaining of the CLRS using the UndoNxtLSN field, during a (partial) roll- back, when the transaction’s very first update to a particular object is undone and a CLR is written for it, the system can release the lock on that object. This makes it possible to consider resolving deadlocks using partial rollbacks rather than always resorting to total rollbacks. 5.3 Transaction Termination Assume that some form of two-phase commit protocol (e. g., Presumed Abort or Presumed Commit (see [63, 64])) is used to terminate transactions and that the prepare record which is synchronously written to the log as part of the protocol includes the list of update-type locks (IX, X, SIX, etc.) held by the transaction. The logging of the locks is done to ensure that if a system failure were to occur after a transaction enters the in-doubt state, then those locks could be reacquired, during restart recovery, to protect the uncommitted updates of the in-doubt transaction. 5 When the prepare record is written, the read locks (e.g., S and IS) could be released, if no new locks would be acquired later as part of getting into the prepare state in some other part of the distributed transaction (at the same site or a different site). To deal with actions (such as the dropping of objects) which may cause files to be erased, for the sake of avoiding the logging of such objects’ complete contents, we postpone performing actions like erasing files until we are sure that the transaction is definitely committing [191. We need to log these pending actions in the prepare record. Once a transaction enters the in-doubt state, it is committed by writing an end record and releasing its locks. Once the end record is written, if there are any pending actions, they they must be performed. For each pending action which involves erasing or returning a file to the operating system, we write an OSfile. return redo-only log record. For ease of exposition, we assume that this log record is not associated with any particular transaction and that this action does not take place when a checkpoint is in progress. 5Another possibility is not to log the locks, but to regenerate the lock names during restart recovery by examining all the log records written by the in-doubt transaction— see Sections 6.1 and 64, and item 18 (Section 12) for further ramifications of this approach ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

28 . ARIES: A Transaction Recovery Method . 121 A transaction in the in-doubt state is rolled back by writing a rollback record, rolling back the transaction to its beginning, discarding the pending actions list, releasing its locks, and then writing the end record. Whether or not the rollback and end records are synchronously written to stable storage will depend on the type of two-phase commit protocol used. Also, the writing of the prepare record may be avoided if the transaction is not a distributed one or is read-only. 5.4 Checkpoints Periodically, checkpoints are taken to reduce the amount of work that needs to be performed during restart recovery. The work may relate to the extent of the log that needs to be examined, the number of data pages that have to be read from nonvolatile storage, etc. Checkpoints can be taken asynchronously (i.e., while transaction processing, including updates, is going on). Such a fuzzy checkpoint is initiated by writing a begin-chkpt record. Then the end– chkpt record is constructed by including in it the contents of the normal transaction table, the BP dirty-pages table, and any file mapping informa- tion for the objects (like tablespace, indexspace, etc.) that are “open” (i.e., for which BP dirty–pages table has entries). Only for simplicity of exposition, we assume that all the information can be accommodated in a single end- chkpt record. It is easy to deal with the case where multiple records are needed to log this information. Once the end-chkpt record is constructed, it is written to the log. Once that record reaches stable storage, the LSN of the begin-chkpt record is stored in the master record which is in a well-known place on stable storage. If a failure were to occur before the end–chkpt record migrates to stable storage, but after the begin _chkpt record migrates to stable storage, then that checkpoint is considered an incomplete checkpoint. Between the begin--chkpt and end. chkpt log records, transactions might have written other log records. If one or more transactions are likely to remain in the in-doubt state for a long time because of prolonged loss of contact with the commit coordinator, then it is a good idea to include in the end-chkpt record information about the update-type locks (e.g., X, IX and SIX) held by those transactions. This way, if a failure were to occur, then, during restart recovery, those locks could be reacquired without having to access the prepare records of those transactions. Since latches may need to be acquired to read the dirty _pages table correctly while gathering the needed information, it is a good idea to gather the information a little at a time to reduce contention on the tables. For example, if the dirty _pages table has 1000 rows, during each latch acquisi- tion 100 entries can be examined. If the already examined entries change before the end of the checkpoint, the recovery algorithms remain correct (see Figure 10). This is because, in computing the restart redo point, besides taking into account the minimum of the RecLSNs of the dirty pages included in the end_chkpt record, ARIES also takes into account the log records that were written by transactions since the beginning of the checkpoint. This is important because the effect of some of the updates that were performed since ACM Transactions on Database Systems, Vol. 17, No. 1, March 1992. 

29 .122 . C. Mohan et al. the initiation of the checkpoint might not be reflected in the dirty page list that is recorded as part of the checkpoint. ARIES does not require that any dirty pages be forced to nonvolatile storage during a checkpoint. The assumption is that the buffer manager is, on a continuous basis, writing out dirty pages in the background using system processes. The buffer manager can batch the writes and write multi - ple pages in one 1/0 operation. [961 gives details about how DB2 manages its buffer pools in this fashion. Even if there are some hot-spot pages which are frequently modified, the buffer manager has to ensure that those pages are written to nonvolatile storage reasonably often to reduce restart redo work, just in case a system failure were to occur. To avoid the prevention of updates to such hot-spot pages during an 1/0 operation, the buffer manager could make a copy of each of those pages and perform the 1/0 from the copy. This minimizes the data unavailability time for writes. 6. RESTART PROCESSING When the transaction system restarts after a failure, recovery needs to be performed to bring the data to a consistent state and ensure the atomicity and durability properties of transactions. Figure 9 describes the RESTART routine that gets invoked at the beginning of the restart of a failed system. The input to this routine is the LSN of the master record which contains the pointer to the begin .chkpt record of the last complete checkpoint taken before site failure or shutdown. This routine invokes the routines for the analysis pass, the redo pass and the undo pass, in that order. The buffer pool dirty _pages table is updated appropriately. At the end of restart recovery, a checkpoint is taken. For high availability, the duration of restart processing must be as short as possible. One way of accomplishing this is by exploiting parallelism during the redo and undo passes. Only if parallelism is going to be employed is it necessary to latch pages before they are modified during restart recovery. Ideas for improving data availability by allowing new transaction processing during recovery are explored in [601. 6.1 Analysis Pass The first pass of the log that is made during restart recovery is the analysis pass. Figure 10 describes the RESTART_ ANALYSIS routine that imple- ments the analysis pass actions. The input to this routine is the LSN of the master record. The outputs of this routine are the transaction table, which contains the list of transactions which were in the in-doubt or unprepared state at the time of system failure or shutdown; the dirty–pages table, which contains the list of pages that were potentially dirty in the buffers when the system failed or was shut down; and the RedoLSN, which is the location on the log from which the redo pass must start processing the log. The only log records that may be written by this routine are end records for transactions that had totally rolled back before system failure, but for whom end records are missing. ACM ‘llansactlons on Database Systems, Vol. 17, No. 1, March 1992.