OS Structures

1 . Goals for Today CS194-24 • Finish up our discussion about OS basics Advanced Operating Systems • Test-Driven Design, Behavior-Driven Design Structures and Implementation • Where are we going next? Lecture 2 Interactive is important! OS Structure: Intro Ask Questions! TDD, BDD, and all that January 27th, 2014 Prof. John Kubiatowicz http://inst.eecs.berkeley.edu/~cs194-24 Note: Some slides and/or pictures in the following are adapted from slides ©2013 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.2 Review: Virtual Machines Review: Virtual Machines (con’t): Layers of OSs • Software emulation of an abstract machine • Useful for OS development – Make it look like hardware has features you want – When OS crashes, restricted to one VM – Programs from one hardware & OS on another one – Can aid testing programs on other OSs • Programming simplicity – Each process thinks it has all memory/CPU time – Each process thinks it owns all devices – Different Devices appear to have same interface – Device Interfaces more powerful than raw hardware » Bitmapped display  windowing system » Ethernet card  reliable, ordered, networking (TCP/IP) • Fault Isolation – Processes unable to directly impact other processes – Bugs cannot crash whole machine • Protection and Portability – Stability of POSIX interface between systems – Limits to what Users programs are allowed to do 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.3 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.4 

2 . Review: How to work on OSes easily? What does an Operating System do? • Traditional: • Silerschatz and Gavin: “An OS is Similar to a government” – Sit at serial console, – Begs the question: does a government do anything useful by – Upload new OS image somehow itself? – Reboot and possibly crash (“Panic”) • Coordinator and Traffic Cop: – Debug with very limited tools – Manages all resources – Settles conflicting requests for resources • How we will do it in this class: Virtual Machines! – Prevent errors and improper use of the computer • In fact – Nested Virtual Machines: • Facilitator: – Provides facilities that everyone needs VMware – Standard Libraries, Windowing systems KVM OS Under Test – Make application programming easier, faster, less error-prone Virtual Machine Virtual Machine (Experimental) • Some features reflect both tasks: – E.g. File system is needed by everyone (Facilitator) Linux Development Environment – But File system must be Protected (Traffic Cop) 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.5 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.6 What is an Operating System,… Really? Operating System Definition (Cont.) • Components: • No universally accepted definition – Memory Management • “Everything a vendor ships when you order an operating – I/O Management system” is good approximation – CPU Scheduling – But varies wildly – Communications? (Does Email belong in OS?) • “The one program running at all times on the – Multitasking/multiprogramming? computer” is the kernel. – Everything else is either a system program (ships with • What about? the operating system) or an application program – File System? • Studying OSes is really about the Hardware/Software – Multimedia Support? interface (API) – User Interface? – Thus, we will hope to give you enough knowledge to: – Internet Browser?  » Understand this interface » Modify this interface • Is this only interesting to Academics?? » Change the support underneath the interface 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.7 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.8 

3 . Review: Protecting Processes from Each Other Address Translation • Address Space • Problem: Run multiple applications in such a way that they are protected from one another – A group of memory addresses usable by something – Each program (process) and kernel has potentially • Goal: different address spaces. – Keep User Programs from Crashing OS • Address Translation: – Keep User Programs from Crashing each other – Translate from Virtual Addresses (emitted by CPU) – [Keep Parts of OS from crashing other parts?] into Physical Addresses (of memory) • (Some of the required) Mechanisms: – Mapping often performed in Hardware by Memory – Address Translation Management Unit (MMU) – Dual Mode Operation Virtual Physical Addresses Addresses • Simple Policy: CPU MMU – Programs are not allowed to read/write memory of other Programs or of Operating System 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.9 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.10 Example of Address Translation Address Translation Details Data 2 • For now, assume translation happens with table Code Code Stack 1 (called a Page Table): Data Data Heap Heap 1 10 Heap Virtual V page no. offset Address Stack Code 1 Stack Stack 2 Page Table Prog 1 Prog 2 Data 1 Virtual Virtual index V Access Rights PA Address Heap 2 Address into Space 1 Space 2 page Code 2 table table located in physical P page no. offset Physical OS code memory 10 Address OS data • Translation helps protection: Translation Map 1 Translation Map 2 – Control translations, control access OS heap & Stacks – Should Users be able to change Page Table??? Physical Address Space 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.11 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.12 

4 . Dual Mode Operation UNIX System Structure • Hardware provides at least two modes: – “Kernel” mode (or “supervisor” or “protected”) – “User” mode: Normal programs executed Applications User Mode • Some instructions/ops prohibited in user mode: Standard Libs – Example: cannot modify page tables in user mode » Attempt to modify  Exception generated • Transitions from user mode to kernel mode: Kernel Mode – System Calls, Interrupts, Other exceptions Hardware 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.13 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.14 Administrivia Administrivia (Con’t) • If you don’t have an account form, get one from Vedant or • You should be well on the way to reading the Cucumber book! Palmer today: – Reading suggestions are up on the lecture page – Need it get VMware license – In fact, I suggest that you work through the calculator example – Log into your email account on http://inst.eecs.berkeley.edu in the book! • For Redmine accounts: • Get moving on the other suggested readings as well – I’ll put up readings in the lecture schedule for the day we talk – Click on Google logo and use your XXX@berkeley.edu address! about them (look ahead!) – Sometimes Google Chrome doesn’t give you a chance to choose • Don’t worry about groups until Monday which google account you are using unless you logout first. – I will put up the official web form by then – If you get a comment indicating that a new account has been • You should have already started on Project 0! created for you, you have probably logged in incorrectly. – Project 0 is due on Wednesday (in two days!) • Redmine project development site – Project 0.5 starts on Wednesday (in two days!) – We are using a Redmine project development site for all – Lots of “stuff” to set up about your environment resource control, bug tracking, etc • Why do we start with Project 0/0.5 (individually)? – Your course account gives you access to the server – We want everyone to be productive with the tools so that we can get down to the good stuff » Log in right away and update your name/email – We want to shake out the bugs in the infrastructure » Generate an ssh key and upload that to the server • Check out “Resources” link off the home page » Remote access to git repositories. – It gives some additional resources that might be useful 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.15 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.16 

5 . Map of Linux Components Linux Structure (More as term progresses!) (http://www.makelinux.net/kernel_map/) 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.17 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.18 History of Unix Microsoft Windows Structure 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.19 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.20 

6 . Major Windows Components History of Windows • Hardware Abstraction Layer – Hides hardware chipset differences from upper levels of OS • Kernel Layer – Thread Scheduling – Low-Level Processors Synchronization – Interrupt/Exception Handling – Switching between User/Kernel Mode. • Executive – Set of services that all environmental subsystems need » Object Manager » Virtual Memory Manager » Process Manager » Advanced Local Procedure Call Facility » I/O manager » Cache Manager » Security Reference Monitor » Plug-and-Plan and Power Managers » Registry » Booting • Programmer Interface: Win32 API 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.21 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.22 Brave New World: The Swarm? Example: a Smart Space An Application Model • Displays Everywhere – Walls, Tables, Appliances, Smart Phones, Google Glasses…. Channel Sensors • Audio Output Everywhere Real-Time with Components • Inputs Everywhere Transform Aggregation SwarmLet and Archive – Touch Surfaces (“The Application”) Distributed – Cameras/ Archival Gesture Tracking • A Swarm Application is a Storage – Voice Connected graph of Components • Context Tracking – Globally distributed, but locality and QoS aware – Who is Where – Avoid Stovepipe solutions through reusability – What do they want • Many components are Shared Services written by – Which Inputs map to which applications programmers with a variety of skill-sets and motivations • How do we hope to organize this complexity? – Well-defined semantics and a managed software version scheme – Not via Stovepipe solutions! Today’s typical solution! – Service Level Agreements (SLA) with micropayments – Need something more global! • Many are “Swarmlets” written by domain programmers 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.23 – They care what application 1/27/14 does, ©UCB Kubiatowicz CS194-24 how2014 not Fall it does it Lec 2.24 

7 . Meeting the needs of the Swarm Recall: Increasing Software Complexity Cloud Services Personal/Local Swarm Metropolitan Middleware • Discover and Manage resource • Integrate sensors, portable devices, cloud components • Guarantee responsiveness, real-time behavior, throughput • Self-adapt to failure and provide performance predictability • Secure, high-performance, durable, available information From MIT’s 6.033 course • Monetize resources when necessary: micropayments 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.25 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.26 What are typical problems with development? What is wrong with traditional development? • Delivering the “Wrong Thing” • Rigid framework for development – Planning phase: – Spend lots of time designing things only to discover » How many people, what resources, etc? that your solution doesn’t do what you need! – Analysis phase: – Development process often divorced from » Try to articulate in detail problem trying to solve » Ideally without prescribing how it should be solved (almost never “stakeholders”, namely the people who know what the happens) software actually needs to do – Design phase: » Think about design and architecture • Unstable in Production » Standards to use, large and small technical decisions – Works great when you experiment with it » Decompose problem into manageable chunks to produce functional specifications – Doesn’t work well in the field – Coding phase: » Write the software according to the spec • Costly to maintain » In theory, all the “hard thinking” already done – Software is brittle and each new feature causes you » (Why programming and testing often sent off-shore to third parties) to break previous features – Testing phase: – Unexpected regressions are common » We save testing until late in the process – Deployment » Here we finally send the product out to users 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.27 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.28 

8 . In reality, this doesn’t really work well Why is software designed this way? • Why so much structure and ceremony? • Perhaps in analogy with Civil projects – Because traditionally the later in the process that we discover a problem, the more expensive it is to fix! – Need to be really sure you are doing the right thing when building a bridge – hard to go back a redesign the support – Each piece done by different team, thus phases structure after you poured the cement! represent handoff from one set of people to another • But – Software is SOFT • In reality, there is much back and forth between analysis, design, and coding – Perhaps a development process that reflects the nature of software is better? – As design or coding proceeds, problems are discovered forcing redesign of major components – Software is brittle only if it is design with a rigid process! – These changes force increasingly complex • The Agile Manifesto instead: communication between teams – We have come to value: • This complex interaction makes it increasingly unlikely » Individuals and interactions over processess and tools that changes will happen through official channels » Working software over comprehensive documentation – So, work done outside process » Customer collaboration over contract negotiation – Documents get out of sync with software itself » Responding to change over following a plan – Testing gets squeezed – While there is value in the things on the right, we value the things on the left more 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.29 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.30 One step forward: Test-Driven Development (TDD) From TDD  BDD • What is problem with bare-bones TDD? Write Minimimum – Where to start? amount of – What to test or not test? Code to Pass – What tests to write? – How much to test at a time? Write Refactor Code – What should the test files be called? Failing While Still Passing Test Repeat as necessary (DRY out code) • Much better: Check for Behavior rather than Testing – Now, the checking that is done is to show that a particular behavior wanted by someone is happening • Test-driven development (TDD) is a software development » Should be performed at higher level (APIs, User Interface, etc) and independent of implementation process that relies on the repetition of a very short development cycle: • Need a language (DSL) to express Behavior and way to automate verification of behavior – First the developer writes an (initially failing) automated test – For instance – Cucumber! case that defines a desired improvement or new function, – Expresses human-readable analysis and executable – Then produces the minimum amount of code to pass that acceptance tests test, and – Write the minimum amount of code required to meet your behavioral checks – Finally refactors the new code to acceptable standards. » Don’t write code you don’t need! • Key thing – Tests come before Code » If behavior isn’t specified, don’t bother writing code for it 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.31 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.32 

9 . A Ubiquitous Language for Analysis Acceptance Criteria Should Be Executable! • Need a framework for analyzing the process: • Cucumber provides an execution environment for – As a [X] Acceptance tests: – I want [Y] – so that [Z] development_directory/features: *.feature # Cucumber Files • Then, need a way of expressing the acceptance criteria in terms step_definitions/*.rb # Step Definitions of scenarios: support/*.rb # Supporting code – Given some initial context (the givens), • How does this all connect? – When an event occurs, – Files in ‘support’ get loaded early, set up environment before – Then ensure some outcomes starting execution and hooks to execute before and after • Example in cucumber (called, say “valid_card_withdrawal.feature”) each scenario – Files in ‘step_definitions’ are global definitions of what to do Feature: The Customer tries to withdraw cache using valid ATM card when a particular step (Given, When, Then, And, But) happens As a customer, I want to withdraw cache from an ATM • Step definitions should be treated like you are designing so that I don’t have to wait in line at the bank scenario: Successful Cache Withdrawal your own language! Given I have an ATM card that is owned by me – They also connect the high-level language of feature files to When I request $40 and my account has enough money the actual implementation Then I will receive $40 scenario: Unsuccessful Cache Withdrawal – They may need to tweak the design in interesting ways Given I have an ATM card that is owned by me • Step definitions typically call code in the implementation When I request $40 And my account does not have enough money before it has been implemented! Then I will receive an error – Write the code “you wish you had” 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.33 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.34 What do Step definitions look like? How does this work in practice? • What do these steps translate into? Cucumber Step Given I have an ATM card that is owned by me When I request $40 Execution Connector Definition and my account has enough money Then I will receive $40 • Can use Step definitions to call out across interfaces: • Answer: Regular expressions in a step file: – Project 0 – Given /^I have an ATM card that is owned by me$/ do » Before and After hooks in ‘support/hooks.rb’ start up and shut down virtual machine; For autograder, will perform # Set up machine with card and valid PIN @my_account ||= Account.new end “git pull” and “make” on your kernel as well! When /^I request \$(\d+)$/ do |amount| » Step definitions use custom protocol across serial interface to communicate with virtual machine @my_request = amount end And /^my account has enough money$/ – Cucumber-cpp @my_account.balance.should <= @my_request end » Uses “Cucumber wire protocol” to send steps across Then /^I will receive \$(\d+)$/ do |amount| TCP/IP channel to Step definitions written in C++! end @my_account.request_money(@my_request).should = amount » “GIVEN, WHEN, THEN” are c-preprocessor macros! – Cucumber+Capybara • Steps interact with actual implementation » Adds special DSL to Steps for talking with various web – Reference code you “wish you had”, not “code you already have” browsers (can speak about content returned, etc) 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.35 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.36 

10 . Amusing example: Verify Apple-II Cucumber tests for Apple II: Video by Armando Fox • Start with Apple-II Emulator, then add BDD testing with Cucumber (Thanks to Armando Fox): https://github.com/armandofox/cucumber-virtualii Feature: enter and run a short BASIC program As a beginning programmer in the late 1970's So that I can get excited about CS and become a professor someday I want to learn BASIC by entering and running Scenario: enter and run a Fibonacci program simple programs When I enter the following program: | lines | Background: The Apple II is booted and the | 10 INPUT "COMPUTE FIBONACCI NUMBER "; F | BASIC interpreter is activated | 20 N1 = 1 : N2 = 1 | Given there is no current BASIC program | 30 FOR I = F TO 3 STEP -1 | | 40 T = N2 | Scenario: enter and run Hello World | 50 N2 = N2 + N1 | When I enter the following program: | 60 N1 = T | | lines | | 70 NEXT I | | 10 HOME | | 80 PRINT "RESULT IS "; N2 | | 20 PRINT "HELLO WORLD!" | And I type "RUN" And I clear the screen Then I should see "COMPUTE FIBONACCI NUMBER" And I type "RUN" When I type "6" Then I should see "HELLO WORLD!" Then I should see "RESULT IS 8" 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.37 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.38 Verification Methodology How Agile Methods Address Project Risks • Need for both User Stories (Behaviors) and Component Tests • No longer Delivering Late or Over Budget (Unit testing) – Deliver system in tiny, one- or two-week iterations (or mini- – Behavioral Tests represent desired behavior from standpoint of projects) stakeholders and involve whole code base » Executable documentation! – Always have a working release » Slower, whole-system acceptance testing – Know exactly how much it costs » Run after every change • No Longer Delivering the Wrong Thing – Unit testing frameworks (Like Rspec, CUnix, CPPSpec, etc) – Can demonstrate new features to stakeholders and make any thoroughly test modules tweaks or correct any misunderstandings while work fresh in » Fast execution developer’s minds » Only run tests when change actual module • No Longer Unstable in Production • Behavioral tests – Deliver something on every iteration – High-level description independent of implementation – Must get good at building and deploying the application – Test files named for behaviors being tested » Releasing to production or testing hardware just another build to just another environment » When failures happen, know where to start looking » Rely on software automation to manage this – Always in sync with code: tests run after every change – Application servers automatically configured, database schemas – JBehave, Cucumber, etc automatically updated, code automatically built, assembled, and • Unit tests deployed – Express individual details of implementation – All types of tests automatically executed to ensure system working – Consider writing one or more unit test for every module • No Longer Costly to Maintain – Can use CPPSpec, Cunit, etc. – With first iteration –team is effectively in maintenance mode! – Can be systematic, catch corner cases, etc – Adding code to a working system, so they have to be very careful 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.39 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.40 

11 . Conclusion • Studying OSes is really about the Hardware/Software interface (API) – Thus, we will hope to give you enough knowledge to: » Understand this interface » Modify this interface » Change the support underneath the interface • Test-Driven Development (TDD) – Write tests first, then write code, then refactor • Behavior-Driven Development (BDD) – Instead of Tests, think about writing Executable Behavior specifications – Cucumber for Integration Behaviors, Unit tests for implementation. 1/27/14 Kubiatowicz CS194-24 ©UCB Fall 2014 Lec 2.41