Retrofitting Memory Protection in the Zephyr OS

Modern microcontrollers offer mechanisms to protect memory regions from being inadvertently accessed, either by programmer error or by malice. Until recently, Zephyr was not making use of it. This talk will present the work performed to retrofit memory protection in the Zephyr kernel, discussing the constraints, design decisions (affecting portability, security, and performance), its current state, and the next steps. We will discuss details on a novel method of tracking and validating kernel objects, techniques for defining system calls with a minimal amount of boilerplate code, details on our APIs for managing memory permissions, and details on the implementation of the protection feature for the ARC architecture.

1.Retrofitting Zephyr Memory Protection Wayne Ren, Sensor Software Engineer, Synopsys

2.Acknowledgements These slides are based on Andrew Boie (Intel)’s “Retrofitting Zephyr Memory Protection” Thank you, Andrew!

3. 01 Background Design and Implementation in CONTENTS 02 Zephyr 03 ARC Specific Implementation 04 Future Work


5.What is Zephyr • Zephyr: a modular RTOS and a complete solution stack – RTOS for use on connected resource-constrained and embedded devices – Focused on safety, security, connections with Bluetooth support and a full native networking stack – Apache 2.0 license, hosted at Linux Foundation – Support diverse use cases and architectures: ARC, ARM, RISC-V, X86 … – Web site: • More Zephyr related events: – “Introduction to the Zephyr Project” - Ryan Qian, NXP & Kate Stewart, The Linux Foundation, Tuesday, June 26 • 11:20 - 12:00 – “License Information Management ” – Kate Stewart, The Linux Foundation, Tuesday, June 25 • 15:50 - 16:30 – IoT Meetup, Tuesday, June 26 • 18:00 - 21:00, 北京海淀区科学院南路2号, 融科资讯中心C座南楼1层 南极洲会议室

6.Why Required • Security concern of IoT devices – More and more “things” are connected, traditionally offline->online • Secure communication • Trusted execution (secure boot) • Data protection • …. • Zephyr uses systematic approaches for security – Static: high quality design, code review, tests, certification – Dynamic: secure communication, cryptography, memory protection • Memory protection – One important approach for more secure, reliable and safe system – 1st step to implement security

7.Memory Protection Hardware Memory Protection Unit(MPU) Memory Management Unit(MMU) • Popular in low-end device, ARM Cortex M4, ARC • Popular in Application processors, x86, ARM Cortex EM A series, ARC HS series • Fixed number of configurable regions, each with • Address space divided into equal sized pages their own access policy (typically 4K). • No virtualization, physical memory addresses • Configuration for caching and access, policy for each page set in page tables • Typically have constraints on region specification, • MPU-like behavior with identity page table, but e.g. region sizes must be power of two, aligned to Optional support for virtual memory their size

8.Memory Protection in Zephyr • Zephyr had no means of preventing unwanted memory access before • Joint effort with most contributions from Linaro (ARM), Synopsys (ARC), and Intel (x86), initial efforts targeting MPU-based systems • Milestones – 1.9 release (7/2017): MMU/MPU enabled, stack overflow protection on ARM/x86 – 1.10 release (11/2017): user mode support on x86 MMU – 1.11 release (3/2018): user mode support on ARC/ARM MPU – 1.12 release (6/2018): more tests, refinement • Future work – Additional CPU architecture support – Flesh out APIs and iterative refinement – Support of TEE (Trusted Execution Environment), e.g., secure and non-secure world (1.13 or later)

9.Use Cases • Protect against unintentional programming errors – Stack overflows – Writing to bad memory – Data corruption • Sandbox complex data parsers and interpreters – Network stacks/protocols – File systems – Reduce likelihood of third-party data compromising the system • Support the notion of multiple logical isolated applications

10.Comparison with Other RTOSes • FreeRTOS-MPU • NuttX Protected Build • not default configuration of • Supports ARM MPU and MMU (with FreeRTOS identity page table) • Unprivileged "User" threads with • Unprivileged threads similar to configurable memory access, system FreeRTOS-MPU calls for privileged operations • Separately loaded applications • Not well maintained, often doesn't • Many features proposed but still compile WIP • ThreadX Modules • Zephyr • MPU or MMU Virtualized address • Thread-level protection spaces for separately loaded • Support for lost of different CPUS modules with thread-level memory • Same kernel & driver APIs for kernel protection features and user mode threads • Support for lots of different CPUs • Free, Apache 2.0 License • Not free. Royalty-free license with • More features in the future significant upfront cost, modules feature costs extra

11.Design and Implementation in Zephyr

12.Threat Model • User thread – Untrusted – Isolated from the kernel and each other User Kernel threads threads • Kernel thread and kernel MPU hardware separation – Trusted, privilege to access all CPU mode separation Unprivileged/User thread 1 thread 2 • A flawed or malicious user thread thread 3 cannot: – Leak or modify private data of another thread 4 thread 5 thread 6 thread unless specifically granted permission Zephyr Kernel – Interfere with or control another thread except through designed thread Privileged/Kernel communication APIs (pipes, semaphores, etc.)

13.User Mode • Control access to kernel objects and device drivers – Per-object and per-thread basis • Maintain compatibility with existing Zephyr APIs • Implement system calls for privilege elevation • Arch-specific code to enter user mode • Validate system call parameters including kernel object pointers • Do not require changes to individual drivers • Manage user mode access to memory

14.High-Level Policy • User threads are by default granted only – Read/write access to their own stack memory and application memory – Read-only/execute access to program text and ROM – Memory domain APIs to configure access to additional regions with child thread inheritance • User threads cannot use device drivers or kernel objects without being granted permission – Permission granted by other threads with sufficient permission or inherited • System call API parameters are rigorously checked • User mode stack overflows are safely caught

15.Permission Model • Each kernel object has a bitfield indicating what user threads have access to it Kernel object • Kernel threads can grant object access to any user thread 01100110 0 • User threads may grant object access to Thread another user thread if the calling thread has permissions on both the object and the Thread target user thread Thread • Newly created user threads may optionally inherit object permissions of the parent thread.

16.Kernel Object • Three main types of kernel-private data structures – Kernel API data structures - k_thread, k_sem, k_mutex, k_pipe, etc. – All device driver instances – All thread stacks, instead of individual structs, these are arrays of a special typedef to character data • To preserve Zephyr API compatibility, all are referenced by memory address – Act as a handle for user threads, object memory not accessible – Need a system for validating object pointers passed to system calls

17.Kernel Object Permissions • Kernel threads can access all objects – Permissions still tracked, because • Thread drops to user mode • Creates child user threads with object permission inheritance enabled – May designate some objects as "public" and usable by all threads • User threads – If created with permission inheritance, gain access to all parent thread's permissions except parent thread object – k_object_access_grant() calls must have permission on both the target thread and the object being granted permission to

18.Kernel Object: New Type • Creating new kernel object types is easy! – Add the name of the associated data structure to the build • Struct name itself for new kernel APIs • API struct name for new device driver subsystem types – Small modifications to some lists in two C files • Could eventually be automated • Recognizing instances of kernel objects and providing a validation function for them is all handled automatically at build time

19.Kernel Object: Constraints • Must be declared as a top-level global – Needs to appear in the kernel's ELF symbol table – OK to declare with static scope – May be embedded as members of larger data structures • Memory for an object must be exclusive to that object – Can't be part of a union data type • Must be in the kernel data section • Objects that do not meet these constraints will not be accessible from user mode • Future work: support runtime allocation use-cases from slabs/kernel heap

20.Kernel Object: Definition • perms: permission bitfield indicating thread permissions for that object • type: object type information struct _k_object { enum, K_OBJ_THREAD, char *name; K_OBJ_UART_DRIVER, ... u8_t perms[CONFIG_MAX_THREAD_BYTES]; u8_t type; • flags: initialization state, u8_t flags; public/private, others as needed u32_t data; } • data: extra data in some cases – Stack object size extern struct _k_object * – Build-time assigned thread ID _k_object_find(void *obj);

21.How to Get Kernel Object Info? • Problem: need to find all the kernel objects – Map object memory addresses to instantiations of struct _k_object containing metadata – Validate kernel object pointers passed in from user thread • Solution: – • Use pyelftools to unpack ELF binary and fetches all the DWARF debug information, and does object identification – gperf • a GNU tool for creating perfect hash tables • Generate the hash table of kernel objects for efficiency

22.Kernel Object: Flow zephyr pre-built.elf kobject_hash.gperf kobject_hash_prep gperf tool (1st build) rocess.c zephyr.elf kobject_hash.c 2nd build preprocess tool

23.System Call • Typical OS mechanism for allowing user threads to perform operations they can’t do User mode User threads • On all arches, API ID and parameters are marshaled into registers and a software interrupt/exception is triggered System call wrapper – Up to six registers used; additional args passed in via SW irq/exception irq/exception return struct and stack call checker clean up • Common landing site for system calls on kernel side Kernel mode – Validate API ID, execute the handler function – Clean general purpose registers on exit to prevent Zephyr Kernel private data leakage • Use build-time logic to make adding new system calls as painless as possible

24.System Call: Components • Very easy for developers to define • Created by developer for each system call: – System call header prototype __syscall void k_sleep(s32_t duration) – Handler function for argument validation Z_SYSCALL_HANDLER(k_sleep, duration) • Verify caller permissions on provided memory buffers or data passed via pointer • Copy any parameter data passed in via pointer to local memory • Verify object pointers, permission, initialization state • Verify parameter values which are otherwise left to assertions or simply un-checked – Implementation function void _impl_k_sleep(s32_t duration) • Kernel object API code under kernel/ • Driver subsystem API functions defined at the subsystem level • Auto-generated for each system call: – System call ID enumerated type – Handler function prototypes – _k_syscall_table entry mapping ID to handler function – __weak handler function for system calls excluded from kernel config – System call invocation function

25.System Call: Flow k_oops() N User Y Marshal Valid Y Lookup handler in API Call Mode args, Trigger call dispatch table ? SW IRQ ID? N Return to Marshal Return Implementation Y Handler Caller Value, exit IRQ Function Checks N k_oops()

26.System Call: Build-Time Magic • Limited parsing of kernel header files, looking for function prototypes prefixed with "__syscall“ • Parsing limited to determining return value and argument types to generate additional functions – Some minor limitations in parsing with array/function pointer argument types which can be easily worked around • Generated headers contains implementation of API as an inline function - invokes system call trap or direct call to implementation as appropriate • Some generated C code for default handler and dispatch table entry

27.Memory Domain • User threads by default can't look at any RAM except their own stacks • Need a flexible way to designate additional memory areas that a thread has access to • Limited number of total MPU regions needs to be taken into consideration • Grant access to top-level data or BSS section globals defined and used by the thread, or application data that needs to be shared between threads • Memory Domain APIs exist to handle re-programming the MPU for the incoming thread's memory access policy on context switch

28.Memory Domain: Implementation • Memory domain APIs are kernel-access only, no system calls mem_domain 0 • Implemented as an object struct k_mem_domain – Contains some number of memory partitions (struct mem partition 0 k_mem_partition) • Up to the maximum number of regions supported by MPU hardware, no limit for MMU • Each partition is a starting address, size, and access policy mem partition n • Hardware dictates alignment and size constraints – APIs to add/remove partitions to an initialized memory domain object • Any thread may be added/removed to a particular memory domain to implement an access policy for mem_domain 1 that thread • MPU region registers or MMU page tables updated mem partition 0 upon context switch to activate policy for incoming thread • Special Case: Application Memory mem partition n – Shared to all threads, CONFIG_APPLICATION_MEMORY in Kconfig – All top level globals in non-kernel object files (libs, application code) placed in user read/writable section by linker and access policy configured in MMU/MPU at boot • Facilities for grouping data by the linker (WIP)

29.ARC Specific Implementation