Using Docker in QEMU Testing

The QEMU project has adopted a Docker-based approach to run tests since a few years ago, and it works! In this talk, Fam will review this experience and explain why and how we did that, covering the interesting details, the benefits and the challenges we have had. He will then summarize how the tight integration of Docker-based testing, especially for Continuous Integration purposes, could apply generically to other projects.

1.Using Docker in QEMU Testing Fam Zheng Senior Software Engineer LC3-2018

2. Outline • Introduction • The devils^Wdetails • Integrating into the build-sys • Passing the source tree • ccache • Dockerfile checksum • Handling image dependencies • Using qemu-user • Cross build environment • Local Travis tests • Docker testing in VM • Summary 2


4. Introduction • Why use Docker to test? • CI testing requirements • Reproducibility → Widely available in distros • Self-contained → Dockerfile • Coverage → DockerHub resources • Docker testing framework in QEMU • Since QEMU 2.7 (2016.9) • Used to test patches on (with • 24 images • 7 test scripts 4

5. QEMU Docker testing framework Features • Proved powerful and flexible • Actively used and developed since introduction • Different build env • Fedora / CentOS / Ubuntu / Debian • Different compilers • gcc vs clang • Cross build • win32 (mingw) • arm/mips/ppc/s390 • Dependent libraries • min-glib • Replicate .travis.yml testing • qemu-user enabled cross 5

6.The implementation

7. Integrating into the build-sys • ‘make docker’ to print the help text • Tests can run from a clean source tree, no need to ./configure • Special rules are created for (test, image) combination: • make docker-test-mingw@fedora • make docker-test-clang@ubuntu • Filter tests/images with env var TESTS=”test-mingw test-clang” IMAGES=”centos6 debian” make docker-test • A wrapper script to manage docker command line details • tests/docker/ • An entry script to set up environment in container • tests/docker/run 7

8. Passing the source tree • What’s wrong with passing the source tree with --volume ? • The container mustn’t modify source tree (incl. creating files) • Do out-of-tree build there? Won’t work if there is in-tree build files. • SELinux may prevent build from working • Source tree is copied carefully, including submodules: • git diff-index HEAD → check if the source tree is clean • if yes → use HEAD • if not → git stash create • git clone -- shared → efficient source tree copy • git submodule update → submodules are needed for build • git ls-files > $list_file → prepare file list for creating src archive • tar -cf $tar_file -T $list_file → archive source to pass to container • The ‘run’ script extracts the tar file in container 8

9. Persistent ccache DB • To speed up repetitive/incremental builds • A directory is created on host for persistence • DOCKER_CCACHE_DIR=$HOME/.cache/qemu-docker-ccache • Set CCACHE env var to let the container use it • docker run … -v $DOCKER_CCACHE_DIR:/var/tmp/ccache -e CCACHE_DIR=/var/tmp/ccache ... 9

10. Dockerfile checksum • Similar to the docker-build cache • Checksum of • Dockerfile content • Added file from host (fix pending) • Parent image’s checksum • Appends a • LABEL com.qemu.dockerfile-checksum=XXXX line to the Dockerfile during image build • Compared before running tests → only rebuild the image if mismatch • Faster than invoking ‘docker build’ each time 10

11. Handling image dependencies • FROM directive are specially handled to work with images too: • E.g. debian-armhf-cross.docker: FROM qemu:debian9 • Checksums are considered recursively • Makefile rules are needed to express dependencies: • docker-image-debian-armhf-cross: docker-image-debian9 11

12. Using qemu-user • qemu-user is “user mode emulation of qemu target” • Executes programs built for other architectures. e.g. run ARM binary on x86 machine • A bit tricky to set up • Need to register the interpreter to host binfmt_misc • Must have the right run-time libraries to run programs • Enabled by Docker: • The Debian Bootstrap image automates everything for you • Usage: $ dnf install fakeroot debootstrap qemu-user qemu-user- binfmt $ EXECUTABLE=/usr/bin/qemu-aarch64 DEB_TYPE=testing DEB_ARCH=arm64 make docker-image-debian-bootstrap V=1 $ make docker-test-build@debian-bootstrap 12

13. Cross build environments • Supports a range • arm64/armel/armhf/mips64el/mips/mipsel/powerpc/ppc64el/s390x/ win32/win64 • Using Debian foreign architecture repos. E.g. FROM qemu:debian9 RUN dpkg --add-architecture arm64 RUN apt-get install -y crossbuild-essential-arm64 RUN apt-get build-dep -yy -a arm64 qemu RUN apt-get install -y libbz2-dev:arm64 liblzo2-dev:arm64 ... ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu- • Usage: make docker-test-build@debian-powerpc-cross V=1 J=8 13

14. Local Travis tests • About Travis-CI • is a hosted CI service that can do continuous build and test for your project • Per-project .travis.yml to define the build/test matrix • Different compilers: gcc, clang • Different configure options: feature selection, compiler flags • Different OSes: OSX/ubuntu • Sometimes hard to debug when build fails • A Docker ‘travis’ script is added so you can “replicate” the matrix locally, as simple as • make docker-travis@travis 14

15. Local Travis tests • Implemented with ~50 LOC • Used the image that is the same env as on • A script is written to parse (the main parts of) .travis.yml 15

16. Docker in VM • Problem: hanging tests are hard to clean up • Stalls CI from time to time • Solution: wrapping docker tests in a VM • VM based testing was introduced in QEMU in 2017 initially for non-Linux_x86_64 builds (*BSD and i386) $ make vm-test vm-test: Test QEMU in preconfigured virtual machines vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM vm-build-freebsd - Build QEMU in FreeBSD VM vm-build-netbsd - Build QEMU in NetBSD VM vm-build-openbsd - Build QEMU in OpenBSD VM 16

17. Docker in VM • TODO: • Add a x86_64 Linux image just to run Docker tests • Add a mechanism to use persistent cache • both for Docker images in the VM and ccache data • Unprivileged user can run it now! (with access to /dev/kvm) • Proposed usage: $ make vm-build-centos V=1 J=8 • Patches on • msg00248.html 17

18. Summary • Why you should use Docker to test your projects? • Easily cover multiple setups • Very extensible • Easy to maintain • Things can be improved (suggestions welcome!) • Copying source code from host to container is awkward • Still fairly Linux+x86_64 focused • Clean up is harder when things go wrong • Docker command requires privilege to run • Make -j from host has no effective in container (as a workaround, a magic env var J=$N is defined) 18

19. Credits $ git shortlog -nse tests/docker 55 Fam Zheng <> 44 Philippe Mathieu-Daudé <> 26 Alex Bennée <> 13 Paolo Bonzini <> 8 Sascha Silbe <> 3 Eduardo Habkost <> 3 Marc-André Lureau <> 2 Daniel P. Berrangé <> 1 Greg Kurz <> 1 Peter Maydell <> 1 Peter Xu <> 1 Stefan Hajnoczi <> 19