We spend an awful lot of CI time building QEMU, even though most changes don't
touch the subset of tools/libs/ used by QEMU. Some numbers taken at a time
when CI was otherwise quiet:
With Without
Alpine: 13m38s 6m04s
Debian 12: 10m05s 8m10s
OpenSUSE Tumbleweed: 11m40s 7m54s
Ubuntu 24.04: 14m56s 8m06s
which is a >50% improvement in wallclock time in some cases.
The only build we have that needs QEMU is alpine-3.18-gcc-debug. This is the
build deployed and used by the QubesOS ADL-* and Zen3p-* jobs.
Xilinx-x86_64 deploys it too, but is PVH-only and doesn't use QEMU.
QEMU is also built by CirrusCI for FreeBSD (fully Clang/LLVM toolchain).
This should help quite a lot with Gitlab CI capacity.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
A recent update to qemu-xen has bumped the build requirements, with Python 3.8
being the new baseline but also needing the 'ensurepip' and 'tomllib/tomli'
packages.
* Ubuntu/Debian package 'ensurepip' separately, but it can be obtained by
installing the python3-venv package.
* 'tomllib' was added to the python standard library in Python 3.11, but
previously it was a separate package named 'tomli'.
In terms of changes required to build QEMU:
* Ubuntu 24.04 (Noble) has Python 3.12 so only needs python3-venv
* Ubuntu 22.04 (Jammy) has Python 3.10 but does have a python3-tomli package
that QEMU is happy with.
* FreeBSD has Python 3.9, but Python 3.11 is available.
In terms of exclusions:
* Ubuntu 20.04 (Focal) has Python 3.8, but lacks any kind of tomli package.
* Fedora 29 (Python 3.7), OpenSUSE Leap 15.6 (Python 3.6), and Ubuntu
18.04/Bionic (Python 3.6) are now too old.
Detecting tomllib/tomli is more than can fit in build's one-liner, so break it
out into a proper script.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Change automation build script to call the make defconfig target before
setting CONFIG_DEBUG and extra options. This fixes issues on Power where
the build fails without using the ppc64_defconfig.
Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Shawn Anastasio <sanastasio@raptorengineering.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Since QEMU commit 74a1b256d775 ("configure: Bump minimum Clang version
to 10.0"), or QEMU v8.0, Clang 10.0 is now the minimum to build QEMU.
QEMU 8.0 fails to build on Ubuntu Bionic.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Calling build twice in the same environment will fail because the
directory 'binaries' was already created before. Use mkdir -p to ignore
an existing directory and move on to the actual build.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
As the recent work to get PCI Passthrough testing working shows, putting
`/bin/false` as a ROM into guest context doesn't work so well.
For all ROM paths where we're skipping the build, use a slightly-plausible but
likely non-existent path instead.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
It seems to build just fine with Alpine 3.12, and SeaBIOS is necessary
for a HVM test (that use the Alpine build).
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Add a debian container with cppcheck installation routine inside,
capable of performing cppcheck analysis on Xen-only build including
cross-builds for arm32 and x86_64.
Populate build jobs making use of that container to run cppcheck
analysis to produce a text report (xen-cppcheck.txt) containing the list
of all the findings.
This patch does not aim at performing any sort of bisection. Cppcheck is
imperfect and for now, our goal is to at least be aware of its reports,
so that we can compare them with the ones produced by better tools and
to be able to see how these reports change as a result of further
infrastructure improvements (e.g. exception list, rules exclusion).
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
There's no need to do ad-hoc string parsing. Use grep -q instead.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Some testcases use a cross compiler. Presently it's only arm32 and due to
previous cleanup the only thing which is now wrong is printing the compiler
version at the start of day.
Construct $cc to match what `make` will eventually choose given CROSS_COMPILE,
taking care not to modify $CC. Use $cc throughout the rest of the script.
Also correct the compiler detection logic. Plain "gcc" was wrong, and
"clang"* was a bodge highlighting the issue, but neither survive the
CROSS_COMPILE correction. Instead, construct cc_is_{gcc,clang} booleans like
we do elsewhere in the build system, by querying the --version text for gcc or
clang.
While making this change, adjust cc_ver to be calculated once at the same time
as cc_is_* are calculated.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Whether to build only Xen, or everything, is a property of container,
toolchain and/or testcase. It is not a property of XEN_TARGET_ARCH.
Capitalise HYPERVISOR_ONLY and have it set by all the
debian-unstable-gcc-arm32-* testcases at the point that arm32 get matched with
a container that can only build Xen.
To reduce the churn elsewhere, retain the RANDCONFIG implies HYPERVISOR_ONLY
property.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
This is purely code motion of the cfgargs construction, into the case where it
is used.
No practical change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Preserve the artefacts based on the `make` rune we actually ran, rather than
guesswork about which rune we would have run based on other settings.
Note that the ARM qemu smoke tests depend on finding binaries/xen even from
full builds. Also the Jessie-32 containers build tools but not Xen.
This means the x86_32 builds now store relevant artefacts. No change in other
configurations.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Having 3 extra hypervisor builds on the end of a full build is deeply
confusing to debug if one of them fails, because the .config file presented in
the artefacts is not the one which caused a build failure. Also, the log
tends to be truncated in the UI.
PV-only is tested as part of PV-Shim in a full build anyway, so doesn't need
repeating. HVM-only and neither appear frequently in randconfig, so drop all
the logic here to simplify things.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
All the build jobs exist in two flavors: debug and non-debug, where the
former sets 'debug' variable to 'y' and the latter to 'n'. This variable
is only being recognized by the toolstack, because Xen requires
enabling/disabling debug build via e.g. menuconfig/config file.
As a corollary, we end up building/testing Xen with CONFIG_DEBUG always
set to a default value ('y' for unstable and 'n' for stable branches),
regardless of the type of the build job.
Fix this behavior by setting CONFIG_DEBUG according to the 'debug' value.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
buster-gcc-ibt is a dedicated test to run a not-yet-upstreamed compiler patch
which is relevant to CONFIG_XEN_IBT in 4.17 and later.
Force it on, rather than having 50% of the jobs not testing what they're
supposed to be testing.
Fixes: 5d59421815 ("x86: Use control flow typechecking where possible")
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
[stefano: minor code style improvement]
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Currently, all the arm64 defconfig build jobs, regardless of the
container used, end up building Xen with the extra config options
specified in the main build script (e.g. CONFIG_EXPERT,
CONFIG_STATIC_MEMORY). Because these options are only needed for
specific test jobs, the current behavior of the CI is incorrect
as we add the extra options to all the defconfig builds. This means
that on arm64 there is not a single job performing proper defconfig build.
To fix this issue, add custom build jobs each time there is a need for
building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
variable to be used by these jobs to store the required options. This
variable will be then read by the main build script to modify the .config
file. This will also help users to understand what is needed to run specific
test.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Add a new test job qemu-smoke-arm64-gcc-boot-cpupools that will execute
script qemu-smoke-arm64.sh to test boot time cpupools feature.
Enable CONFIG_BOOT_TIME_CPUPOOLS for the arm64 build and add a new test
case in qemu-smoke-arm64.sh that if selected will make use of the
ImageBuilder feature to create cpupool with cpu@1, null scheduler and
assign it to domU. Add a check in dom0 xen.start to see if domU is
assigned a Pool-1 with null scheduler.
Take the opportunity to refactor the qemu-smoke-arm64.sh script as
follows:
- use domU_check to store the test's commands to be run from domU
- use dom0_check to store the test's commands to be run from dom0
- use fdtput instead of sed to perform dtb modifications
- use more meaningful messages for "passed" variable. This way we can
grep for messages reported either by domU or dom0 and get rid of
assumption that tests can only be run from domU
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Enable CONFIG_STATIC_MEMORY in the existing arm64 build.
Create a new test job, called qemu-smoke-arm64-gcc-staticmem.
Adjust qemu-smoke-arm64.sh script to accomodate the static memory test as a
new test variant. The test variant is determined based on the first argument
passed to the script. For testing static memory, the argument is 'static-mem'.
The test configures DOM1 with a static memory region and adds a check in the
init script.
The check consists in comparing the contents of the /proc/device-tree
memory entry with the static memory range with which DOM1 was configured.
If the memory layout is correct, a message gets printed by DOM1.
At the end of the qemu run, the script searches for the specific message
in the logs and fails if not found.
Signed-off-by: Xenia Ragiadakou <burzalodowa@gmail.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Penny Zheng <penny.zheng@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
The EXPERT config option cannot anymore be selected via the environmental
variable XEN_CONFIG_EXPERT. Remove stale references to XEN_CONFIG_EXPERT
from the automation code.
Signed-off-by: Xenia Ragiadakou <burzalodowa@gmail.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
To aid debugging build failures, dump /etc/os-release during build.
This helps with rolling releases such as Tumbleweed to understand the
state of the build container.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
ninja is now required to build the latest version of QEMU, and not all
distros have a suitable version. Skip the QEMU build when ninja is not
available.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
It is reported that failed randconfig runs are missing the config file
which makes debugging impossible. Fix this by moving the line that
copies the config file before the build is executed.
Signed-off-by: Wei Liu <wl@xen.org>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Add a debian build container with cross-gcc for arm32 installed.
Add build jobs to cross-compile Xen-only for arm32.
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Acked-by: Wei Liu <wl@xen.org>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
For things like randconfig there's no need to do a full Xen build, a
hypervisor build only will be much faster and will achieve the same
level of testing, as randconfig only changes the hypervisor build
options.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
It's now safe to enable the build of rombios on Alpine systems, as
hvmloader already builds fine there.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Wei Liu <wl@xen.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
Create a test job that starts Xen and Dom0 on QEMU based on the alpine
linux rootfs. Use the Linux kernel and rootfs from the tests-artifacts
containers. Add the Xen tools binaries from the Alpine Linux build job.
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Acked-by: Wei Liu <wl@xen.org>
QEMU upstream builds with warnings when libc is musl:
#warning redirecting incorrect #include <sys/signal.h> to <signal.h>
Disable -Werror by passing --disable-werror to the QEMUU config script
if libc is musl.
hvmloader doesn't build on musl systems today. Disable any guest
firmware build.
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Acked-by: Wei Liu <wl@xen.org>
Use QEMU to start Xen (just the hypervisor) up until it stops because
there is no dom0 kernel to boot.
It is based on the existing build job unstable-arm64v8.
Also use make -j$(nproc) to build Xen.
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Acked-by: Wei Liu <wl@xen.org>
Recent version of QEMU will not build anymore if Python < 3.5.
That is, QEMU 4.3 not released yet.
That check would also prevent the GitLab CI from building QEMU if
python3 binary isn't present.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
Using clang and lld 8 requires installing the packages from the
official llvm apt repositories, so modify the Debian Docker files for
stretch and unstable to add the llvm repo and install clang and lld
from it.
Also add some jobs to test building Xen with clang 8 and lld.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
CentOS 6 is probably the most frequently broken build, so adding it to CI
would be a very good move.
One problem is that CentOS 6 comes with Python 2.6, and Qemu requires 2.7.
There appear to be no sensible ways to get Python 2.7 into a CentOS 6
environments, so modify the build script to skip the Qemu upstream build
instead. Additionally, SeaBIOS requires GCC 4.6 or later, so skip it as well.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Previously it is disabled because the embedded ipxe can't be built
with clang. Now that ipxe is split out we can use --with-system-ipxe
to work around the issue.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Currently randconfig tests are more likely to fail than to succeed
because of a bug in xen's build system: symbols-dummy.o's dependency
is wrong, which causes it to not get rebuild between runs, which
eventually causes linking to fail. There may also be other corner
cases we haven't discovered.
The fix is not straightforward. For now, make sure the tree is cleaned
properly between builds so we don't see random failures in Gitlab CI.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
... so that it can be passed on to test stage.
Note that xen is only extracted for x86_64 build since others may not
have that. Use a directory to account for possibly different file
names on Arm.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
This aids troubleshooting when we notice a failure in the default
configuration.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
The restrictions are mostly from Travis CI days when cross-compilation
was not feasible. Now we will be building natively on Arm, remove
those lines.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
Introduce a new directory to put in configs we care about. Modify
build script to build with those configs.
While we only introduce x86 configs initially, provision for non-x86
configs.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Doug Goldstein <cardoe@cardoe.com>
a75703b2f0 unconditionally enabled
building of rombios and stubdoms but unfortunately these two pieces do
not build successfully with clang. rombios unconditionally depends on
the build of iPXE and upstream iPXE does not support building with
clang. Similiarly the build of the stubdoms depends on the build of
newlib which does not support being built with clang.
Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Added a GitLab CI config which has a lot more flexibility to allow us to
test a lot more distro configurations than Travis can and even build
test on FreeBSD. This includes a modified copy of scripts/travis-build
that is expected to diverge future over time as we build more than what
Travis is currently building.
Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Andrew Cooper
Marek Marczykowski-Górecki
Shawn Anastasio
Anthony PERARD
Michal Orzel
Olaf Hering
Xenia Ragiadakou
Wei Liu
Stefano Stabellini
Roger Pau Monne
Wei Liu
Doug Goldstein