There are a bunch of tests in tools/tests/, let them run in CI.
For each subdirectory expect "make run" will run the test, and observe
its exit code. This way, adding new tests is easy, and they will be
automatically picked up.
For better visibility, log test output to junit xml format, and let
gitlab ingest it. Set SUT_ADDR variable with name/address of the system
under test, so a network can be used to extract the file. The actual
address is set using DHCP. And for the test internal network, still add
the 192.168.0.1 IP (but don't replace the DHCP-provided one).
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <stefano.stabellini@amd.com>
(cherry picked from commit fc10d6ecdd)
Instead use env to find the location of expect.
Additionally do not use the -f flag, as it's only meaningful when passing
arguments on the command line, which we never do for console.exp. From the
expect 5.45.4 man page:
> The -f flag prefaces a file from which to read commands from. The flag
> itself is optional as it is only useful when using the #! notation (see
> above), so that other arguments may be supplied on the command line.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Rework the container to be non-root, use heredocs for readability, and
use apt-get --no-install-recommends to keep the size down. Rename the
job to x86_32, to be consistent with XEN_TARGET_ARCH and the
naming scheme of all the other CI jobs:
${VERSION}-${ARCH}-${BUILD_NAME}
Remove build dependencies for building QEMU. The absence of ninja/meson means
that the container hasn't been able to build QEMU in years.
Remove build dependencies for the documentation as we don't have to
build it for every single arch.
This reduces the size of the container from 2.22GB to 1.32Gb.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
(cherry picked from commit 1ceabff115)
Rework the container to use heredocs for readability, and use
apt-get --no-install-recommends to keep the size down.
This reduces the size of the (uncompressed) container from 3.44GB to
1.97GB.
The container is left running the builds and tests as root. A
subsequent patch will make the necessary changes to the test scripts
to allow test execution as a non-root user.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
(cherry picked from commit 44b742de09)
Right now, various blocks rewrite domU_config= as a whole, even though it is
largely the same.
* dom0pvh-hvm does nothing but change the domain type to hvm
* *-pci sets the domain type, clears vif=[], appends earlyprintk=xen to the
cmdline, and adds some PCI config.
Refactor this to be domU_type (defaults to pvh), domU_vif (defaults to
xenbr0), and domU_extra_config (defaults to empty) and use these variables to
build domU_config= once.
Of note, the default domU_config= now sets cmdline=, and extra= is intended
for inclusion via domU_extra_config as necessary.
No practical change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit 3be3ae0770)
* List all the test_variants and summerise what's going on
* Use case rather than an if/else chain for $test_variant
* Adjust indentation inside the case block
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit 6685a129c7)
We spend an awful lot of CI time building QEMU, even though most changes don't
touch the subset of tools/libs/ used by QEMU. Some numbers taken at a time
when CI was otherwise quiet:
With Without
Alpine: 13m38s 6m04s
Debian 12: 10m05s 8m10s
OpenSUSE Tumbleweed: 11m40s 7m54s
Ubuntu 24.04: 14m56s 8m06s
which is a >50% improvement in wallclock time in some cases.
The only build we have that needs QEMU is alpine-3.18-gcc-debug. This is the
build deployed and used by the QubesOS ADL-* and Zen3p-* jobs.
Xilinx-x86_64 deploys it too, but is PVH-only and doesn't use QEMU.
QEMU is also built by CirrusCI for FreeBSD (fully Clang/LLVM toolchain).
This should help quite a lot with Gitlab CI capacity.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit e305256e69)
Check if xen.efi is bootable with an XTF dom0.
The multiboot2+EFI path is tested on hardware tests already.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
(cherry picked from commit 2d1c673bae)
TEST_TIMEOUT is set as a CI/CD project variable, as it should be, to
match the capability and speed of the testing infrastructure.
As it turns out, TEST_TIMEOUT defined in test.yaml cannot override
TEST_TIMEOUT defined as CI/CD project variable. As a consequence, today
the TEST_TIMEOUT setting in test.yaml for the Xilinx jobs is ignored.
Instead, rename TEST_TIMEOUT to TEST_TIMEOUT_OVERRIDE in test.yaml and
check for TEST_TIMEOUT_OVERRIDE first in console.exp.
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit d82e0e094e)
The test showed unreliable behavior due to unsupported console settings.
Update the baud rate used to connect to the UART.
Signed-off-by: Victor Lira <victorm.lira@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit c23571fe31)
Fixes: 95764a0817 (automation: update xilinx test scripts (tty))
This patch introduced a CI failure due to a timeout in xilinx-x86_64 test.
Change xilinx-x86_64 and xilinx-arm64 scripts to use "expect" utility
to determine test result and allow early exit from tests.
Add "expect" to xilinx container environment (dockerfile).
Rename references to "QEMU" in "qemu-key.exp" expect script to "TEST" to be
used by both QEMU and hardware tests.
Signed-off-by: Victor Lira <victorm.lira@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit 5e99a40ea5)
Fix flaw in qemu-*.sh tests that producess a false success. The following
lines produces success despite the "expect" script producing nonzero exit
status:
set +e
...
./automation/scripts/qemu-key.exp | sed 's/\r\+$//'
(end of file)
The default exit status for a pipeline using "|" operator is that of the
rightmost command. Fix this by setting the "pipefail" option in the shell,
and removing "set +e" allowing the expect script to determine the result.
Signed-off-by: Victor Lira <victorm.lira@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit 740c41ca05)
Update serial device names from ttyUSB* to test board specific names.
Update xilinx-smoke-dom0-x86_64 with new Xen command line console options,
which are now set as Gitlab CI/CD variables. Abstract the directory where
binaries are stored. Increase the timeout to match new setup.
Signed-off-by: Victor Lira <victorm.lira@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit 95764a0817)
After commit c36efb7fce ("automation: use expect to run QEMU") we lost
the \r filtering introduced by b576497e3b ("automation: remove CR
characters from serial output"). This patch reintroduced it.
Fixes: c36efb7fce ("automation: use expect to run QEMU")
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
(cherry picked from commit aa80a04df4)
The expectation is that QEMU_TIMEOUT should be set as a Gitlab CI/CD
variable but if not we should be able to run the pipeline anyway.
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
(cherry picked from commit 1e2a5f991f)
Add a test script and related job for running x86_64 dom0 tests.
Signed-off-by: Victor Lira <victorm.lira@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
(cherry picked from commit 6979e17b3f)
Use expect to invoke QEMU so that we can terminate the test as soon as
we get the right string in the output instead of waiting until the
final timeout.
For timeout, instead of an hardcoding the value, use a Gitlab CI
variable "QEMU_TIMEOUT" that can be changed depending on the latest
status of the Gitlab CI runners.
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
master commit: c36efb7fce
master date: 2024-08-09 23:59:20 -0700
The containers are exactly as per 20.04 (Focal). However, this now brings us
to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.
The oldest and newest toolchains are the most likely to find problems with new
code, so reduce the middle 3 releases (18/20/22) to just a single smoke test
each.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
As with 16.04 (Xenial), with python3-setuptools included. Having this package
only in some containers was intentional; see commit bbc72a7877 ("automation:
Add python3's setuptools to some containers") for the rational.
This saves ~500M:
registry.gitlab.com/xen-project/xen/ubuntu 20.04-x86_64 1.06GB
registry.gitlab.com/xen-project/xen/ubuntu focal 1.57GB
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
As with 16.04 (Xenial), except that ninja-build package is available so QEMU
can be built.
This halves the size of the container:
registry.gitlab.com/xen-project/xen/ubuntu 18.04-x86_64 857MB
registry.gitlab.com/xen-project/xen/ubuntu bionic 1.44GB
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Rework the container to be non-root, use heredocs for legibility, and use
apt-get --no-install-recommends to keep the size down.
Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced
by Xen 4.16. Therefore, drop the dependencies too.
This saves ~500M:
registry.gitlab.com/xen-project/xen/ubuntu 16.04-x86_64 698MB
registry.gitlab.com/xen-project/xen/ubuntu xenial 1.21GB
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
See prior patch for most discussion.
Despite appearing to be a fixed release (and therefore not marked as permitted
failure), the dockerfile references the `leap` tag which is rolling in
practice. Switch to 15.6 explicitly, for better test stability.
Vs tumbleweed, use `zypper update` rather than dist-upgrade, and retain the
RomBIOS dependencies; bin86 and dev86.
In terms of size, this saves ~700M:
registry.gitlab.com/xen-project/xen/opensuse leap-15.6-x86_64 1.33GB
registry.gitlab.com/xen-project/xen/suse opensuse-leap 2.05GB
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.
Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.
Trim the dependencies substantially. Testing docs isn't very interesting and
saves a lot of space. Other savings come from removing a huge pile of
optional QEMU dependencies (QEMU just needs to build the Xen parts to be
useful here, not have a full GUI environment).
Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.
This shaves 1G off the size of the container:
registry.gitlab.com/xen-project/xen/opensuse tumbleweed-x86_64 1.39GB
registry.gitlab.com/xen-project/xen/suse opensuse-tumbleweed 2.35GB
Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.
Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
The containers are both much smaller, with stable toolchains over time, and
this at least means we're not doing all testing with a single compiler.
Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and
reposition the jobs to optimise starting the smoke test.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Reviewed-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
... in the style of debian:12-ppc64le.
Rename the jobs and reposition them later as they're not a dependency for the
smoke testing any more.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's QEMU didn't
understand the powernv9 machine. However bookworm's QEMU does and this is
preferable to maintaining a random build of QEMU ourselves.
Use the debian:12-ppc64le container and test the output of that build too.
Remove qemu-system-ppc64-8.1.0-ppc64-export which is unused now.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
... conforming to the new naming scheme; $DISTRO-$VERSION-$ARCH-* so the jobs
sort more coherently.
Make it non-root by default, and set XEN_TARGET_ARCH=ppc64. Include QEMU too,
which will be used subsequently.
Add build jobs too, with debian-12-ppc64le-gcc-debug specifically early as it
will be used for smoke testing shortly.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Right now, most build containers use root. Archlinux, Fedora and Yocto set up
a regular user called `user`.
For those containers, trying to containerize as root fails, because
CONTAINER_UID0=1 does nothing, whereas CONTAINER_UID0=0 forces the user away
from root.
To make CONTAINER_UID0=1 work reliably, force to root if requested.
Fixes: 17fbe6504d ("automation: introduce a new variable to control container user")
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
This tests if QEMU works in PVH dom0. QEMU in dom0 requires enabling TUN
in the kernel, so do that too.
Add it to both x86 runners, similar to the PVH domU test.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Introduce accepted_guidelines.sh: a script to autogenerate the
configuration file accepted.ecl from docs/misra/rules.rst which enables
all accepted guidelines.
Introduce monitored.ecl: a manual selection of accepted guidelines
which are clean or almost clean, it is intended to be used for the
analyses triggered by commits.
Reorganize tagging.ecl:
-Remove "accepted" tags: keeping track of accepted guidelines tagging
them as "accepted" in the configuration file tagging.ecl is no
longer needed since docs/rules.rst is keeping track of them.
-Tag more guidelines as clean.
Reorganize eclair pipelines:
- Set1, Set2, Set3 are now obsolete: remove the corresponding
pipelines and ecl files.
- Amend scheduled eclair pipeline to use accepted.ecl.
- Amend triggered eclair pipeline to use monitored.ecl.
Rename and improve action_check_clean_regressions.sh to print a
diagnostic in case a commit introduces a violation of a clean guideline.
An example of diagnostic is the following:
Failure: 13 regressions found for clean guidelines
service MC3R1.R8.2: (required) Function types shall be in prototype form with named parameters:
violation: 13
Signed-off-by: Simone Ballarin <simone.ballarin@bugseng.com>
Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>
Signed-off-by: Alessandro Zucchelli <alessandro.zucchelli@bugseng.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
At the moment, all the Arm64 Qemu tests use GICv2 which is the default
GIC version used by Qemu. Improve the coverage by adding a new test in
which Qemu will be configured to have GICv3.
Rename host device tree name to "virt.dtb" to be GIC version agnostic.
Use "gic-version" Qemu option to select the version to use. Unless the
test variant is set to "gicv3", version 2 will be used.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Introduce qemu based Arm earlyprintk test and build jobs to cover this
feature in debug variant. The tests simply check for the presence of the
last message printed by the bootstrap code before entering the C world.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Current issues with this test are:
- when the job timeout, the log file is lost as there is no chance to
run the `mv` command.
- GitLab job log is limited in size, so one usually have to download
the artifacts, which may be missing.
Use $GITLAB_CI to detect when the script is run as part of a GitLab
pipeline. GitLab will add "GITLAB_CI=true" in the environment
variables.
When run as part of $GITLAB_CI, ignore "dirty" worktree to allow to
write "build-each-commit-gcc.log", which can then be grabbed as
artifacts, even when the job timeout. The `git clean` command is
changed to keep those build logs.
When run as part of $GITLAB_CI, we will also store the build output in
a log file instead of writing it to stdout, because GitLab's job log
is limited. But we will write the log to stdout in case of error, so
we can find out more quickly why there's been an error.
This patch also make use of a GitLab feature, "log sections", which we
will collapse by default. One section per commit been built.
There's a bit of colour added to the logs.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Thanks to recent changes added to ImageBuilder to support the bootz
command, which allows obtaining the effective image size (including NOLOAD
sections) from the zImage header, switch the BOOT_CMD for arm32 tests to
bootz. Among other scenarios, this change will enable us, in the future,
to add tests with UBSAN enabled Xen, which would otherwise fail due to
incorrect image placement resulting in overlapping.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
The serial is already setup by the firmware, be consistent with the rest of the
QEMU tests and don't specify a com1 setup.
Note it's also bogus, as the extra ',' will make the intended DPS argument to
be parsed as the io-base.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
The gitlab CI webpage seems to have issues displaying the \CR\CR\LF "\r\r\n"
sequence on the web interface used as line returns by the Linux kernel serial
output. This leads to the QEMU tests output looking like:
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 664kB init memory
mapping kernel into physical memory
about to get started...
qemu-system-x86_64: terminating on signal 15 from pid 52 (timeout)
This not helpful, so strip the CR characters from the output that goes to
stdout, leaving the output in the serial output file untouched.
All usages of `tee` are adjusted to strip CR from the output.
Fixes: 3030a73bf8 ('automation: add a QEMU based x86_64 Dom0/DomU test')
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
If system suspends too quickly, the message for the test controller to
wake up the system may be not sent to the console before suspending.
This will cause the test to timeout.
Fix this by calling sync on the console and waiting a bit after printing
the message. The test controller then resumes the system 30s after the
message, so as long as the delay + suspending takes less time it is
okay.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Checking /proc/interrupts is unreliable because different drivers set
different names there. Install pciutils and use lspci instead.
In fact, the /proc/interrupts content was confusing enough that
adl-pci-hvm had it wrong (MSI-X is in use there). Fix this too.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Marek Marczykowski-Górecki
Stefano Stabellini
Roger Pau Monne
Javi Merino
Andrew Cooper
Victor Lira
Stefano Stabellini
Michal Orzel
Simone Ballarin
Anthony PERARD