mirror of
https://github.com/xen-project/xen.git
synced 2026-10-29 02:30:03 +01:00
Konrad Rzeszutek Wilk
f6dd295381
The existing tasklet mechanism has a single global spinlock that is
taken every-time the global list is touched. And we use this lock quite
a lot - when we call do_tasklet_work which is called via an softirq and
from the idle loop. We take the lock on any operation on the
tasklet_list.
The problem we are facing is that there are quite a lot of tasklets
scheduled. The most common one that is invoked is the one injecting the
VIRQ_TIMER in the guest. Guests are not insane and don't set the
one-shot or periodic clocks to be in sub 1ms intervals (causing said
tasklet to be scheduled for such small intervalls).
The problem appears when PCI passthrough devices are used over many
sockets and we have an mix of heavy-interrupt guests and idle guests.
The idle guests end up seeing 1/10 of its RUNNING timeslice eaten by
the hypervisor (and 40% steal time).
The mechanism by which we inject PCI interrupts is by hvm_do_IRQ_dpci
which schedules the hvm_dirq_assist tasklet every time an interrupt is
received. The callchain is:
_asm_vmexit_handler
-> vmx_vmexit_handler
->vmx_do_extint
-> do_IRQ
-> __do_IRQ_guest
-> hvm_do_IRQ_dpci
tasklet_schedule(&dpci->dirq_tasklet);
[takes lock to put the tasklet on]
[later on the schedule_tail is invoked which is 'vmx_do_resume']
vmx_do_resume
-> vmx_asm_do_vmentry
-> call vmx_intr_assist
-> vmx_process_softirqs
-> do_softirq
[executes the tasklet function, takes the
lock again]
While on other CPUs they might be sitting in a idle loop and invoked to
deliver an VIRQ_TIMER, which also ends up taking the lock twice: first
to schedule the v->arch.hvm_vcpu.assert_evtchn_irq_tasklet (accounted
to the guests' BLOCKED_state); then to execute it - which is accounted
for in the guest's RUNTIME_state.
The end result is that on a 8 socket machine with PCI passthrough,
where four sockets are busy with interrupts, and the other sockets have
idle guests - we end up with the idle guests having around 40% steal
time and 1/10 of its timeslice (3ms out of 30 ms) being tied up taking
the lock. The latency of the PCI interrupts delieved to guest is also
hindered.
With this patch the problem disappears completly. That is removing the
lock for the PCI passthrough use-case (the 'hvm_dirq_assist' case) by
not using tasklets at all.
The patch is simple - instead of scheduling an tasklet we schedule our
own softirq - HVM_DPCI_SOFTIRQ, which will take care of running
'hvm_dirq_assist'. The information we need on each CPU is which
'struct hvm_pirq_dpci' structure the 'hvm_dirq_assist' needs to run on.
That is simple solved by threading the 'struct hvm_pirq_dpci' through a
linked list. The rule of only running one 'hvm_dirq_assist' for only
one 'hvm_pirq_dpci' is also preserved by having 'schedule_dpci_for'
ignore any subsequent calls for an domain which has already been
scheduled.
== Code details ==
Most of the code complexity comes from the '->dom' field in the
'hvm_pirq_dpci' structure. We use it for ref-counting and as such it
MUST be valid as long as STATE_SCHED bit is set. Whoever clears the
STATE_SCHED bit does the ref-counting and can also reset the '->dom'
field.
To compound the complexity, there are multiple points where the
'hvm_pirq_dpci' structure is reset or re-used. Initially (first time
the domain uses the pirq), the 'hvm_pirq_dpci->dom' field is set to
NULL as it is allocated. On subsequent calls in to 'pt_irq_create_bind'
the ->dom is whatever it had last time.
As this is the initial call (which QEMU ends up calling when the guest
writes an vector value in the MSI field) we MUST set the '->dom' to a
the proper structure (otherwise we cannot do proper ref-counting).
The mechanism to tear it down is more complex as there are three ways
it can be executed. To make it simpler everything revolves around
'pt_pirq_softirq_active'. If it returns -EAGAIN that means there is an
outstanding softirq that needs to finish running before we can continue
tearing down. With that in mind:
a) pci_clean_dpci_irq. This gets called when the guest is being
destroyed. We end up calling 'pt_pirq_softirq_active' to see if it
is OK to continue the destruction.
The scenarios in which the 'struct pirq' (and subsequently the
'hvm_pirq_dpci') gets destroyed is when:
- guest did not use the pirq at all after setup.
- guest did use pirq, but decided to mask and left it in that state.
- guest did use pirq, but crashed.
In all of those scenarios we end up calling 'pt_pirq_softirq_active'
to check if the softirq is still active. Read below on the
'pt_pirq_softirq_active' loop.
b) pt_irq_destroy_bind (guest disables the MSI). We double-check that
the softirq has run by piggy-backing on the existing
'pirq_cleanup_check' mechanism which calls 'pt_pirq_cleanup_check'.
We add the extra call to 'pt_pirq_softirq_active' in
'pt_pirq_cleanup_check'.
NOTE: Guests that use event channels unbind first the event channel
from PIRQs, so the 'pt_pirq_cleanup_check' won't be called as 'event'
is set to zero. In that case we either clean it up via the a) or c)
mechanism.
There is an extra scenario regardless of 'event' being set or not:
the guest did 'pt_irq_destroy_bind' while an interrupt was triggered
and softirq was scheduled (but had not been run). It is OK to still
run the softirq as hvm_dirq_assist won't do anything (as the flags
are set to zero). However we will try to deschedule the softirq if
we can (by clearing the STATE_SCHED bit and us doing the
ref-counting).
c) pt_irq_create_bind (not a typo). The scenarios are:
- guest disables the MSI and then enables it (rmmod and modprobe in
a loop). We call 'pt_pirq_reset' which checks to see if the
softirq has been scheduled. Imagine the 'b)' with interrupts in
flight and c) getting called in a loop.
We will spin up on 'pt_pirq_is_active' (at the start of the
'pt_irq_create_bind') with the event_lock spinlock dropped and waiting
(cpu_relax). We cannot call 'process_pending_softirqs' as it might
result in a dead-lock. hvm_dirq_assist will be executed and then the
softirq will clear 'state' which signals that that we can re-use the
'hvm_pirq_dpci' structure. In case this softirq is scheduled on a
remote CPU the softirq will run on it as the semantics behind an
softirq is that it will execute within the guest interruption.
- we hit once the error paths in 'pt_irq_create_bind' while an
interrupt was triggered and softirq was scheduled.
If the softirq is in STATE_RUN that means it is executing and we should
let it continue on. We can clear the '->dom' field as the softirq has
stashed it beforehand. If the softirq is STATE_SCHED and we are
successful in clearing it, we do the ref-counting and clear the '->dom'
field. Otherwise we let the softirq continue on and the '->dom' field
is left intact. The clearing of the '->dom' is left to a), b) or again
c) case.
Note that in both cases the 'flags' variable is cleared so
hvm_dirq_assist won't actually do anything.
Suggested-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
#################################
__ __ _ _ ____ _ _ _
\ \/ /___ _ __ | || | | ___| _ _ _ __ ___| |_ __ _| |__ | | ___
\ // _ \ '_ \ | || |_ |___ \ _____| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
/ \ __/ | | | |__ _| ___) |_____| |_| | | | \__ \ || (_| | |_) | | __/
/_/\_\___|_| |_| |_|(_)____/ \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
#################################
http://www.xen.org/
What is Xen?
============
Xen is a Virtual Machine Monitor (VMM) originally developed by the
Systems Research Group of the University of Cambridge Computer
Laboratory, as part of the UK-EPSRC funded XenoServers project. Xen
is freely-distributable Open Source software, released under the GNU
GPL. Since its initial public release, Xen has grown a large
development community, spearheaded by xen.org (http://www.xen.org).
The 4.3 release offers a number of improvements, including NUMA
scheduling affinity, openvswitch integration, and defaulting to
qemu-xen rather than qemu-traditional for non-stubdom guests.
(qemu-xen is kept very close to the upstream project.) We also have a
number of updates to vTPM, and improvements to XSM and Flask to allow
greater disaggregation. Additionally, 4.3 contains a basic version of
Xen for the new ARM server architecture, both 32- and 64-bit. And as
always, there are a number of performance, stability, and security
improvements under-the hood.
This file contains some quick-start instructions to install Xen on
your system. For more information see http:/www.xen.org/ and
http://wiki.xen.org/
Quick-Start Guide
=================
First, there are a number of prerequisites for building a Xen source
release. Make sure you have all the following installed, either by
visiting the project webpage or installing a pre-built package
provided by your OS distributor:
* GCC v4.1 or later
* GNU Make
* GNU Binutils
* Development install of zlib (e.g., zlib-dev)
* Development install of Python v2.3 or later (e.g., python-dev)
* Development install of curses (e.g., libncurses-dev)
* Development install of openssl (e.g., openssl-dev)
* Development install of x11 (e.g. xorg-x11-dev)
* Development install of uuid (e.g. uuid-dev)
* Development install of yajl (e.g. libyajl-dev)
* Development install of libaio (e.g. libaio-dev) version 0.3.107 or
greater.
* Development install of GLib v2.0 (e.g. libglib2.0-dev)
* Development install of Pixman (e.g. libpixman-1-dev)
* pkg-config
* bridge-utils package (/sbin/brctl)
* iproute package (/sbin/ip)
* GNU bison and GNU flex
* GNU gettext
* 16-bit x86 assembler, loader and compiler (dev86 rpm or bin86 & bcc debs)
* ACPI ASL compiler (iasl)
In addition to the above there are a number of optional build
prerequisites. Omitting these will cause the related features to be
disabled at compile time:
* Development install of Ocaml (e.g. ocaml-nox and
ocaml-findlib). Required to build ocaml components which
includes the alternative ocaml xenstored.
* cmake (if building vtpm stub domains)
* markdown
* figlet (for generating the traditional Xen start of day banner)
* systemd daemon development files
* Development install of libnl3 (e.g., libnl-3-200,
libnl-3-dev, etc). Required if network buffering is desired
when using Remus with libxl. See docs/README.remus for detailed
information.
Second, you need to acquire a suitable kernel for use in domain 0. If
possible you should use a kernel provided by your OS distributor. If
no suitable kernel is available from your OS distributor then refer to
http://wiki.xen.org/wiki/XenDom0Kernels for suggestions for
suitable kernels to use.
If you are looking to compile a Dom0 kernel from source, please refer to
http://wiki.xen.org/wiki/XenParavirtOps.
[NB. Unless noted otherwise, all the following steps should be
performed with root privileges.]
1. Download and untar the source tarball file. This will be a
file named xen-unstable-src.tgz, or xen-$version-src.tgz.
You can also pull the current version from the git or mercurial
repositories at http://xenbits.xen.org/
# tar xzf xen-unstable-src.tgz
Assuming you are using the unstable tree, this will
untar into xen-unstable. The rest of the instructions
use the unstable tree as an example, substitute the
version for unstable.
2. cd to xen-unstable (or whatever you sensibly rename it to).
3. For the very first build, or if you want to destroy build trees,
perform the following steps:
# ./configure
# make world
# make install
See the documentation in the INSTALL file for more info.
This will create and install onto the local machine. It will build
the xen binary (xen.gz), the tools and the documentation.
You can override the destination for make install by setting DESTDIR
to some value.
4. To rebuild an existing tree without modifying the config:
# make dist
This will build and install xen, tools, and docs into the local dist/
directory.
You can override the destination for make install by setting DISTDIR
to some value.
make install and make dist differ in that make install does the
right things for your local machine (installing the appropriate
version of udev scripts, for example), but make dist includes all
versions of those scripts, so that you can copy the dist directory
to another machine and install from that distribution.
xenstore: xenstored and oxenstored
====================================
Xen uses a configuration database called xenstore [0] to maintain configuration
and status information shared between domains. A daemon is implemented as part
of xenstore to act as an interface for access to the database for dom0 and
guests. Two xenstored daemons are supported, one written in C which we refer
to as the xenstored (sometimes referred to as cxenstored), and another written
in Ocaml called oxenstored. Details for xenstore and the different
implementations can be found on the wiki's xenstore reference guide [1] and
the xenstored [2] page. You can choose which xenstore you want to enable as
default on a system through configure:
./configure --with-xenstored=xenstored
./configure --with-xenstored=oxenstored
By default oxenstored will be used if the ocaml development tools are found.
If you enable oxenstored the xenstored will still be built and installed,
the xenstored used can be changed through the configuration file:
/etc/sysconfig/xencommons
or
/etc/default/xencommons
You can change the preferred xenstored you want to use in the configuration
but since we cannot stop the daemon a reboot will be required to make the
change take effect.
[0] http://wiki.xen.org/wiki/XenStore
[1] http://wiki.xen.org/wiki/XenStoreReference
[2] http://wiki.xen.org/wiki/Xenstored
Python Runtime Libraries
========================
Various tools, such as pygrub, have the following runtime dependencies:
* Python 2.3 or later.
URL: http://www.python.org/
Debian: python
Intel(R) Trusted Execution Technology Support
=============================================
Intel's technology for safer computing, Intel(R) Trusted Execution Technology
(Intel(R) TXT), defines platform-level enhancements that provide the building
blocks for creating trusted platforms. For more information, see
http://www.intel.com/technology/security/.
Intel(R) TXT support is provided by the Trusted Boot (tboot) module in
conjunction with minimal logic in the Xen hypervisor.
Tboot is an open source, pre- kernel/VMM module that uses Intel(R) TXT to
perform a measured and verified launch of an OS kernel/VMM.
The Trusted Boot module is available from
http://sourceforge.net/projects/tboot. This project hosts the code in a
mercurial repo at http://tboot.sourceforge.net/hg/tboot.hg and contains
tarballs of the source. Instructions in the tboot README describe how
to modify grub.conf to use tboot to launch Xen.
There are optional targets as part of Xen's top-level makefile that will
download and build tboot: install-tboot, build-tboot, dist-tboot, clean-tboot.
These will download the latest tar file from the SourceForge site using wget,
then build/install/dist according to Xen's settings.