mirror of
https://github.com/xen-project/xen.git
synced 2026-10-28 02:01:24 +01:00
Dan Magenheimer
65c9792df6
When guests memory consumption is volatile (multiple guests ballooning up/down) we are presented with the problem of being able to determine exactly how much memory there is for allocation of new guests without negatively impacting existing guests. Note that the existing models (xapi, xend) drive the memory consumption from the tool-stack and assume that the guest will eventually hit the memory target. Other models, such as the dynamic memory utilized by tmem, do this differently - the guest drivers the memory consumption (up to the d->max_pages ceiling). With dynamic memory model, the guest frequently can balloon up and down as it sees fit. This presents the problem to the toolstack that it does not know atomically how much free memory there is (as the information gets stale the moment the d->tot_pages information is provided to the tool-stack), and hence when starting a guest can fail during the memory creation process. Especially if the process is done in parallel. In a nutshell what we need is a atomic value of all domains tot_pages during the allocation of guests. Naturally holding a lock for such a long time is unacceptable. Hence the goal of this hypercall is to attempt to atomically and very quickly determine if there are sufficient pages available in the system and, if so, "set aside" that quantity of pages for future allocations by that domain. Unlike an existing hypercall such as increase_reservation or populate_physmap, specific physical pageframes are not assigned to the domain because this cannot be done sufficiently quickly (especially for very large allocations in an arbitrarily fragmented system) and so the existing mechanisms result in classic time-of-check-time-of-use (TOCTOU) races. One can think of claiming as similar to a "lazy" allocation, but subsequent hypercalls are required to do the actual physical pageframe allocation. Note that one of effects of this hypercall is that from the perspective of other running guests - suddenly there is a new guest occupying X amount of pages. This means that when we try to balloon up they will hit the system-wide ceiling of available free memory (if the total sum of the existing d->max_pages >= host memory). This is OK - as that is part of the overcommit. What we DO NOT want to do is dictate their ceiling should be (d->max_pages) as that is risky and can lead to guests OOM-ing. It is something the guest needs to figure out. In order for a toolstack to "get" information about whether a domain has a claim and, if so, how large, and also for the toolstack to measure the total system-wide claim, a second subop has been added and exposed through domctl and libxl (see "xen: XENMEM_claim_pages: xc"). == Alternative solutions == There has been a variety of discussion whether the problem hypercall is solving can be done in user-space, such as: - For all the existing guest, set their d->max_pages temporarily to d->tot_pages and create the domain. This forces those domains to stay at their current consumption level (fyi, this is what the tmem freeze call is doing). The disadvantage of this is that needlessly forces the guests to stay at the memory usage instead of allowing it to decide the optimal target. - Account only using d->max_pages of how much free memory there is. This ignores ballooning changes and any over-commit scenario. This is similar to the scenario where the sum of all d->max_pages (and the one to be allocated now) on the host is smaller than the available free memory. As such it ignores the over-commit problem. - Provide a ring/FIFO along with event channel to notify an userspace daemon of guests memory consumption. This daemon can then provide up-to-date information to the toolstack of how much free memory there is. This duplicates what the hypervisor is already doing and introduced latency issues and catching breath for the toolstack as there might be millions of these updates on heavily used machine. There might not be any quiescent state ever and the toolstack will heavily consume CPU cycles and not ever provide up-to-date information. It has been noted that this claim mechanism solves the underlying problem (slow failure of domain creation) for a large class of domains but not all, specifically not handling (but also not making the problem worse for) PV domains that specify the "superpages" flag, and 32-bit PV domains on large RAM systems. These will be addressed at a later time. Code overview: Though the hypercall simply does arithmetic within locks, some of the semantics in the code may be a bit subtle. The key variables (d->unclaimed_pages and total_unclaimed_pages) starts at zero if no claim has yet been staked for any domain. (Perhaps a better name is "claimed_but_not_yet_possessed" but that's a bit unwieldy.) If no claim hypercalls are executed, there should be no impact on existing usage. When a claim is successfully staked by a domain, it is like a watermark but there is no record kept of the size of the claim. Instead, d->unclaimed_pages is set to the difference between d->tot_pages and the claim. When d->tot_pages increases or decreases, d->unclaimed_pages atomically decreases or increases. Once d->unclaimed_pages reaches zero, the claim is satisfied and d->unclaimed pages stays at zero -- unless a new claim is subsequently staked. The systemwide variable total_unclaimed_pages is always the sum of d->unclaimed_pages, across all domains. A non-domain- specific heap allocation will fail if total_unclaimed_pages exceeds free (plus, on tmem enabled systems, freeable) pages. Claim semantics could be modified by flags. The initial implementation had three flag, which discerns whether the caller would like tmem freeable pages to be considered in determining whether or not the claim can be successfully staked. This in later patches was removed and there are no flags. A claim can be cancelled by requesting a claim with the number of pages being zero. A second subop returns the total outstanding claimed pages systemwide. Note: Save/restore/migrate may need to be modified, else it can be documented that all claims are cancelled. This patch of the proposed XENMEM_claim_pages hypercall/subop, takes into account review feedback from Jan and Keir and IanC and Matthew Daley, plus some fixes found via runtime debugging. Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Acked-by: Tim Deegan <tim@xen.org> Acked-by: Keir Fraser <keir@xen.org>
#################################
__ __ _ _ ____
\ \/ /___ _ __ | || | |___ \
\ // _ \ '_ \ | || |_ __) |
/ \ __/ | | | |__ _| / __/
/_/\_\___|_| |_| |_|(_)_____|
#################################
http://www.xen.org/
What is Xen?
============
Xen is a Virtual Machine Monitor (VMM) originally developed by the
Systems Research Group of the University of Cambridge Computer
Laboratory, as part of the UK-EPSRC funded XenoServers project. Xen
is freely-distributable Open Source software, released under the GNU
GPL. Since its initial public release, Xen has grown a large
development community, spearheaded by xen.org (http://www.xen.org).
The 4.2 release offers excellent performance, hardware support and
enterprise-grade features such as x86_32-PAE, x86_64, SMP guests and
live relocation of VMs. Ports to Linux, NetBSD, FreeBSD and Solaris
are available from the community.
This file contains some quick-start instructions to install Xen on
your system. For more information see http:/www.xen.org/ and
http://wiki.xen.org/
Quick-Start Guide
=================
First, there are a number of prerequisites for building a Xen source
release. Make sure you have all the following installed, either by
visiting the project webpage or installing a pre-built package
provided by your OS distributor:
* GCC v4.1 or later
* GNU Make
* GNU Binutils
* Development install of zlib (e.g., zlib-dev)
* Development install of Python v2.3 or later (e.g., python-dev)
* Development install of curses (e.g., libncurses-dev)
* Development install of openssl (e.g., openssl-dev)
* Development install of x11 (e.g. xorg-x11-dev)
* Development install of uuid (e.g. uuid-dev)
* Development install of yajl (e.g. libyajl-dev)
* Development install of libaio (e.g. libaio-dev) version 0.3.107 or
greater. Set CONFIG_SYSTEM_LIBAIO in .config if this is not available.
* Development install of GLib v2.0 (e.g. libglib2.0-dev)
* Development install of Pixman (e.g. libpixman-1-dev)
* pkg-config
* bridge-utils package (/sbin/brctl)
* iproute package (/sbin/ip)
* udev
* GNU bison and GNU flex
* GNU gettext
* 16-bit x86 assembler, loader and compiler (dev86 rpm or bin86 & bcc debs)
* ACPI ASL compiler (iasl)
In addition to the above there are a number of optional build
prerequisites. Omitting these will cause the related features to be
disabled at compile time:
* Development install of Ocaml (e.g. ocaml-nox and
ocaml-findlib). Required to build ocaml components which
includes the alternative ocaml xenstored.
* cmake (if building vtpm stub domains)
* markdown
Second, you need to acquire a suitable kernel for use in domain 0. If
possible you should use a kernel provided by your OS distributor. If
no suitable kernel is available from your OS distributor then refer to
http://wiki.xen.org/wiki/XenDom0Kernels for suggestions for
suitable kernels to use.
If you are looking to compile a Dom0 kernel from source, please refer to
http://wiki.xen.org/wiki/XenParavirtOps.
[NB. Unless noted otherwise, all the following steps should be
performed with root privileges.]
1. Download and untar the source tarball file. This will be a
file named xen-unstable-src.tgz, or xen-$version-src.tgz.
You can also pull the current version from the mercurial
repository at http://xenbits.xen.org/
# tar xzf xen-unstable-src.tgz
Assuming you are using the unstable tree, this will
untar into xen-unstable. The rest of the instructions
use the unstable tree as an example, substitute the
version for unstable.
2. cd to xen-unstable (or whatever you sensibly rename it to).
3. For the very first build, or if you want to destroy build trees,
perform the following steps:
# ./configure
# make world
# make install
If you want, you can run ./configure --help to see the list of
options available options when building and installing Xen.
This will create and install onto the local machine. It will build
the xen binary (xen.gz), the tools and the documentation.
You can override the destination for make install by setting DESTDIR
to some value.
4. To rebuild an existing tree without modifying the config:
# make dist
This will build and install xen, tools, and docs into the local dist/
directory.
You can override the destination for make install by setting DISTDIR
to some value.
make install and make dist differ in that make install does the
right things for your local machine (installing the appropriate
version of udev scripts, for example), but make dist includes all
versions of those scripts, so that you can copy the dist directory
to another machine and install from that distribution.
Python Runtime Libraries
========================
Xend (the Xen daemon) has the following runtime dependencies:
* Python 2.3 or later.
In some distros, the XML-aspects to the standard library
(xml.dom.minidom etc) are broken out into a separate python-xml package.
This is also required.
In more recent versions of Debian and Ubuntu the XML-aspects are included
in the base python package however (python-xml has been removed
from Debian in squeeze and from Ubuntu in intrepid).
URL: http://www.python.org/
Debian: python
* For optional SSL support, pyOpenSSL:
URL: http://pyopenssl.sourceforge.net/
Debian: python-pyopenssl
* For optional PAM support, PyPAM:
URL: http://www.pangalactic.org/PyPAM/
Debian: python-pam
* For optional XenAPI support in XM, lxml:
URL: http://lxml.de/
Debian: python-lxml
YUM: python-lxml
Intel(R) Trusted Execution Technology Support
=============================================
Intel's technology for safer computing, Intel(R) Trusted Execution Technology
(Intel(R) TXT), defines platform-level enhancements that provide the building
blocks for creating trusted platforms. For more information, see
http://www.intel.com/technology/security/.
Intel(R) TXT support is provided by the Trusted Boot (tboot) module in
conjunction with minimal logic in the Xen hypervisor.
Tboot is an open source, pre- kernel/VMM module that uses Intel(R) TXT to
perform a measured and verified launch of an OS kernel/VMM.
The Trusted Boot module is available from
http://sourceforge.net/projects/tboot. This project hosts the code in a
mercurial repo at http://tboot.sourceforge.net/hg/tboot.hg and contains
tarballs of the source. Instructions in the tboot README describe how
to modify grub.conf to use tboot to launch Xen.
There are optional targets as part of Xen's top-level makefile that will
download and build tboot: install-tboot, build-tboot, dist-tboot, clean-tboot.
These will download the latest tar file from the SourceForge site using wget,
then build/install/dist according to Xen's settings.