Files
xen/docs/user.tex
T
2004-08-24 22:32:56 +00:00

1262 lines
49 KiB
TeX

\documentclass[11pt,twoside,final,openright]{xenstyle}
\usepackage{a4,graphicx,setspace}
\setstretch{1.15}
\input{style.tex}
\begin{document}
% TITLE PAGE
\pagestyle{empty}
\begin{center}
\vspace*{\fill}
\includegraphics{eps/xenlogo.eps}
\vfill
\vfill
\vfill
\begin{tabular}{l}
{\Huge \bf Users' manual} \\[4mm]
{\huge Xen v2.0 for x86} \\[80mm]
{\Large Xen is Copyright (c) 2004, The Xen Team} \\[3mm]
{\Large University of Cambridge, UK} \\[20mm]
{\large Last updated on 30th July, 2004}
\end{tabular}
\vfill
\end{center}
\cleardoublepage
% TABLE OF CONTENTS
\pagestyle{plain}
\pagenumbering{roman}
{ \parskip 0pt plus 1pt
\tableofcontents }
\cleardoublepage
% PREPARE FOR MAIN TEXT
\pagenumbering{arabic}
\raggedbottom
\widowpenalty=10000
\clubpenalty=10000
\parindent=0pt
\renewcommand{\topfraction}{.8}
\renewcommand{\bottomfraction}{.8}
\renewcommand{\textfraction}{.2}
\renewcommand{\floatpagefraction}{.8}
\setstretch{1.15}
\newcommand{\path}[1]{{\tt #1}}
\part{Introduction and Tutorial}
\chapter{Introduction}
{\bf
DISCLAIMER: This documentation is currently under active development
and as such there may be mistakes and omissions --- watch out for
these and please report any you find to the developer's mailing list.
Contributions of material, suggestions and corrections are welcome.
}
Xen is a { \em paravirtualising } virtual machine monitor (VMM) or
``Hypervisor'' for the x86 processor architecture. Xen can securely
multiplex heterogeneous virtual machines on a single physical with
near-native performance. The virtual machine technology facilitates
enterprise-grade functionality, including:
\begin{itemize}
\item Virtual machines with close to native performance.
\item Live migration of running virtual machines.
\item Excellent hardware support (use unmodified Linux device drivers).
\item Suspend to disk / resume from disk of running virtual machines.
\item Transparent copy on write disks.
\item Sandboxed, restartable device drivers.
\item Pervasive debugging - debug whole OSes, from kernel to applications.
\end{itemize}
Xen support is available for increasingly many operating systems. The
following OSs have either been ported already or a port is in
progress:
\begin{itemize}
\item Dragonfly BSD
\item FreeBSD 4.8
\item Linux 2.4
\item Linux 2.6
\item NetBSD 2.0
\item Plan 9
\item Windows XP
\end{itemize}
Right now, Linux 2.4 and 2.6 are available for on Xen 2.0. NetBSD
port will be updated to run on Xen 2.0, hopefully in time for the Xen
2.0 release. It is intended that Xen support be integrated into the
official releases of Linux 2.6, NetBSD 2.0, FreeBSD and Dragonfly BSD.
Even running multiple copies of Linux can be very useful, providing a
means of containing faults to one OS image, providing performance
isolation between the various OS instances and trying out multiple
distros.
The Windows XP port is only available to those who have signed the
Microsoft Academic Source License.
Possible usage scenarios for Xen include:
\begin{description}
\item [Kernel development] test and debug kernel modifications in a
sandboxed virtual machine --- no need for a separate test
machine
\item [Multiple OS Configurations] run multiple operating systems
simultaneously, for instance for compatibility or QA purposes
\item [Server consolidation] move multiple servers onto one box,
provided performance and fault isolation at virtual machine
boundaries
\item [Cluster computing] improve manageability and efficiency by
running services in virtual machines, isolated from
machine-specifics and load balance using live migration
\item [High availability computing] run device drivers in sandboxed
domains for increased robustness
\item [Hardware support for custom OSes] export drivers from a
mainstream OS (e.g. Linux) with good hardware support
to your custom OS, avoiding the need for you to port existing
drivers to achieve good hardware support
\end{description}
\section{Structure}
\subsection{High level}
A Xen system has multiple layers. The lowest layer is Xen itself ---
the most privileged piece of code in the system. On top of Xen run
guest operating system kernels. These are scheduled pre-emptively by
Xen. On top of these run the applications of the guest OSs. Guest
OSs are responsible for scheduling their own applications within the
time allotted to them by Xen.
One of the domains --- { \em Domain 0 } --- is privileged. It is
started by Xen at system boot and is responsible for initialising and
managing the whole machine. Domain 0 builds other domains and manages
their virtual devices. It also performs suspend, resume and
migration of other virtual machines. Where it is used, the X server
is also run in domain 0.
Within Domain 0, a process called ``Xend'' runs to manage the system.
Xend is responsible for managing virtual machines and providing access
to their consoles. Commands are issued to Xend over an HTTP
interface, either from a command-line tool or from a web browser.
XXX need diagram(s) here to make this make sense
\subsection{Paravirtualisation}
Paravirtualisation allows very high performance virtual machine
technology, even on architectures (like x86) which are traditionally
hard to virtualise.
Paravirtualisation requires guest operating systems to be { \em ported
} to run on the VMM. This process is similar to a port of an
operating system to a new hardware platform. Although operating
system kernels must explicitly support Xen in order to run in a
virtual machine, { \em user space applications and libraries
do not require modification }.
\section{Hardware Support}
Xen currently runs on the x86 architecture, but could in principle be
ported to others. In fact, it would have been rather easier to write
Xen for pretty much any other architecture as x86 is particularly
tricky to handle. A good description of Xen's design, implementation
and performance is contained in the October 2003 SOSP paper, available
at:\\
{\tt http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf}\\
Work to port Xen to x86\_64 and IA64 is currently underway.
Xen is targeted at server-class machines, and the current list of
supported hardware very much reflects this, avoiding the need for us
to write drivers for "legacy" hardware. It is likely that some desktop
chipsets will fail to work properly with the default Xen
configuration: specifying {\tt noacpi} or {\tt ignorebiostables} when
booting Xen may help in these cases.
Xen requires a ``P6'' or newer processor (e.g. Pentium Pro, Celeron,
Pentium II, Pentium III, Pentium IV, Xeon, AMD Athlon, AMD Duron).
Multiprocessor machines are supported, and we also have basic support
for HyperThreading (SMT), although this remains a topic for ongoing
research. We're also working on an x86\_64 port (though Xen should
already run on these systems just fine in 32-bit mode).
Xen can currently use up to 4GB of memory. It is possible for x86
machines to address up to 64GB of physical memory but (unless an
external developer volunteers) there are no plans to support these
systems. The x86\_64 port is the planned route to supporting more
than 4GB of memory.
In contrast to previous Xen versions, in Xen 2.0 device drivers run
within a privileged guest OS rather than within Xen itself. This means
that we should be compatible with the majority of device hardware
supported by Linux. The default XenLinux build contains support for
relatively modern server-class network and disk hardware, but you can
add support for other hardware by configuring your XenLinux kernel in
the normal way (e.g. \verb_# make ARCH=xen xconfig_).
\section{History}
``Xen'' is a Virtual Machine Monitor (VMM) originally developed by the
Systems Research Group of the University of Cambridge Computer
Laboratory, as part of the UK-EPSRC funded XenoServers project.
The XenoServers project aims to provide a ``public infrastructure for
global distributed computing'', and Xen plays a key part in that,
allowing us to efficiently partition a single machine to enable
multiple independent clients to run their operating systems and
applications in an environment providing protection, resource
isolation and accounting. The project web page contains further
information along with pointers to papers and technical reports:
{\tt http://www.cl.cam.ac.uk/xeno}
Xen has since grown into a project in its own right, enabling us to
investigate interesting research issues regarding the best techniques
for virtualizing resources such as the CPU, memory, disk and network.
The project has been bolstered by support from Intel Research
Cambridge, and HP Labs, who are now working closely with us. We're
also in receipt of support from Microsoft Research Cambridge to port
Windows XP to run on Xen.
Xen was first described in the 2003 paper at SOSP \\
({\tt http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf}).
The first public release of Xen (1.0) was made in October 2003. Xen
was developed as a research project by the University of Cambridge
Computer Laboratory (UK). Xen was the first Virtual Machine Monitor
to make use of {\em paravirtualisation} to achieve near-native
performance virtualisation of commodity operating systems. Since
then, Xen has been extensively developed and is now used in production
scenarios on multiple sites.
Xen 2.0 is the latest release, featuring greatly enhanced hardware
support, configuration flexibility, usability and a larger complement
of supported operating systems. We think that Xen has the potential
to become {\em the} definitive open source virtualisation solution and
will work to conclusively achieve that position.
\chapter{Installation}
The Xen distribution includes three main components: Xen itself,
utilities to convert a standard Linux tree to run on Xen and the
userspace tools required to operate a Xen-based system.
This manual describes how to install the Xen 2.0 distribution from
source. Alternatively, there may be packages available for your
operating system distribution.
\section{Prerequisites}
\label{sec:prerequisites}
\begin{itemize}
\item A working installation of your favourite Linux distribution.
\item A working installation of the GRUB bootloader.
\item An installation of Twisted v1.3 or above (see {\tt
http://www.twistedmatrix.com}). There may be a package available for
your distribution; alternatively it can be installed by running {\tt \#
make install-twisted} in the root of the Xen source tree.
\item The Linux bridge control tools (see {\tt
http://bridge.sourceforge.net}). There may be a packages of these
tools available for your distribution.
\item Linux IP Routing Tools
\item make
\item python-dev
\item gcc
\item zlib-dev
\item libcurl
\item python2.3-pycurl
\item python2.3-twisted
\end{itemize}
\section{Optional}
\begin{itemize}
\item The Python logging package (see {\tt http://www.red-dove.com/})
for additional Xend logging functionality.
\end{itemize}
\section{Install Bitkeeper (Optional)}
To fetch a local copy, first download the BitKeeper tools at: \\ {\tt
http://www.bitmover.com/download }
The BitKeeper install program is designed to be run with X. If X is
not available, you can specify the install directory on the command
line.
\section{Download the Xen source code}
\subsection{Using Bitkeeper}
The public master BK repository for the 2.0 release lives at: \\
{\tt bk://xen.bkbits.net/xeno-unstable.bk}. You can use Bitkeeper to
download it and keep it updated with the latest features and fixes.
Change to the directory in which you want to put the source code, then
run:
\begin{verbatim}
# bk clone bk://xen.bkbits.net/xeno-unstable.bk
\end{verbatim}
Under your current directory, a new directory named 'xeno-unstable.bk'
has been created, which contains all the source code for the Xen
hypervisor and the Xen tools. The directory also contains `sparse'
Linux source trees, containing only the files that differ between
XenLinux and standard Linux.
Once you have cloned the repository, you can update to the newest
changes to the repository by running:
\begin{verbatim}
# cd xeno-unstable.bk # to change into the local repository
# bk pull # to update the repository
\end{verbatim}
\subsection{Without Bitkeeper}
The Xen source tree is also available in gzipped tarball form from the
Xen downloads page:\\
{\tt http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html}.
Prebuilt tarballs are also available but are very large.
\section{The distribution}
The Xen source code repository is structured as follows:
\begin{description}
\item[\path{tools/}] Xen node controller daemon (Xend), command line tools,
control libraries
\item[\path{xen/}] The Xen hypervisor itself.
\item[\path{linux-2.4.27-xen/}] Linux 2.4 support for Xen
\item[\path{linux-2.6.7-xen/}] Linux 2.6 support for Xen
\item[\path{doc/}] various documentation files for users and developers
\item[\path{extras/}] currently this contains the Mini OS, aimed at developers
\end{description}
\section{Build and install}
The Xen makefile includes a target ``world'' that will do the
following:
\begin{itemize}
\item Build Xen
\item Build the control tools, including Xend
\item Download the ebtables patch for Linux 2.4
\item Download (if necessary) and unpack the Linux 2.4 source code,
and patch it for use with Xen
\item Build a Linux kernel to use in domain 0 and a smaller
unprivileged kernel, which can optionally be used for
unprivileged virtual machines.
\end{itemize}
Inspect the Makefile if you want to see what goes on during a
build. Building Xen and the tools is straightforward, but XenLinux is
more complicated. The makefile needs a `pristine' linux kernel tree
which it will then add the Xen architecture files to. You can tell the
makefile the location of the appropriate linux compressed tar file by
setting the LINUX\_SRC environment variable, e.g. \\
\verb!# LINUX_SRC=/tmp/linux-2.4.27.tar.gz make world! \\ or by
placing the tar file somewhere in the search path of {\tt LINUX\_SRC\_PATH}
which defaults to ``{\tt .:..}". If the makefile can't find a suitable
kernel tar file it attempts to download it from kernel.org (this won't
work if you're behind a firewall).
After untaring the pristine kernel tree, the makefile uses the {\tt
mkbuildtree} script to add the Xen patches the kernel. It then builds
two different XenLinux images, one with a ``-xen0'' extension which
contains hardware device drivers and is intended to be used in the
first virtual machine (``domain 0''), and one with a ``-xenU''
extension that just contains virtual-device drivers.
The procedure is similar to build the Linux 2.6 port: \\
\verb!# LINUX_SRC=/path/to/linux2.6/source make linux26!
In both cases, if you have an SMP machine you may wish to give the
{\tt '-j4'} argument to make to get a parallel build.
The files produced by the build process are stored under the
\path{install/} directory. To install them in their default
locations, do: \\
\verb_# make install_\\
Alternatively, users with special installation requirements may wish
to install them manually by copying file to their appropriate
destinations.
Take a look at the files in \path{install/boot/}:
\begin{itemize}
\item \path{install/boot/xen.gz} The Xen 'kernel'
\item \path{install/boot/vmlinuz-2.4.27-xen0} Domain 0 XenLinux kernel
\item \path{install/boot/vmlinuz-2.4.27-xenU} Unprivileged XenLinux kernel
\end{itemize}
The difference between the two Linux kernels that are built is due to
the configuration file used for each. The "U" suffixed unprivileged
version doesn't contain any of the physical hardware device drivers
--- it is 30\% smaller and hence may be preferred for your
non-privileged domains. The ``0'' suffixed privileged version can be
used to boot the system, as well as in driver domains and unprivileged
domains.
The \path{install/boot} directory will also contain the config files
used for building the XenLinux kernels, and also versions of Xen and
XenLinux kernels that contain debug symbols (\path{xen-syms} and
\path{vmlinux-syms-2.4.27-xen0}) which are essential for interpreting crash
dumps. Retain these files as the developers may wish to see them if
you post on the mailing list.
\section{Configuration}
\subsection{GRUB Configuration}
An entry should be added to \path{grub.conf} (often found under
\path{/boot/} or \path{/boot/grub/}) to allow Xen / XenLinux to boot.
This file is sometimes called \path{menu.lst}, depending on your
distribution. The entry should look something like the following:
\begin{verbatim}
title Xen 2.0 / XenLinux 2.4.27
kernel /boot/xen.gz dom0_mem=131072 com1=115200,8n1
module /boot/vmlinuz-2.4.27-xen0 root=/dev/sda4 ro console=tty0 console=ttyS0
\end{verbatim}
The first line of the configuration (kernel...) tells GRUB where to
find Xen itself and what boot parameters should be passed to it. The
second line of the configuration describes the location of the
XenLinux kernel that Xen should start and the parameters that should
be passed to it.
As always when installing a new kernel, it is recommended that you do
not remove the original contents of \path{grub.conf} --- you may want
to boot up with your old Linux kernel in future, particularly if you
have problems.
\subsection{Serial Console}
In order to configure serial console output, it is necessary to add a
line into \path{/etc/inittab}. The XenLinux console driver is
designed to make this procedure the same as configuring a normal
serial console. Add the line:
{\tt c:2345:respawn:/sbin/mingetty ttyS0}
\section{Test the new install}
It should now be possible to restart the system and use Xen. Reboot
as usual but choose the new Xen option when the Grub screen appears.
What follows should look much like a conventional Linux boot. The
first portion of the output comes from Xen itself, supplying low level
information about itself and the machine it is running on. The
following portion of the output comes from XenLinux itself.
You may see some errors during the XenLinux boot. These are not
necessarily anything to worry about --- they may result from kernel
configuration differences between your XenLinux kernel and the one you
usually use.
When the boot completes, you should be able to log into your system as
usual. If you are unable to log in to your system running Xen, you
should still be able to reboot with your normal Linux kernel.
\chapter{Starting a domain}
The first step in creating a new domain is to prepare a root
filesystem for it to boot off. Typically, this might be stored in a
normal partition, a disk file and LVM volume, or on an NFS server.
A simple way to do this is simply to boot from your standard OS
install CD and install the distribution into another partition on your
hard drive.
{\em N.b } you can boot with Xen and XenLinux without installing any
special userspace tools but will need to have the prerequisites
described in Section~\ref{sec:prerequisites} and the Xen control tools
are installed before you proceed.
\section{From the web interface}
Boot the Xen machine and start Xensv (see Chapter~\ref{cha:xensv} for
more details) using the command: \\
\verb_# xensv start_ \\
This will also start Xend (see Chapter~\ref{cha:xend} for more information).
The domain management interface will then be available at {\tt
http://your\_machine:8080/}. This provides a user friendly wizard for
starting domains and functions for managing running domains.
\section{From the command line}
Full details of the {\tt xm} tool are found in Chapter~\ref{cha:xm}.
This example explains how to use the \path{xmdefaults} file. If you
require a more complex setup, you will want to write a custom
configuration file --- details of the configuration file formats are
included in Chapter~\ref{cha:config}.
The \path{xmdefaults} file is a template description that is intended
to be reused for multiple virtual machines. Setting the value of the
{\tt vmid} variable fills in parts of this template.
\subsection{Editing \path{xmdefaults}}
At minimum, you should edit the following variables in \path{xmdefaults}:
\begin{description}
\item[kernel] Set this to the path of the kernel you compiled for use
with Xen. [e.g. {\tt kernel =
'/root/xeno-unstable.bk/install/boot/vmlinuz-2.4.27-xenU'}]
\item[memory] Set this to the size of the domain's memory in
megabytes. [e.g. {\tt memory = 64 } ]
\item[disk] Set the first entry in this list to calculate the offset
of the domain's root partition, based on the domain ID. Set the
second to the location of \path{/usr} (if you are sharing it between
domains). [i.e. {\tt disk = ['phy:your\_hard\_drive\%d,sda1,w' \%
(base\_partition\_number + vmid), 'phy:your\_usr\_partition,sda6,r' ]}
\item[dhcp] Uncomment the dhcp variable, so that the domain will
receive its IP address from a DHCP server. [i.e. {\tt dhcp=''dhcp''}]
\end{description}
You may also want to edit the {\bf vif} variable in order to choose
the MAC address of the virtual ethernet interface yourself. For
example: \\ \verb_vif = [`mac=00:06:AA:F6:BB:B3']_\\ If you do not set
this variable, Xend will automatically generate a random MAC address
from an unused range.
\subsection{Starting the domain}
The {\tt xm} tool provides a variety of commands for managing domains.
Use the {\tt create} command to start new domains. To start the
virtual machine with virtual machine ID 1.
\begin{verbatim}
# xm create -c vmid=1
\end{verbatim}
The {\tt -c} switch causes {\tt xm} to turn into the domain's console
after creation. The {\tt vmid=1} sets the {\tt vmid} variable used in
the {\tt xmdefaults} file. The tool uses the
\path{/etc/xen/xmdefaults} file, since no custom configuration file
was specified on the command line.
\chapter{Domain management tasks}
The previous chapter described a simple example of how to configure
and start a domain. This chapter summarises the tools available to
manage running domains.
\section{Command line management}
Command line management tasks are also performed using the {\tt xm}
tool. For online help for the commands available, type:\\
\verb_# xm help_
\subsection{Basic management commands}
The most important {\tt xm} commands are: \\
\verb_# xm list_ : Lists all domains running. \\
\verb_# xm consoles_ : Gives information about the domain consoles. \\
\verb_# xm console_: open a console to a domain.
e.g. \verb_# xm console 1_ (open console to domain 1)
\subsection{\tt xm list}
The output of {\tt xm list} is in rows of the following format:\\
\verb_domid name memory cpu state cputime_
\begin{description}
\item[domid] The number of the domain ID this virtual machine is running in.
\item[name] The descriptive name of the virtual machine.
\item[memory] Memory size in megabytes.
\item[cpu] The CPU this domain is running on.
\item[state] Domain state consists of 5 fields:
\begin{description}
\item[r] running
\item[b] blocked
\item[p] paused
\item[s] shutdown
\item[c] crashed
\end{description}
\item[cputime] How much CPU time (in seconds) the domain has used so far.
\end{description}
The {\tt xm list} command also supports a long output format when the
{\tt -l} switch is used. This outputs the fulls details of the
running domains in Xend's SXP configuration format.
\chapter{Other kinds of storage}
It is possible to use any Linux block device to store virtual machine
disk images. This chapter covers some of the possibilities; note that
it is also possible to use network-based block devices and other
unconventional block devices.
\section{File-backed virtual block devices}
It is possible to use a file in Domain 0 as the primary storage for a
virtual machine. As well as being convenient, this also has the
advantage that the virtual block device will be {\em sparse} --- space
will only really be allocated as parts of the file are used. So if a
virtual machine uses only half its disk space then the file really
takes up a half of the size allocated.
For example, to create a 2GB sparse file-backed virtual block device
(actually only consumes 1KB of disk):
\verb_# dd if=/dev/zero of=vm1disk bs=1k seek=2048k count=1_
Choose a free loop back device, and attach file: \\
\verb_# losetup /dev/loop0 vm1disk_ \\
Make a file system on the loop back device: \\
\verb_# mkfs -t ext3 /dev/loop0_
Populate the file system e.g. by copying from the current root:
\begin{verbatim}
# mount /dev/loop0 /mnt
# cp -ax / /mnt
\end{verbatim}
Tailor the file system by editing \path{/etc/fstab},
\path{/etc/hostname}, etc (don't forget to edit the files in the
mounted file system, instead of your domain 0 filesystem, e.g. you
would edit \path{/mnt/etc/fstab} instead of \path{/etc/fstab} ). For
this example put \path{/dev/sda1} to root in fstab.
Now unmount (this is important!):\\
\verb_# umount /dev/loop0_
In the configuration file set:\\
\verb_disk = [`phy:loop0,sda1,w']_
As the virtual machine writes to its `disk', the sparse file will be
filled in and consume more space up to the original 2GB.
{\em NB.} You will need to use {\tt losetup} to bind the file to
\path{/dev/loop0} (or whatever loopback device you chose) each time
you reboot domain 0. In the near future, Xend will track which loop
devices are currently free and do binding itself, making this manual
effort unnecessary.
\section{LVM-backed virtual block devices}
XXX Put some simple examples here - would be nice if an LVM user could
contribute some, although obviously users would have to read the LVM
docs to do advanced stuff.
\part{Quick Reference}
\chapter{Domain Configuration Files}
\label{cha:config}
XXX Could use a little explanation about possible values
Xen configuration files contain the following standard variables:
\begin{description}
\item[kernel] Path to the kernel image (on the server).
\item[ramdisk] Path to a ramdisk image (optional).
\item[builder] The name of the domain build function (e.g. {\tt'linux'} or {\tt'netbsd'}.
\item[memory] Memory size in megabytes.
\item[cpu] CPU to assign this domain to.
\item[nics] Number of virtual network interfaces.
\item[vif] List of MAC addresses (random addresses are assigned if not given).
\item[disk] Regions of disk to export to the domain.
\item[dhcp] Set to {\tt 'dhcp'} if you want to DHCP allocate the IP address.
\item[netmask] IP netmask.
\item[gateway] IP address for the gateway (if any).
\item[hostname] Set the hostname for the virtual machine.
\item[root] Set the root device.
\item[nfs\_server] IP address for the NFS server.
\item[nfs\_root] Path of the root filesystem on the NFS server.
\item[extra] Extra string to append to the kernel command line.
\item[restart] Three possible options:
\begin{description}
\item[always] Always restart the domain, no matter what
its exit code is.
\item[never] Never restart the domain.
\item[onreboot] (restart the domain if it requests reboot).
\end{description}
\end{description}
It is also possible to include Python scripting commands in
configuration files. This is done in the \path{xmdefaults} file in
order to handle the {\tt vmid} variable.
\chapter{Xend (Node control daemon)}
\label{cha:xensv}
The Xen Daemon (Xend) performs system management functions related to
virtual machines. It forms a central point of control for a machine
and can be controlled using an HTTP-based protocol. Xend must be
running in order to start and manage virtual machines.
Xend must be run as root because it needs access to privileged system
management functions. A small set of commands may be issued on the
Xend command line:
\begin{tabular}{ll}
\verb_# xend start_ & start Xend, if not already running \\
\verb_# xend stop_ & stop Xend if already running \\
\verb_# xend restart_ & restart Xend if running, otherwise start it \\
\end{tabular}
An SysV init script called {\tt xend} is provided to start Xend at
boot time. The {\tt make install} will install this script in
{\path{/etc/init.d} automatically. To enable it, you can make
symbolic links in the appropriate runlevel directories or use the {\tt
chkconfig} tool, where available.
Once Xend is running, more sophisticated administration can be done
using the Xensv web interface (see Chapter~\ref{cha:xensv}).
\chapter{Xensv (Web interface server)}
\label{cha:xensv}
Xensv is the server for the web control interface. It can be started
using:\\
\verb_# xensv start_ \\
and stopped using:
\verb_# xensv stop_ \\
It will automatically start Xend if it is not already running.
By default, Xensv will serve out the web interface on port 8080. This
can be changed by editing {\tt
/usr/lib/python2.2/site-packages/xen/sv/params.py}.
Once Xensv is running, the web interface can be used to manage running
domains and provides a user friendly domain creation wizard.
\chapter{The xm tool}
\label{cha:xm}
XXX Add description of arguments and switches for all the options
The xm tool is the primary tool for managing Xen from the console.
The general format of an xm command line is:
\begin{verbatim}
# xm command [switches] [arguments] [variables]
\end{verbatim}
The available {\em switches } and {\em arguments}are dependent on the
{\em command} chosen. The {\em variables} may be set using
declarations of the form {\tt variable=value} and may be used to set /
override any of the values in the configuration file being used,
including the standard variables described above and any custom
variables (for instance, the \path{xmdefaults} file uses a {\tt vmid}
variable).
The available commands are as follows:
\begin{description}
\item[create] Create a new domain.
\item[destroy] Kill a domain immediately.
\item[list] List running domains.
\item[shutdown] Ask a domain to shutdown.
\item[dmesg] Fetch the Xen (not Linux!) boot output.
\item[consoles] Lists the available consoles.
\item[console] Connect to the console for a domain.
\item[help] Get help on xm commands.
\item[save] Suspend a domain to disk.
\item[restore] Restore a domain from disk.
\item[pause] Pause a domain's execution.
\item[unpause] Unpause a domain.
\item[pincpu] Pin a domain to a CPU.
\item[bvt] Set BVT scheduler parameters for a domain.
\item[bvt\_ctxallow] Set the BVT context switching allowance for the system.
\item[fbvt] Set the FBVT scheduler parameters for a domain.
\item[fbvt\_ctxallow] Set the FBVT context switching allowance for the system.
\item[atropos] Set the atropos parameters for a domain.
\item[rrobin] Set the round robin time slice for the system.
\item[info] Get information about the Xen host.
\item[call] Call a Xend HTTP API function directly.
\end{description}
\chapter{Glossary}
\begin{description}
\item[Atropos] One of the CPU schedulers provided by Xen.
Atropos provides domains with absolute shares
of the CPU, with timeliness guarantees and a
mechanism for sharing out ``slack time''.
\item[BVT] The BVT scheduler is used to give proportional
fair shares of the CPU to domains.
\item[Exokernel] A minimal piece of privileged code, similar to
a {\bf microkernel} but providing a more
`hardware-like' interface to the tasks it
manages. This is similar to a paravirtualising
VMM like {\bf Xen} but was designed as a new
operating system structure, rather than
specifically to run multiple conventional OSs.
\item[FBVT] A derivative of the { \bf BVT } scheduler that
aims to give better fairness performance to IO
intensive domains in competition with CPU
intensive domains.
\item[Domain] A domain is the execution context that
contains a running { \bf virtual machine }.
The relationship between virtual machines
and domains on Xen is similar to that between
programs and processes in an operating
system: a virtual machine is a persistent
entity that resides on disk (somewhat like
a program). When it is loaded for execution,
it runs in a domain. Each domain has a
{ \bf domain ID }.
\item[Domain 0] The first domain to be started on a Xen
machine. Domain 0 is responsible for managing
the system.
\item[Domain ID] A unique identifier for a { \bf domain },
analogous to a process ID in an operating
system. Apart from domain
\item[Full virtualisation] An approach to virtualisation which
requires no modifications to the hosted
operating system, providing the illusion of
a complete system of real hardware devices.
\item[Hypervisor] An alternative term for { \bf VMM }, used
because it means ``beyond supervisor'',
since it is responsible for managing multiple
``supervisor'' kernels.
\item[Microkernel] A small base of code running at the highest
hardware privilege level. A microkernel is
responsible for sharing CPU and memory (and
sometimes other devices) between less
privileged tasks running on the system.
This is similar to a VMM, particularly a
{\bf paravirtualising} VMM but typically
addressing a different problem space and
providing different kind of interface.
\item[NetBSD/Xen] A port of NetBSD to the Xen architecture.
\item[Paravirtualisation] An approach to virtualisation which requires
modifications to the operating system in
order to run in a virtual machine. Xen
uses paravirtualisation but preserves
binary compatibility for user space
applications.
\item[Virtual Machine] The environment in which a hosted operating
system runs, providing the abstraction of a
dedicated machine. A virtual machine may
be identical to the underlying hardware (as
in { \bf full virtualisation }, or it may
differ, as in { \bf paravirtualisation }.
\item[VMM] Virtual Machine Monitor - the software that
allows multiple virtual machines to be
multiplexed on a single physical machine.
\item[Xen] Xen is a paravirtualising virtual machine
monitor, developed primarily by the
Systems Research Group at the University
of Cambridge Computer Laboratory.
\item[XenLinux] Official name for the port of the Linux kernel
that runs on Xen.
\end{description}
\part{Advanced Topics}
XXX More to add here, including config file format
\chapter{Advanced Network Configuration}
For simple systems with a single ethernet interface with a simple
configuration, the default installation should work ``out of the
box''. More complicated network setups, for instance with multiple
ethernet interfaces and / or existing bridging setups will require
some special configuration.
The purpose of this chapter is to describe the mechanisms provided by
xend to allow a flexible configuration for Xen's virtual networking.
\section{Xen networking scripts}
Xen's virtual networking is configured by 3 shell scripts. These are
called automatically by Xend when certain events occur, with arguments
to the scripts providing further contextual information. These
scripts are found by default in \path{/etc/xen}. The names and
locations of the scripts can be configured in \path{xend-config.sxp}.
\subsection{\path{network}}
This script is called once when Xend is started and once when Xend is
stopped. Its job is to do any advance preparation required for the
Xen virtual network when Xend starts and to do any corresponding
cleanup when Xend exits.
In the default configuration, this script creates the bridge
``xen-br0'' and moves eth0 onto that bridge, modifying the routing
accordingly.
In configurations where the bridge already exists, this script could
be replaced with a link to \path{/bin/true} (for instance).
When Xend exits, this script is called with the {\tt stop} argument,
which causes it to delete the Xen bridge and remove {\tt eth0} from
it, restoring the normal IP and routing configuration.
\subsection{\path{vif-bridge}}
This script is called for every domain virtual interface. This should
do things like configuring firewalling rules for that interface and
adding it to the appropriate bridge.
By default, this adds and removes VIFs on the default Xen bridge.
This script can be customized to properly deal with more complicated
bridging setups.
\chapter{Advanced Scheduling Configuration}
\section{Scheduler selection}
Xen offers a boot time choice between multiple schedulers. To select
a scheduler, pass the boot parameter { \tt sched=sched\_name } to Xen,
substituting the appropriate scheduler name. Details of the schedulers
and their parameters are included below; future versions of the tools
will provide a higher-level interface to these tools.
\section{Borrowed Virtual Time}
BVT provides proportional fair shares of the CPU time. It has been
observed to penalise domains that block frequently (e.g. IO intensive
domains), so the FBVT derivative has been included as an alternative.
\subsection{Global Parameters}
\begin{description}
\item[ctx\_allow]
the context switch allowance is similar to the "quantum"
in traditional schedulers. It is the minimum time that
a scheduled domain will be allowed to run before be
pre-empted. This prevents thrashing of the CPU.
\end{description}
\subsection{Per-domain parameters}
\begin{description}
\item[mcuadv]
the MCU (Minimum Charging Unit) advance determines the
proportional share of the CPU that a domain receives. It
is set inversely proportionally to a domain's sharing weight.
\item[warp]
the amount of "virtual time" the domain is allowed to warp
backwards
\item[warpl]
the warp limit is the maximum time a domain can run warped for
\item[warpu]
the unwarp requirement is the minimum time a domain must
run unwarped for before it can warp again
\end{description}
\section{Fair Borrowed Virtual Time}
This is a derivative for BVT that aims to provide better fairness for
IO intensive domains as well as for CPU intensive domains.
\subsection{Global Parameters}
Same as for BVT.
\subsection{Per-domain parameters}
Same as for BVT.
\section{Atropos}
Atropos is a Soft Real Time scheduler. It provides guarantees about
absolute shares of the CPU (with a method for optionally sharing out
slack CPU time on a best-effort basis) and can provide timeliness
guarantees for latency-sensitive domains.
\subsection{Per-domain parameters}
\begin{description}
\item[slice]
The length of time per period that a domain is guaranteed.
\item[period]
The period over which a domain is guaranteed to receive
its slice of CPU time.
\item[latency]
The latency hint is used to control how soon after
waking up a domain should be scheduled.
\item[xtratime]
This is a true (1) / false (0) flag that specifies whether
a domain should be allowed a share of the system slack time.
\end{description}
\section{Round Robin}
The Round Robin scheduler is included as a simple demonstration of
Xen's internal scheduler API. It is not intended for production use
--- the other schedulers included are all more general and should give
higher throughput.
\subsection{Global parameters}
\begin{description}
\item[rr\_slice]
The maximum time each domain runs before the next
scheduling decision is made.
\end{description}
\chapter{Privileged domains}
There are two possible types of privileges: IO privileges and
administration privileges.
\section{Driver domains (IO Privileges)}
IO privileges can be assigned to allow a domain to drive PCI devices
itself. This is used for to support driver domains.
Setting backend privileges is currently only supported in SXP format
config files (??? is this true - there's nothing in xmdefaults,
anyhow). To allow a domain to function as a backend for others,
somewhere within the {\tt vm} element of its configuration file must
be a {\tt backend} element of the form {\tt (backend ({\em type}))}
where {\tt \em type} may be either {\tt netif} or {\tt blkif},
according to the type of virtual device this domain will service.
After this domain has been built, Xend will connect all new and
existing {\em virtual} devices (of the appropriate type) to that
backend.
Note that:
\begin{itemize}
\item a block backend cannot import virtual block devices from other
domains
\item a network backend cannot import virtual network devices from
other domains
\end{itemize}
Thus (particularly in the case of block backends, which cannot import
a virtual block device as their root filesystem), you may need to boot
a backend domain from a ramdisk or a network device.
The privilege to drive PCI devices may also be specified on a
per-device basis. Xen will assign the minimal set of hardware
privileges to a domain that are required to control its devices. This
can be configured in either format of configuration file:
\begin{itemize}
\item SXP Format:
Include {\tt device} elements
{\tt (device (pci (bus {\em x}) (dev {\em y}) (func {\em z}))) } \\
inside the top-level {\tt vm} element. Each one specifies the address
of a device this domain is allowed to drive ---
the numbers {\em x},{\em y} and {\em z} may be in either decimal or
hexadecimal format.
\item Flat Format: Include a list of PCI device addresses of the
format: \\ {\tt pci = ['x,y,z', ...] } \\ where each element in the
list is a string specifying the components of the PCI device
address, separated by commas. The components ({\tt \em x}, {\tt \em
y} and {\tt \em z}) of the list may be formatted as either decimal
or hexadecimal.
\end{itemize}
\section{Administration Domains}
Administration privileges allow a domain to use the ``dom0
operations'' (so called because they are usually available only to
domain 0). A privileged domain can build other domains, set scheduling
parameters, etc.
% Support for other administrative domains is not yet available...
\chapter{Xen build options}
For most users, the default build of Xen will be adequate. For some
advanced uses, Xen provides a number of build-time options:
At build time, these options should be set as environment variables or
passed on make's command-line. For example:
\begin{verbatim}
export option=y; make
option=y make
make option1=y option2=y
\end{verbatim}
\section{List of options}
{\bf debug=y }\\
Enable debug assertions and console output.
(Primarily useful for tracing bugs in Xen). \\
{\bf debugger=y }\\
Enable the in-Xen pervasive debugger (PDB).
This can be used to debug Xen, guest OSes, and
applications. For more information see the
XenDebugger-HOWTO. \\
{\bf perfc=y }\\
Enable performance-counters for significant events
within Xen. The counts can be reset or displayed
on Xen's console via console control keys. \\
{\bf trace=y }\\
Enable per-cpu trace buffers which log a range of
events within Xen for collection by control
software. For more information see the chapter on debugging,
in the Xen Interface Manual.
\chapter{Xen boot options}
These options are used to configure Xen's behaviour at runtime. They
should be appended to Xen's command line, either manually or by
editing \path{grub.conf}.
\section{List of options}
{\bf ignorebiostables }\\
Disable parsing of BIOS-supplied tables. This may help with some
chipsets that aren't fully supported by Xen. If you specify this
option then ACPI tables are also ignored, and SMP support is
disabled. \\
{\bf noreboot } \\
Don't reboot the machine automatically on errors. This is
useful to catch debug output if you aren't catching console messages
via the serial line. \\
{\bf nosmp } \\
Disable SMP support.
This option is implied by 'ignorebiostables'. \\
{\bf noacpi } \\
Disable ACPI tables, which confuse Xen on some chipsets.
This option is implied by 'ignorebiostables'. \\
{\bf watchdog } \\
Enable NMI watchdog which can report certain failures. \\
{\bf noht } \\
Disable Hyperthreading. \\
{\bf badpage=$<$page number$>$[,$<$page number$>$] } \\
Specify a list of pages not to be allocated for use
because they contain bad bytes. For example, if your
memory tester says that byte 0x12345678 is bad, you would
place 'badpage=0x12345' on Xen's command line (i.e., the
last three digits of the byte address are not
included!). \\
{\bf com1=$<$baud$>$,DPS[,$<$io\_base$>$,$<$irq$>$] \\
com2=$<$baud$>$,DPS[,$<$io\_base$>$,$<$irq$>$] } \\
Xen supports up to two 16550-compatible serial ports.
For example: 'com1=9600,8n1,0x408,5' maps COM1 to a
9600-baud port, 8 data bits, no parity, 1 stop bit,
I/O port base 0x408, IRQ 5.
If the I/O base and IRQ are standard (com1:0x3f8,4;
com2:0x2f8,3) then they need not be specified. \\
{\bf console=$<$specifier list$>$ } \\
Specify the destination for Xen console I/O.
This is a comma-separated list of, for example:
\begin{description}
\item[vga] use VGA console and allow keyboard input
\item[com1] use serial port com1
\item[com2H] use serial port com2. Transmitted chars will
have the MSB set. Received chars must have
MSB set.
\item[com2L] use serial port com2. Transmitted chars will
have the MSB cleared. Received chars must
have MSB cleared.
\end{description}
The latter two examples allow a single port to be
shared by two subsystems (e.g. console and
debugger). Sharing is controlled by MSB of each
transmitted/received character.
[NB. Default for this option is 'com1,tty'] \\
{\bf conswitch=$<$switch-char$><$auto-switch-char$>$ } \\
Specify how to switch serial-console input between
Xen and DOM0. The required sequence is CTRL-<switch-char>
pressed three times. Specifying '`' disables switching.
The <auto-switch-char> specifies whether Xen should
auto-switch input to DOM0 when it boots -- if it is 'x'
then auto-switching is disabled. Any other value, or
omitting the character, enables auto-switching.
[NB. Default for this option is 'a'] \\
{\bf nmi=xxx } \\
Specify what to do with an NMI parity or I/O error. \\
'nmi=fatal': Xen prints a diagnostic and then hangs. \\
'nmi=dom0': Inform DOM0 of the NMI. \\
'nmi=ignore': Ignore the NMI. \\
{\bf dom0\_mem=xxx } \\
Set the maximum amount of memory for domain0. \\
{\bf tbuf\_size=xxx } \\
Set the size of the per-cpu trace buffers, in pages
(default 1). Note that the trace buffers are only
enabled in debug builds. Most users can ignore
this feature completely. \\
{\bf sched=xxx } \\
Select the CPU scheduler Xen should use. The current
possibilities are 'bvt', 'atropos' and 'rrobin'. The
default is 'bvt'. For more information see
Sched-HOWTO.txt. \\
{\bf pci\_dom0\_hide=(xx.xx.x)(yy.yy.y)... } \\
Hide selected PCI devices from domain 0 (for instance, to stop it
taking ownership of them so that they can be driven by another
domain). Device IDs should be given in hex format. Bridge devices do
not need to be hidden --- they are hidden implicitly, since guest OSes
do not need to configure them.
\chapter{Further Support}
If you have questions that are not answered by this manual, the
sources of information listed below may be of interest to you. Note
that bug reports, suggestions and contributions related to the
software (or the documentation) should be sent to the Xen developers'
mailing list (address below).
\section{Other documentation}
For developers interested in porting operating systems to Xen, the
{\em Xen Interface Manual} is distributed in the \path{docs/}
directory of the Xen source distribution. Various HOWTOs are
available in \path{docs/HOWTOS} but this content is being integrated
into this manual.
\section{Online references}
The official Xen web site is found at: \\
{\tt
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/] }.
Links to other
documentation sources are listed at: \\ {\tt
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/documentation.html}.
\section{Mailing lists}
There are currently two official Xen mailing lists:
\begin{description}
\item[xen-devel@lists.sourceforge.net] Used for development
discussions and requests for help. Subscribe at: \\
{\tt http://lists.sourceforge.net/mailman/listinfo/xen-devel}
\item[xen-announce@lists.sourceforge.net] Used for announcements only.
Subscribe at: \\
{\tt http://lists.sourceforge.net/mailman/listinfo/xen-announce}
\end{description}
Although there is no specific user support list, the developers try to
assist users who post on xen-devel. As the bulk of traffic on this
list increases, a dedicated user support list may be introduced.
\end{document}