xen/README.CD

#############################
 __  __            _   ___  
 \ \/ /___ _ __   / | / _ \ 
  \  // _ \ '_ \  | || | | |
  /  \  __/ | | | | || |_| |
 /_/\_\___|_| |_| |_(_)___/ 
                                                   
#############################

 XenDemoCD 1.0 
 University of Cambridge Computer Laboratory
 29 Sep 2003

 http://www.cl.cam.ac.uk/netos/xen

Welcome to the Xen Demo CD! 

Executive Summary
=================

This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM)
and Linux-2.4 OS port (XenoLinux). It runs entirely off the CD,
without requiring hard disk installation. This is achieved using a RAM
disk to store mutable file system data while using the CD for
everything else. The CD can also be used for installing Xen/XenoLinux
to disk, and includes a source code snapshot along with all of the
tools required to build it.

Booting the CD
==============

The Xen VMM is currently fairly h/w specific, but porting new device
drivers is relatively straightforward thanks to Xen's Linux driver
compatibility layer. The current snapshot supports the following
hardware:

CPU:  Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported
IDE:  Intel PIIX chipset, others will be PIO only (slow)
SCSI: Adaptec / Dell PERC Raid (aacraid), megaraid, Adaptec aic7xxx
Net:  Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x)
      Working, but require extra copies : pcnet32, Intel e100, tulip

Because of the demo CD's use of RAM disks, make sure you have plenty
of RAM (256MB+).

To try out the Demo, boot from CD (you may need to change your BIOS
configuration to do this), then select one of the four boot options 
from the Grub menu:

 Xen / linux-2.4.22 
 Xen / linux-2.4.22 using cmdline IP configuration
 Xen / linux-2.4.22 in "safe mode"
 linux-2.4.22

The last option is a plain linux kernel that runs on the bare machine,
and is included simply to help diagnose driver compatibility
problems. The "safe mode" boot option might be useful if you're having
problems getting Xen to work with your hardware, as it disables various
features such as SMP, and enables some debugging.

If you are going for a command line IP config, hit "e" at
the grub menu, then edit the "ip=" parameters to reflect your setup
e.g. "ip=<ipaddr>::<gateway>:<netmask>::eth0:off". It shouldn't be
necessary to set either the nfs server or hostname
parameters. Alternatively, once XenoLinux has booted you can login and
setup networking with 'dhclient' or 'ifconfig' and 'route' in the
normal way. 

To make things easier for yourself, it's worth trying to arrange for an
IP address which is the first in a sequential range of free IP
addresses.  It's useful to give each VM instance its own public IP
address (though it is possible to do NAT or use private addresses), 
and the configuration files on the CD allocate IP addresses
sequentially for subsequent domains unless told otherwise.

After selecting the kernel to boot, stand back and watch Xen boot,
closely followed by "domain 0" running the XenoLinux kernel. The boot
messages can also sent to the serial line by specifying the baud rate
on the Xen cmdline (e.g., 'ser_baud=9600'); this can be very useful
for debugging should anything important scroll off the screen. Xen's
startup messages will look quite familiar as much of the hardware
initialisation (SMP boot, apic setup) and device drivers are derived
from Linux.

If everything is well, you should see the linux rc scripts start a
bunch of standard services including sshd.  Login on the console or
via ssh::
 username: user         root
 password: xendemo      xendemo

Once logged in, it should look just like any regular linux box. All
the usual tools and commands should work as per usual.  However,
because of the poor random access performance of CD drives, the
machine will feel rather slugish, and you may run out of memory if you
make significant modifications to the ramfs filesystem -- for the full
experience, install a Xen and XenoLinux image on you hard drive :-)

You can configure networking, either with 'dhclient' or manually via
'ifconfig' and 'route', remembering to edit /etc/resolv.conf if you
want DNS.

You can start an X server with 'startx'. It defaults to a conservative
1024x768, but you can edit the script for higher resoloutions.  The CD
contains a load of standard software. You should be able to start
Apache, PostgreSQL, Mozilla etc in the normal way, but because
everything is running off CD the performance will be very sluggish and
you may run out of memory for the 'tmpfs' file system.  You may wish
to go ahead and install Xen/XenoLinux on your hard drive, either
dropping Xen and the XenoLinux kernel down onto a pre-existing Linux
distribution, or using the file systems from the CD (which are based
on RH9). See the installation instructions later in this document.

If your video card requires 'agpgart' then it unfortunately won't yet
work with Xen, and you'll only be able to configure a VGA X
server. We're working on a fix for this for the next release.

If you want to browse the Xen / XenoLinux source, it's all located
under /usr/local/src, complete with BitKeeper repository. We've also
included source code and configuration information for the various
benchmarks we used in the SOSP paper.


Starting other domains
======================

There's a web interface for starting and managing other domains (VMs),
but since we generally use the command line tools they're probably
rather better debugged at present. The key command is 'xenctl' which
lives in /usr/local/bin and uses /etc/xenctl.xml for its default
configuration. Run 'xenctl' without any arguments to get a help
message. Note that xenctl is a java front end to various underlying 
internal tools written in C (xi_*). Running off CD, it seems to take 
an age to start...

Abyway, the first thing to do is to set up a window in which you will
receive console output from other domains. Console output will arrive
as UDP packets destined for 169.254.1.0, so its necessary to setup an
alias on eth0. The easiest way to do this is to run:

  xen_nat_enable

This also inserts a few NAT rules into "domain0", in case you'll be
starting other domains without their own IP addresses. Alternatively,
just do "ifconfig eth0:0 169.254.1.0 up". NB: The intention is that in
future Xen will do NAT itsel (actually RSIP), but this is part of a
larger work package that isn't stable enough to release.

Next, run a the xen UDP console displayer:

  xen_read_console &


As mentioned above, xenctl uses /etc/xenctl.xml as its default
configuration. The directory contains two different configs depending
on whether you want to use NAT, or multiple sequential external IPs
(it's possible to override any of the parameters on the command line
if you want to set specific IPs, etc).

The default configuration file supports NAT. To change to use multiple IPs:

 cp /etc/xenctl.xml-publicip /etc/xenctl.xml

A sequence of commands must be given to xenctl to start a new 
domain. First a new domain must be created, which requires specifying
the initial memory allocation, the kernel image to use, and the kernel
command line. As well as the root file system details, you'll need to
set the IP address on the command line: since Xen currently doesn't
support a virtual console for domains >1, you won't be able to log to
your new domain unless you've got networking configured and an sshd
running! (using dhcp for new domains should work too).

After creating the domain, xenctl must be used to grant the domain
access to other resources such as physical or virtual disk partions.
Then, the domain must be started.

These commands can be entered manually, but for convenience, xenctl
will also read them from a script and infer which domain number you're
referring to (-nX). To use the sample script:

 xenctl script -f/etc/xen-mynewdom     [NB: no space after the -f]

You should see the domain booting on your xen_read_console window.

The xml defaults start another domain running off the CD, using a
separate RAM-based file system for mutable data in root (just like
domain 0). 

The new domain is started with a '4' on the kernel command line to
tell 'init' to go to runlevel 4 rather than the default of 3. This is
done simply to suppress a bunch of harmless error messages that would
otherwise occur when the new (unprivileged) domain tried to access
physical hardware resources to try setting the hwclock, system font,
run gpm etc.

After it's booted, you should be able to ssh into your new domain. If
you went for a NATed address, from domain 0 you should be able to ssh
into '169.254.1.X' where X is the domain number.  If you ran the
xen_enable_nat script, a bunch of port redirects have been installed
to enable you to ssh in to other domains remotely.  To access the new
virtual machine remotely, use:

 ssh -p2201 root@IP.address.Of.Domain0  # use 2202 for domain 2 etc.

If you configured the new domain with its own IP address, you should
be able to ssh into it directly.


"xenctl domain list" provides status information about running domains,
though is currently only allowed to be run by domain 0. It accesses
/proc/xeno/domains to read this information from Xen. You can also use
xenctl to 'stop' (pause) a domain, or 'kill' a domain. You can either
kill it nicely by sending a shutdown event and waiting for it to
terminate, or blow the sucker away with extreme prejudice. 

If you want to configure a new domain differently, type 'xenctl' to
get a list of arguments, e.g. at the 'xenctl domain new' command line
use the "-4" option to set a diffrent IPv4 address. 

xenctl can be used to set the new kernel's command line, and hence
determine what it uses as a root file system, etc. Although the default
is to boot in the same manner that domain0 did (using the RAM-based
file system for root and the CD for /usr) it's possible to configure any
of the following possibilities, for example:

 * initrd=/boot/initrd init=/linuxrc
     boot using an initial ram disk, executing /linuxrc (as per this CD)     

 * root=/dev/hda3 ro
     boot using a standard hard disk partition as root
     !!! remember to do "xenctl physical grant -phda3 -w -n<dom>" first
     (grant domain <dom> read/write access to partition 3)

 * root=/dev/xvda1 ro
     boot using a pre-configured 'virtual block device' that will be
     attached to a virtual disk that previously has had a file system
     installed on it.

 * root=/dev/nfs nfsroot=/path/on/server ip=<blah_including server_IP>
     Boot using an NFS mounted root file system. This could be from a
     remote NFS server, or from an NFS server running in another
     domain. The latter is rather a useful option.


A typical setup might be to allocate a standard disk partition for
each domain and populate it with files. To save space, having a shared
read-only usr partition might make sense.

Alternatively, you can use 'virtual disks', which are stored as files
within a custom file system. "xenctl partitions add" can be used to
'format' a partition with the file system, and then virtual disks can
be created with "xenctl vd create". Virtual disks can then be attached
to a running domain as a 'virtual block device' using "xenctl vbd
create". The virtual disk can optionally be partitioned (e.g. "fdisk
/dev/xvda") or have a file system created on it directly (e.g. "mkfs
-t ext3 /dev/xvda").  The virtual disk can then be accessed by a
virtual block device associated with another domain, and even used as
a boot device.

Both virtual disks and real partitions should only be shared between
domains in a read-only fashion otherwise the linux kernels will
obviously get very confused as the file system structure may change
underneath them (having the same partition mounted rw twice is a sure
fire way to cause irreparable damage)!  If you want read-write
sharing, export the directory to other domains via NFS from domain0.


Troubleshooting Problems
========================

If you have problems booting Xen, there are a number of boot parameters 
that may be able to help diagnose problems:

 ignorebiostables Disable parsing of BIOS-supplied tables. This may
                  help with some chipsets that aren't fully supported
                  by Xen. If you specify this option then ACPI tables are
                  also ignored, and SMP support is disabled.

 noreboot         Don't reboot the machine automatically on errors.
                  This is useful to catch debug output if you aren't
                  catching console messages via the serial line.

 nosmp		  Disable SMP support.
                  This option is implied by 'ignorebiostables'.

 noacpi   	  Disable ACPI tables, which confuse Xen on some chipsets.
                  This option is implied by 'ignorebiostables'.

 watchdog	  Enable NMI watchdog which can report certain failures.

 noht		  Disable Hyperthreading.

 ifname=ethXX	  Select which Ethernet interface to use.

 ifname=dummy	  Don't use any network interface.

 ser_baud=xxx	  Enable serial I/O and set the baud rate.

 dom0_mem=xxx 	  Set the initial amount of memory for domain0.
		  

It's probably a good idea to join the Xen developer's mailing list on
Sourceforge:    http://lists.sourceforge.net/lists/listinfo/xen-devel   


About The Xen Demo CD
=====================

The purpose of the Demo CD is to distribute a snapshot of Xen's
source, and simultaneously provide a convenient means for enabling
people to get experience playing with Xen without needing to install
it on their hard drive. If you decide to install Xen/XenoLinux you can
do so simply by following the installation instructions below -- which
essentially involves copying the contents of the CD on to a suitably
formated disk partition, and then installing or updating the Grub
bootloader.

This is a bootable CD that loads Xen, and then a Linux 2.4.22 OS image
ported to run on Xen. The CD contains a copy of a file system based on
the RedHat 9 distribution that is able to run directly off the CD
("live ISO"), using a "tmpfs" RAM-based file system for root (/etc
/var etc). Changes you make to the tmpfs will obviously not be
persistent across reboots!

Because of the use of a RAM-based file system for root, you'll need
plenty of memory to run this CD -- something like 96MB per VM. This is
not a restriction of Xen : once you've installed Xen, XenoLinux and
the file system images on your hard drive you'll find you can boot VMs
in just a few MBs.

The CD contains a snapshot of the Xen and XenoLinux code base that we
believe to be pretty stable, but lacks some of the features that are
currently still work in progress e.g. OS suspend/resume to disk, and
various memory management enhancements to provide fast inter-OS
communication and sharing of memory pages between OSs. We'll release
newer snapshots as required, making use of a BitKeeper repository
hosted on http://xen.bkbits.net (follow instructions from the project
home page).  We're obviously grateful to receive any bug fixes or
other code you can contribute. We suggest you join the
xen-devel@lists.sourceforge.net mailing list.


Installing from the CD
======================

If you're installing Xen/XenoLinux onto an existing linux file system
distribution, just copy the Xen VMM (/boot/image.gz) and XenoLinux
kernels (/boot/xenolinux.gz), then modify the Grub config
(/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system.
It should work on pretty much any distribution.

Xen is a "multiboot" standard boot image. Despite being a 'standard',
few boot loaders actually support it. The only two we know of are
Grub, and our modified version of linux kexec (for booting off a
XenoBoot CD -- PlanetLab have adopted the same boot CD approach).

If you need to install grub on your system, you can do so either by
building the Grub source tree
/usr/local/src/grub-0.93-iso9660-splashimage or by copying over all
the files in /boot/grub and then running /sbin/grub and following the
usual grub documentation. You'll then need to edit the Grub
config file.

A typical Grub menu option might look like:

title Xen / XenoLinux 2.4.22
        kernel /boot/image.gz dom0_mem=131072 ser_baud=115200 noht
        module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0

The first line specifies which Xen image to use, and what command line
arguments to pass to Xen. In this case we set the maximum amount of
memory to allocate to domain0, and enable serial I/O at 9600 baud.
We could also disable smp support (nosmp) or disable hyper-threading
support (noht). If you have multiple network interface you can use
ifname=ethXX to select which one to use. If your network card is
unsupported, use ifname=dummy

The second line specifies which xenolinux image to use, and the
standard linux command line arguments to pass to the kernel. In this
case, we're configuring the root partition and stating that it should
be mounted read-only (normal practice). 

If we were booting with an initial ram disk (initrd), then this would
require a second "module" line.


Installing the file systems from the CD
=======================================

If you haven't an existing Linux installation onto which you can just
drop down the Xen and XenoLinux images, then the file systems on the
CD provide a quick way of doing an install.

Choose one or two partitions, depending on whether you want a separate
/usr or not. Make file systems on it/them e.g.: 
  mkfs -t ext3 /dev/hda3
  [or mkfs -t ext2 /dev/hda3 && tune2fs -j /dev/hda3 if using an old
version of mkfs]

Next, mount the file system(s) e.g.:
  mkdir /mnt/root && mount /dev/hda3 /mnt/root
  [mkdir /mnt/usr && mount /dev/hda4 /mnt/usr]
  
To install the root file system, simply untar /usr/XenDemoCD/root.tar.gz:
  cd /mnt/root && tar -zxpf /usr/XenDemoCD/root.tar.gz

You'll need to edit /mnt/root/etc/fstab to reflect your file system
configuration. Changing the password file (etc/shadow) is probably a
good idea too.

To install the usr file system, copy the file system from CD on /usr,
though leaving out the "XenDemoCD" and "boot" directories:
  cd /usr && cp -a X11R6 etc java libexec root src bin dict kerberos local sbin tmp doc include lib man share /mnt/usr

If you intend to boot off these file systems (i.e. use them for
domain 0), then you probably want to copy the /usr/boot directory on
the cd over the top of the current symlink to /boot on your root
filesystem (after deleting the current symlink) i.e.:
  cd /mnt/root ; rm boot ; cp -a /usr/boot .

The XenDemoCD directory is only useful if you want to build your own
version of the XenDemoCD (see below).


Debugging
=========

Xen has a set of debugging features that can be useful to try and
figure out what's going on. Hit 'h' on the serial line (if you
specified a baud rate on the Xen command line) or ScrollLock-h on the
keyboard to get a list of supported commands.

If you have a crash you'll likely get a crash dump containing an EIP
(PC) which, along with an 'objdump -d image', can be useful in
figuring out what's happened.  Debug a XenoLinux image just as you
would any other Linux kernel.

We supply a handy debug terminal program which you can find in
/usr/local/src/xen-1.0/xeno-1.0.bk/tools/misc/miniterm/
This should be built and executed on another machine that is connected
via a null modem cable. Documentation is included.
Alternatively, telnet can be used in 'char mode' if the Xen machine is
connected to a serial-port server.


Installing Xen / XenoLinux on a RedHat distribution
===================================================

When using Xen / Xenolinux on a standard Linux distribution there are
a couple of things to watch out for:

The first Linux VM that is started when Xen boots start (Domain 0) is
given direct access to the graphics card, so it may use it as a
console. Other domains don't have ttyN consoles, so attempts to run a
'mingetty' against them will fail, generating periodic warning
messages from 'init' about services respawning too fast. They should
work for domain0 just fine.

In future, we may make the current 'xencons' accept input as well as
output, so that a getty can be run against it. In the meantime, other
domains don't have a console suitable for logging in on, so you'll
have to run sshd and ssh in to them.

To prevent the warning messages you'll need to remove them from
/etc/inittab for domains>0.  Due to a bug in the RH9 /etc/rc.sysinit
script #'ing the lines out of /etc/inittab won't work as it ignores
the '#' and tries to access them anyway.

Also, because domains>0 don't have any privileged access at all,
certain commands in the default boot sequence will fail e.g. attempts
to update the hwclock, change the console font, update the keytable
map, start apmd (power management), or gpm (mouse cursor).  Either
ignore the errors, or remove them from the startup scripts.  Deleting
the following links are a good start: S24pcmcia S09isdn S17keytable
S26apmd S85gpm

If you want to use a single root file system that works cleanly for
domain0 and domains>0, one trick is to use different 'init' run
levels. For example, on the Xen Demo CD we use run level 3 for domain
0, and run level 4 for domains>0. This enables different startup
scripts to be run in depending on the run level number passed on the
kernel command line. The xenctl.xml config file on the CD passes '4'
on the kernel command line to domains that it starts.

Xenolinux kernels can be built to use runtime loadable modules just
like normal linux kernels. Modules should be installed under
/lib/modules in the normal way.

If there's some kernel feature that hasn't been built into our default
kernel, there's a pretty good change that if its a non-hardware
related option you'll just be able to enable it and rebuild.  If its
not on the xconfig menu, hack the arch/xeno/config.in to put the menu
back in.

If you're going to use the link local 169.254.1.x addresses to
communicate between VMs, there are a couple of other issues to watch
out for. RH9 appears to have a bug where by default it configures the
loopback interface with a 169.254 address, which stops it working
properly on eth0 for communicating with other domains.

This utterly daft RH9 behaviour can be stopped by appending
"NOZEROCONF=yes" to /etc/sysconfig/networking-scripts/ifcfg-lo

If you're going to use NFS root files systems mounted either from an
external server or from domain0 there are a couple of other gotchas.
The default /etc/sysconfig/iptables rules block NFS, so part way
through the boot sequence things will suddenly go dead.

If you're planning on having a separate NFS /usr partition, the RH9
boot scripts don't make life easy, as they attempt to mount NFS file
systems way to late in the boot process. The easiest way I found to do
this was to have a '/linuxrc' script run ahead of /sbin/init that
mounts /usr:
 #!/bin/bash
 /sbin/ipconfig lo 127.0.0.1
 /sbin/portmap
 /bin/mount /usr
 exec /sbin/init "$@" <>/dev/console 2>&1

The one slight complication with the above is that /sbib/portmap is
dynamically linked against /usr/lib/libwrap.so.0 Since this is in
/usr, it won't work. I solved this by copying the file (and link)
below the /usr mount point, and just let the file be 'covered' when
the mount happens.

In some installations, where a shared read-only /usr is being used, it
may be desirable to move other large directories over into the
read-only /usr. For example, on the XenDemoCD we replace /bin /lib and
/sbin with links into /usr/root/bin /usr/root/lib and /usr/root/sbin
respectively. This creates other problems for running the /linuxrc
script, requiring bash, portmap, mount, ifconfig, and a handful of
other shared libraries to be copied below the mount point. I guess I
should have written a little statically linked C program...



Description of how the XenDemoCD boots
======================================

1. Grub is used to load Xen, a XenoLinux kernel, and an initrd (initial
ram disk). [The source of the version of Grub used is in /usr/local/src]

2. the init=/linuxrc command line causes linux to execute /linuxrc in
the initrd. 

3. the /linuxrc file attempts to mount the CD by trying the likely
locations : /dev/hd[abcd].  

4. it then creates a 'tmpfs' file system and untars the
'XenDemoCD/root.tar.gz' file into the tmpfs. This contains hopefully
all the files that need to be mutable (this would be so much easier
if Linux supported 'stacked' or union file systems...)

5. Next, /linuxrc uses the pivot_root call to change the root file
system to the tmpfs, with the CD mounted as /usr.

6. It then invokes /sbin/init in the tmpfs and the boot proceeds
normally.


Building your own version of the XenDemoCD
==========================================

The 'live ISO' version of RedHat is based heavily on Peter Anvin's
SuperRescue CD version 2.1.2 and J. McDaniel's Plan-B:

 http://www.kernel.org/pub/dist/superrescue/v2/
 http://projectplanb.org/

Since Xen uses a "multiboot" image format, it was necessary to change
the bootloader from isolinux to Grub0.93 with Leonid Lisovskiy's
<lly@pisem.net> grub.0.93-iso9660.patch

The Xen Demo CD contains all of the build scripts that were used to
create it, so it is possible to 'unpack' the current iso, modifiy it,
then build a new iso. The procedure for doing so is as follows:

First, mount either the CD, or the iso image of the CD:
 
  mount /dev/cdrom /mnt/cdrom 
or:
  mount -o loop xendemo-1.0.iso  /mnt/cdrom

cd to the directory you want to 'unpack' the iso into then run the
unpack script:

  cd /local/xendemocd
  /mnt/cdrom/XenDemoCD/unpack-iso.sh

The result is a 'build' directory containing the file system tree
under the 'root' directory. e.g. /local/xendemocd/build/root

To add or remove rpms, its possible to use 'rpm' with the --root
option to set the path. For more complex changes, it easiest to boot a
machine using using the tree via NFS root. Before doing this, you'll
need to edit fstab to comment out the seperate mount of /usr.

One thing to watch out for: as part of the CD build process, the
contents of the 'rootpatch' tree gets copied over the existing 'root'
tree replacing various files. The intention of the rootpatch tree is
to contain the files that have been modified from the original RH
distribution (e.g. various /etc files). This was done to make it
easier to upgrade to newer RH versions in the future. The downside of
this is that if you edit an existing file in the root tree you should
check that you don't also need to propagate the change to the
rootpatch tree to avoid it being overwritten.

Once you've made the changes and want to build a new iso, here's the
procedure:

cd /local/xendemocd/build  
echo '<put_your_name_here>' > Builder
./make.sh put_your_version_id_here >../buildlog 2>&1            

This process can take 30 mins even on a fast machine, but you should
eventually end up with an iso image in the build directory.

Notes:

  root      - the root of the file system heirarchy as presented to the
              running system

  rootpatch - contains files that have been modified from the standard
              RH, and copied over the root tree as part of the build
              procedure.

  irtree    - the file system tree that will go into the initrd (initial
              ram disk)

  work      - a working directory used in the build process

  usr       - this should really be in 'work' as its created as part of the
              build process. It contains the 'immutable' files that will
              be served from the CD rather than the tmpfs containing the
              contents of root.tar.gz. Some files that are normally in /etc 
              or /var that are large and actually unlikely to need changing 
              have been moved into /usr/root and replaced with links.


Ian Pratt
9 Sep 2003