xen/tools/xm-test/README

----------------------------------------------------------------------

Status of xm-test

The xm-test environment was written in 2006.  It was mostly not
updated or adapted to new / changed functionality since then.
("it's slowly rotting", Keir Fraser, xen-devel mailing list,
June 2009)
During the last weeks there was the attempt to fix most of the
problems - but there are still some major issues:
o The reporting infrastructure is 'long gone' (Dan Smith, Mail from
  Aug 2009).  Therefore there is no way to send out reports. When
  running the runtests.sh you should therefore specify the '-d'
  option.
o The initrd.img (which is downloaded) is quite old and needs an
  update. Especially functionality for migration it is missing.
  [It should be discussed using an existing distribution instead of
  compiling a micro distribution and put this into the initrd.)

Andreas Florath <xen@flonatel.org>
August 2009

----------------------------------------------------------------------


xm-test README

Copyright (C) International Business Machines Corp., 2005
Author(s): Dan Smith <danms@us.ibm.com> 
  	   Woody Marvel <marvel@us.ibm.com>

Overview
========

This suite provides a framework for testing the Xen userspace tools.
The directory structure is:

  ./xm-test
   |
   +-/lib: Python support libraries
   |
   +-/ramdisk: Staging area for building the test ramdisk
   |
   +-/tests
   | |
   | +-/create: Tests for the 'xm create' command
   | +-/destroy: Tests for the 'xm destroy' command
   |  . . .
   |
   +-/utils: Utility scripts for ramdisk building

Reports are posted here:

   http://xmtest.dague.org


Building
========

Before the test suite can be used, the ramdisk must be built from
source.  All source needed for this process is automatically
downloaded, extracted, and compiled.  Due to the need to create
special files, this process must be done as root:

   # ./autogen
   # ./configure
   # make

NB: If you have the initrd.img from another installation of xm-test,
you can copy it into the ramdisk directory to eliminate the need to
rebuild it.  If you do this, there is no need to run 'make' again.
Simply copy the initrd-X.Y-ARCH.img file into ramdisk/ and then run:

   # make existing

Or, you can run:
   # INITRD="http://url.of.initrd.repo/" make existing

You do not need to include the name of the image itself in the url, 
however, an initrd with the right name (initrd.X.Y-ARCH.img) and version 
number must exist at that location.  The script will determine which 
version of the initrd it needs and try to download the right file from 
that location.

This will set up the link so that xm-test will use the existing
ramdisk.  Next, just run "runtest.sh" normally.  Note that in general,
you should not attempt to use a ramdisk from a previous minor version
of xm-test (i.e., don't use a ramdisk from 0.4.0 with 0.5.0.  0.5.0
should work for 0.5.3 though)


BUILDING with HVM Support
=========================

If you'd like to build and run this with hardware virtual machine assist
(HVM) support to test fully virtualized disk images on VMX/SVM hardware, 
please add the --enable-hvm-support option to configure:

  # ./autogen
  # ./configure --enable-hvm-support
  # make

The ramdisk/bin/create_disk_image script, which builds the full virt
disk.img, requires Lilo 22.7+ to be installed on the system. Lilo is 
used to install the bootloader on the disk.img.

If HVM support is enabled, the ramdisk/bin/create_disk_image script
will be run to create a full virt disk.img in the ramdisk directory. The
script, by default, will look in /boot for the first non-Xen kernel it
runs across. If you'd like to set xm-test to use a specific kernel,
rather than the first one it finds in /boot, you can configure it in
with the "--with-hvm-kernel=KERNEL" option:
 
# ./autogen
# ./configure --enable-hvm-support --with-hvm-kernel=KERNEL
# make

Otherwise, you can always rerun the create script using the -k option
to use a specific kernel. 

The disk.img created for HVM testing must contain a pcnet32 driver for
network tests. The ramdisk/bin/create_disk_image script will, by default,
look in the /lib/modules directory associated with the kernel being
used. If you'd like to specify a different location for the driver or
want to tell the script that the driver is built into the kernel, please
use the "--with-driver-dir=DRVDIR" configure option. If built into
the kernel, please use the key word "builtin" with the option:
 
# ./autogen
# ./configure --enable-hvm-support --with-driver-dir=builtin
- or -
# ./configure --enable-hvm-support --with-driver-dir=/driver/directory
# make
 
Xm-test will look for disk.img in the ramdisk directory when run by
default.


Running
=======

To run the full test suite, do the following as root:

   # ./runtest.sh <logfile>

This will run all tests, as well as generate and submit a report at
the end. All output files will begin with "<logfile>."
If you wish to prevent submission of a report, add "-d" to the 
command line like this:

   # ./runtest.sh -d <logfile>

It may be useful to run tests without submission as above, and then
submit the report at a later time.  To do so, run runtest.sh with the
-s flag and the name of the previously-generated report:

   # ./runtest.sh -s <logfile>

Group test sets are supported in xm-test. This is form of layering of 
tests groups/cases/tests. In the framework directory "grouptest",
files exist for group processing. The user can add groups, casenames
and test lists as required. Default group run is "grouptest/default".

   # ./runtest.sh -g <groupname> <logfile>

* NOTE: There is a quick set of tests in group mode, that was added to
run certain casenames and tests, and there is a "medium" group, which is a
medium-length run (around 20 minutes).  Neither is a substitute for the full
xm-test test suite.
   # ./runtest.sh -g quick <logfile>
   # ./runtest.sh -g medium <logfile>



It may be desirable to run a specific test group.  This can be
accomplished by doing the following:

   # cd tests/create
   # TEST_VERBOSE=1 make check

When developing or debugging a specific feature, a single test can be
run to avoid having to run even a whole test group:

   # cd tests/create
   # TEST_VERBOSE=1 make check TESTS=01_create_basic_pos.test

The runtest.sh script will create several files, including a .report
file, which is the cleaned up, email-friendly report of failures.
Additionally, the script will submit your results to the development
team for trend analysis.  This helps us determine the level of success
people "out there" are having with different versions of Xen.

Note: you should generally run xm-test with a minimum of memory
allocated to Dom0.  More memory available for allocation to DomUs
means a more rigorous test.

BIG FAT WARNING: The test framework assumes it is running on a
dedicated machine.  As such, the library automatically destroys any
running DomUs on the system to provide each test with a "clean slate".


Testing the XML-RPC and Xen-API interfaces of xend
==================================================

The xm-test suite can be used to test xm's interface with xend using
either XML-RPC or the Xen-API. In order to use either one of these modes,
xm needs to be configured using its configuration file
'/etc/xen/xm-config.xml'.
Note: The current default configuration after a fresh install of the xen
sources currently is to use the XML-RPC interface for communication with xend.

Example content for the xm-config.xml for using the Xen-API looks as
follows:

<xm>
  <server type='Xen-API'
          uri='http://localhost:9363/'
          username='me'
          password='mypassword' />
</xm>

This configuration makes xm talk to xend using port 9363. For this to
work, also xend needs to be configured to listen to port 9363. Therefore
The following line must be in /etc/xen/xend-config.sxp.

(xen-api-server (( 127.0.0.1:9363 none )))

To communicate via the legacy XML-RPC interface, the file
'/etc/xen/xm-config.xml' may simply have the following content or
may be complete remove from the /etc/xen directory.

<xm>
</xm>

A few tests have been written for the xm-test suite that test the
Xen-API interface directly without relying on 'xm'. These tests can be
found in the grouptest 'xapi' and for them to work properly, xm must have
been configured to use the Xen-API following the instructions above. To
run these test, the following command line can be invoked:

   # ./runtest.sh -g xapi <logfile>



Extending
=========

Additional tests may be added in existing groups to test additional
cases for a given xm subcommand.  Test programs should be named
according to the following scheme:
	
  XY_group_name_{pos,neg}.py

Where:
  XY        is the next number in line
  group     is the name of the subcommand being tested
  name      is the short name of the test
  {pos,neg} denotes whether this is a positive or negative test case

New subcommand groups should be added as directories named after the
subcommand itself.  The "Makefile.am.template" should be copied into
the new group directory as "Makefile.am".

See the Writing_Tests_HOWTO file for more detailed information on
adding tests to the suite.


Developer Notes
===============

Our library provides a DomU console abstraction for automated
execution of commands.  Please note that this is relatively fragile,
and is intended for use only with the ramdisk built by the framework.
Because the console experiences some occasional corruption, this
method is not completely perfect at the moment, although the authors
use it with relatively few problems.


Known Issues
============

If you create a domain with a small amount of memory, under 32MBs, you
may run into out of memory situations for the domain. There's no way
to know the amount of memory needed by the kernel and modules used. Xm-test
uses 64MBs as default and that should work. If there are out of memory
issues, the default can be changed. Edit xm-test/lib/XmTestLib/XenDomain.py
and change ParavirtDefaults and HVMDefaults "memory".

There are two tests that work with small memory, 15_create_smallmem_pos.py 
and 16_create_smallmem_neg.py. The first makes sure the default 32 MBs 
limit works. The second checks a low memory fail situation. These tests
are located in the xm-test/tests/create directory and can be easily edited
to change the MEM value they should test. If the 32MBs test fails, the
failure should be reported to the Xen xen-devel mailing list. The Xen
tools use 32MBs as a lower acceptable limit for domain creation. The Xen
mailing lists are located here:

http://lists.xensource.com/


Reporting Bugs
==============

If you find a bug in the test framework, report it to:

   Dan Smith <danms@us.ibm.com>

If you find a bug in a specific test case, contact the author of the
test case first.