xen/tools/xm-test/README

xm-test README

Copyright (C) International Business Machines Corp., 2005
Author(s): Dan Smith <danms@us.ibm.com> 
  	   Woody Marvel <marvel@us.ibm.com>

Overview
========

This suite provides a framework for testing the Xen userspace tools.
The directory structure is:

  ./xm-test
   |
   +-/lib: Python support libraries
   |
   +-/ramdisk: Staging area for building the test ramdisk
   |
   +-/tests
   | |
   | +-/create: Tests for the 'xm create' command
   | +-/destroy: Tests for the 'xm destroy' command
   |  . . .
   |
   +-/utils: Utility scripts for ramdisk building

Reports are posted here:

   http://xmtest.dague.org


Building
========

Before the test suite can be used, the ramdisk must be built from
source.  All source needed for this process is automatically
downloaded, extracted, and compiled.  Due to the need to create
special files, this process must be done as root:

   # ./autogen
   # ./configure
   # make

NB: If you have the initrd.img from another installation of xm-test,
you can copy it into the ramdisk directory to eliminate the need to
rebuild it.  If you do this, there is no need to run 'make' again.
Simply copy the initrd-X.Y-ARCH.img file into ramdisk/ and then run:

   # make existing

Or, you can run:
   # INITRD="http://url.of.initrd.repo/" make existing

You do not need to include the name of the image itself in the url, 
however, an initrd with the right name (initrd.X.Y-ARCH.img) and version 
number must exist at that location.  The script will determine which 
version of the initrd it needs and try to download the right file from 
that location.

This will set up the link so that xm-test will use the existing
ramdisk.  Next, just run "runtest.sh" normally.  Note that in general,
you should not attempt to use a ramdisk from a previous minor version
of xm-test (i.e., don't use a ramdisk from 0.4.0 with 0.5.0.  0.5.0
should work for 0.5.3 though)


BUILDING with HVM Support
=========================

If you'd like to build and run this with hardware virtual machine assist
(HVM) support to test fully virtualized disk images on VMX/SVM hardware, 
please add the --enable-hvm-support option to configure:

  # ./autogen
  # ./configure --enable-hvm-support
  # make

The ramdisk/bin/create_disk_image script, which builds the full virt
disk.img, requires Lilo 22.7+ to be installed on the system. Lilo is 
used to install the bootloader on the disk.img.

If HVM support is enabled, the ramdisk/bin/create_disk_image script
will be run to create a full virt disk.img in the ramdisk directory. The
script, by default, will look in /boot for the first non-Xen kernel it
runs across. If you'd like to set xm-test to use a specific kernel,
rather than the first one it finds in /boot, you can configure it in
with the "--with-hvm-kernel=KERNEL" option:
 
# ./autogen
# ./configure --enable-hvm-support --with-hvm-kernel=KERNEL
# make

Otherwise, you can always rerun the create script using the -k option
to use a specific kernel. 

The disk.img created for HVM testing must contain a pcnet32 driver for
network tests. The ramdisk/bin/create_disk_image script will, by default,
look in the /lib/modules directory associated with the kernel being
used. If you'd like to specify a different location for the driver or
want to tell the script that the driver is built into the kernel, please
use the "--with-driver-dir=DRVDIR" configure option. If built into
the kernel, please use the key word "builtin" with the option:
 
# ./autogen
# ./configure --enable-hvm-support --with-driver-dir=builtin
- or -
# ./configure --enable-hvm-support --with-driver-dir=/driver/directory
# make
 
Xm-test will look for disk.img in the ramdisk directory when run by
default.


BUILDING for ACM Security Testing
=================================

A number of tests have been added to test the access control module (ACM)
in the Xen hypervisor and the tools for supporting ACM. Those tests are
located in the security-acm directory. If ACM support is compiled into Xen
(see the user guide for how to do this) those tests can be run with the
following command from the xm-test directory

./runtest.sh [...] -g security <report>

Some of these tests will work even without support of ACM by Xen.

The xm test suite has been extended to support labeling of resources
as required by the existing tests. However, by default the test suite
is not allowed to automatically label resources since this may affect
existing labels. To enable this, the test suite must be configured with
the following parameter passed to the configure scripts (in addition to
any other desired parameters)

./configure --enable-full-labeling

To revoke the privilege at a later time run the configure scripts without
this parameter:

./configure

If a 'make' has previously been run for building the test suite, it is not
necessary to run 'make' again just for enabling or disabling the automatic
labeling of resources.


Running
=======

To run the full test suite, do the following as root:

   # ./runtest.sh <logfile>

This will run all tests, as well as generate and submit a report at
the end. All output files will begin with "<logfile>."
If you wish to prevent submission of a report, add "-d" to the 
command line like this:

   # ./runtest.sh -d <logfile>

It may be useful to run tests without submission as above, and then
submit the report at a later time.  To do so, run runtest.sh with the
-s flag and the name of the previously-generated report:

   # ./runtest.sh -s <logfile>

Group test sets are supported in xm-test. This is form of layering of 
tests groups/cases/tests. In the framework directory "grouptest",
files exist for group processing. The user can add groups, casenames
and test lists as required. Default group run is "grouptest/default".

   # ./runtest.sh -g <groupname> <logfile>

* NOTE: There is a quick set of tests in group mode, that was added to
run certain casenames and tests, and there is a "medium" group, which is a
medium-length run (around 20 minutes).  Neither is a substitute for the full
xm-test test suite.
   # ./runtest.sh -g quick <logfile>
   # ./runtest.sh -g medium <logfile>



It may be desirable to run a specific test group.  This can be
accomplished by doing the following:

   # cd tests/create
   # TEST_VERBOSE=1 make check

When developing or debugging a specific feature, a single test can be
run to avoid having to run even a whole test group:

   # cd tests/create
   # TEST_VERBOSE=1 make check TESTS=01_create_basic_pos.test

The runtest.sh script will create several files, including a .report
file, which is the cleaned up, email-friendly report of failures.
Additionally, the script will submit your results to the development
team for trend analysis.  This helps us determine the level of success
people "out there" are having with different versions of Xen.

Note: you should generally run xm-test with a minimum of memory
allocated to Dom0.  More memory available for allocation to DomUs
means a more rigorous test.

BIG FAT WARNING: The test framework assumes it is running on a
dedicated machine.  As such, the library automatically destroys any
running DomUs on the system to provide each test with a "clean slate".


Extending
=========

Additional tests may be added in existing groups to test additional
cases for a given xm subcommand.  Test programs should be named
according to the following scheme:
	
  XY_group_name_{pos,neg}.py

Where:
  XY        is the next number in line
  group     is the name of the subcommand being tested
  name      is the short name of the test
  {pos,neg} denotes whether this is a positive or negative test case

New subcommand groups should be added as directories named after the
subcommand itself.  The "Makefile.am.template" should be copied into
the new group directory as "Makefile.am".

See the Writing_Tests_HOWTO file for more detailed information on
adding tests to the suite.


Developer Notes
===============

Our library provides a DomU console abstraction for automated
execution of commands.  Please note that this is relatively fragile,
and is intended for use only with the ramdisk built by the framework.
Because the console experiences some occasional corruption, this
method is not completely perfect at the moment, although the authors
use it with relatively few problems.


Known Issues
============

If you create a domain with a small amount of memory, under 32MBs, you
may run into out of memory situations for the domain. There's no way
to know the amount of memory needed by the kernel and modules used. Xm-test
uses 64MBs as default and that should work. If there are out of memory
issues, the default can be changed. Edit xm-test/lib/XmTestLib/XenDomain.py
and change ParavirtDefaults and HVMDefaults "memory".

There are two tests that work with small memory, 15_create_smallmem_pos.py 
and 16_create_smallmem_neg.py. The first makes sure the default 32 MBs 
limit works. The second checks a low memory fail situation. These tests
are located in the xm-test/tests/create directory and can be easily edited
to change the MEM value they should test. If the 32MBs test fails, the
failure should be reported to the Xen xen-devel mailing list. The Xen
tools use 32MBs as a lower acceptable limit for domain creation. The Xen
mailing lists are located here:

http://lists.xensource.com/


Reporting Bugs
==============

If you find a bug in the test framework, report it to:

   Dan Smith <danms@us.ibm.com>

If you find a bug in a specific test case, contact the author of the
test case first.