mirror of
https://github.com/xen-project/xen.git
synced 2026-10-29 02:30:03 +01:00
Andrew Cooper
df7f3ce32b
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Reviewed-by: Jan Beulich <jbeulich@suse.com> Release-acked-by: Wei Liu <wei.liu2@citrix.com>
217 lines
8.2 KiB
Plaintext
217 lines
8.2 KiB
Plaintext
% Feature Levelling
|
|
% Revision 1
|
|
|
|
\clearpage
|
|
|
|
# Basics
|
|
|
|
---------------- ----------------------------------------------------
|
|
Status: **Supported**
|
|
|
|
Architecture: x86
|
|
|
|
Component: Hypervisor, toolstack, guest
|
|
---------------- ----------------------------------------------------
|
|
|
|
|
|
# Overview
|
|
|
|
On native hardware, a kernel will boot, detect features, typically optimise
|
|
certain codepaths based on the available features, and expect the features to
|
|
remain available until it shuts down.
|
|
|
|
The same expectation exists for virtual machines, and it is up to the
|
|
hypervisor/toolstack to fulfill this expectation for the lifetime of the
|
|
virtual machine, including across migrate/suspend/resume.
|
|
|
|
|
|
# User details
|
|
|
|
Many factors affect the featureset which a VM may use:
|
|
|
|
* The CPU itself
|
|
* The BIOS/firmware/microcode version and settings
|
|
* The hypervisor version and command line settings
|
|
* Further restrictions the toolstack chooses to apply
|
|
|
|
A firmware or software upgrade might reduce the available set of features
|
|
(e.g. Intel disabling TSX in a microcode update for certain Haswell/Broadwell
|
|
processors), as may editing the settings.
|
|
|
|
It is unsafe to make any assumption about features remaining consistent across
|
|
a host reboot. Xen recalculates all information from scratch each boot, and
|
|
provides the information for the toolstack to consume.
|
|
|
|
`xl` currently has no facilities to help the user collect appropriate feature
|
|
information from relevant hosts and compute appropriate feature specifications
|
|
for use in host or domain configurations. (`xl` being a single-host
|
|
toolstack, it would in any case need external support for accessing remote
|
|
hosts eg via ssh, in the form of automation software like GNU parallel or
|
|
ansible.)
|
|
|
|
# Technical details
|
|
|
|
The `CPUID` instruction is used by software to query for features. In the
|
|
virtualisation usecase, guest software should query Xen rather than hardware
|
|
directly. However, `CPUID` is an unprivileged instruction which doesn't
|
|
fault, complicating the task of hiding hardware features from guests.
|
|
|
|
Important files:
|
|
|
|
* Hypervisor
|
|
* `xen/arch/x86/cpu/*.c`
|
|
* `xen/arch/x86/cpuid.c`
|
|
* `xen/include/asm-x86/cpuid-autogen.h`
|
|
* `xen/include/public/arch-x86/cpufeatureset.h`
|
|
* `xen/tools/gen-cpuid.py`
|
|
* `libxc`
|
|
* `tools/libxc/xc_cpuid_x86.c`
|
|
|
|
## Ability to control CPUID
|
|
|
|
### HVM
|
|
|
|
HVM guests (using `Intel VT-x` or `AMD SVM`) will unconditionally exit to Xen
|
|
on all `CPUID` instructions, allowing Xen full control over all information.
|
|
|
|
### PV
|
|
|
|
The `CPUID` instruction is unprivileged, so executing it in a PV guest will
|
|
not trap, leaving Xen no direct ability to control the information returned.
|
|
|
|
### Xen Forced Emulation Prefix
|
|
|
|
Xen-aware PV software can make use of the 'Forced Emulation Prefix'
|
|
|
|
> `ud2a; .ascii 'xen'; cpuid`
|
|
|
|
which Xen recognises as a deliberate attempt to get the fully-controlled
|
|
`CPUID` information rather than the hardware-reported information. This only
|
|
works with cooperative software.
|
|
|
|
### Masking and Override MSRs
|
|
|
|
AMD CPUs from the `K8` onwards support _Feature Override_ MSRs, which allow
|
|
direct control of the values returned for certain `CPUID` leaves. These MSRs
|
|
allow any result to be returned, including the ability to advertise features
|
|
which are not actually supported.
|
|
|
|
Intel CPUs between `Nehalem` and `SandyBridge` have differing numbers of
|
|
_Feature Mask_ MSRs, which are a simple AND-mask applied to all `CPUID`
|
|
instructions requesting specific feature bitmap sets. The exact MSRs, and
|
|
which feature bitmap sets they affect are hardware specific. These MSRs allow
|
|
features to be hidden by clearing the appropriate bit in the mask, but does
|
|
not allow unsupported features to be advertised.
|
|
|
|
### CPUID Faulting
|
|
|
|
Intel CPUs from `IvyBridge` onwards have _CPUID Faulting_, which allows Xen to
|
|
cause `CPUID` instruction executed in PV guests to fault. This allows Xen
|
|
full control over all information, exactly like HVM guests.
|
|
|
|
## Compile time
|
|
|
|
As some features depend on other features, it is important that, when
|
|
disabling a certain feature, we disable all features which depend on it. This
|
|
allows runtime logic to be simplified, by being able to rely on testing only
|
|
the single appropriate feature, rather than the entire feature dependency
|
|
chain.
|
|
|
|
To speed up runtime calculation of feature dependencies, the dependency chain
|
|
is calculated and flattened by `xen/tools/gen-cpuid.py` to create
|
|
`xen/include/asm-x86/cpuid-autogen.h` from
|
|
`xen/include/public/arch-x86/cpufeatureset.h`, allowing the runtime code to
|
|
disable all dependent features of a specific disabled feature in constant
|
|
time.
|
|
|
|
## Host boot
|
|
|
|
As Xen boots, it will enumerate the features it can see. This is stored as
|
|
the _raw\_featureset_.
|
|
|
|
Errata checks and command line arguments are then taken into account to reduce
|
|
the _raw\_featureset_ into the _host\_featureset_, which is the set of
|
|
features Xen uses. On hardware with masking/override MSRs, the default MSR
|
|
values are picked from the _host\_featureset_.
|
|
|
|
The _host\_featureset_ is then used to calculate the _pv\_featureset_ and
|
|
_hvm\_featureset_, which are the maximum featuresets Xen is willing to offer
|
|
to PV and HVM guests respectively.
|
|
|
|
In addition, Xen will calculate how much control it has over non-cooperative
|
|
PV `CPUID` instructions, storing this information as _levelling\_caps_.
|
|
|
|
## Domain creation
|
|
|
|
The toolstack can query each of the calculated featureset via
|
|
`XEN_SYSCTL_get_cpu_featureset`, and query for the levelling caps via
|
|
`XEN_SYSCTL_get_cpu_levelling_caps`.
|
|
|
|
These data should be used by the toolstack when choosing the eventual
|
|
featureset to offer to the guest.
|
|
|
|
Once a featureset has been chosen, it is set (implicitly or explicitly) via
|
|
`XEN_DOMCTL_set_cpuid`. Xen will clamp the toolstacks choice to the
|
|
appropriate PV or HVM featureset. On hardware with masking/override MSRs, the
|
|
guest cpuid policy is reflected in the MSRs, which are context switched with
|
|
other vcpu state.
|
|
|
|
# Limitations
|
|
|
|
A guest which ignores the provided feature information and manually probes for
|
|
features will be able to find some of them. e.g. There is no way of forcibly
|
|
preventing a guest from using 1GB superpages if the hardware supports it.
|
|
|
|
Some information simply cannot be hidden from guests. There is no way to
|
|
control certain behaviour such as the hardware MXCSR\_MASK or x87 FPU exception
|
|
behaviour.
|
|
|
|
|
|
# Testing
|
|
|
|
Feature levelling is a very wide area, and used all over the hypervisor.
|
|
Please ask on xen-devel for help identifying more specific tests which could
|
|
be of use.
|
|
|
|
|
|
# Known issues / Areas for improvement
|
|
|
|
The feature querying and levelling functions should exposed in a
|
|
convenient-to-use way by `xl`.
|
|
|
|
Xen currently has no concept of per-{socket,core,thread} CPUID information.
|
|
As a result, details such as APIC IDs, topology and cache information do not
|
|
match real hardware, and do not match the documented expectations in the Intel
|
|
and AMD system manuals.
|
|
|
|
The CPU feature flags are the only information which the toolstack has a
|
|
sensible interface for querying and levelling. Other information in the CPUID
|
|
policy is important and should be levelled (e.g. maxphysaddr).
|
|
|
|
The CPUID policy is currently regenerated from scratch by the receiving side,
|
|
once memory and vcpu content has been restored. This means that the receiving
|
|
Xen cannot verify the memory/vcpu content against the CPUID policy, and can
|
|
end up running a guest which will subsequently crash. The CPUID policy should
|
|
be at the head of the migration stream.
|
|
|
|
MSRs are another source of features for guests. There is no general provision
|
|
for controlling the available MSRs. E.g. 64bit versions of Windows notice
|
|
changes in IA32\_MISC\_ENABLE, and suffer a BSOD 0x109 (Critical Structure
|
|
Corruption)
|
|
|
|
|
|
# References
|
|
|
|
[Intel Flexmigration](http://www.intel.co.uk/content/dam/www/public/us/en/documents/application-notes/virtualization-technology-flexmigration-application-note.pdf)
|
|
|
|
[AMD Extended Migration Technology](http://developer.amd.com/wordpress/media/2012/10/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf)
|
|
|
|
|
|
# History
|
|
|
|
------------------------------------------------------------------------
|
|
Date Revision Version Notes
|
|
---------- -------- -------- -------------------------------------------
|
|
2016-05-31 1 Xen 4.7 Document written
|
|
---------- -------- -------- -------------------------------------------
|