xen-project/xen
1
0 Fork 1
mirror of https://github.com/xen-project/xen.git synced 2026-10-28 10:11:27 +01:00
Code Issues Activity
Files
449caecbdfa3f423eae46a53822b7e403607cfee
xen/docs/features/intel_psr_mba.pandoc
T
Yi Sun 0062b96961 docs: create Memory Bandwidth Allocation (MBA) feature document 
This patch creates MBA feature document in doc/features/. It describes
key points to implement MBA which is described in details in Intel SDM
"Introduction to Memory Bandwidth Allocation".

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
2017-12-15 11:02:23 +01:00

298 lines
11 KiB
Plaintext
Raw Blame History

% Intel Memory Bandwidth Allocation (MBA) Feature
% Revision 1.8
\clearpage
# Basics
---------------- ----------------------------------------------------
Status: **Tech Preview**
Architecture(s): Intel x86
Component(s): Hypervisor, toolstack
Hardware: MBA is supported on Skylake Server and beyond
---------------- ----------------------------------------------------
# Terminology
* CAT Cache Allocation Technology
* CBM Capacity BitMasks
* CDP Code and Data Prioritization
* COS/CLOS Class of Service
* HW Hardware
* MBA Memory Bandwidth Allocation
* MSRs Machine Specific Registers
* PSR Intel Platform Shared Resource
* THRTL Throttle value or delay value
# Overview
The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate
control over memory bandwidth available per-core. This feature provides OS/
hypervisor the ability to slow misbehaving apps/domains by using a credit-based
throttling mechanism.
# User details
* Feature Enabling:
Add "psr=mba" to boot line parameter to enable MBA feature.
* xl interfaces:
1. `psr-mba-show [domain-id|domain-name]`:
Show memory bandwidth throttling for domain. Under different modes, it
shows different type of data.
There are two modes:
Linear mode: the input precision is defined as 100-(MBA_MAX). For instance,
if the MBA_MAX value is 90, the input precision is 10%. Values not an even
multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10%
delay applied) by HW automatically. The response of throttling value is
linear.
Non-linear mode: input delay values are powers-of-two from zero to the
MBA_MAX value from CPUID. In this case any values not a power of two will
be rounded down the next nearest power of two by HW automatically. The
response of throttling value is non-linear.
For linear mode, it shows the decimal value. For non-linear mode, it shows
hexadecimal value.
2. `psr-mba-set [OPTIONS] <domain-id|domain-name> <throttling>`:
Set memory bandwidth throttling for domain.
Options:
'-s': Specify the socket to process, otherwise all sockets are processed.
Throttling value set in register implies the approximate amount of delaying
the traffic between core and memory. Higher throttling value result in
lower bandwidth. The max throttling value (MBA_MAX) supported can be
obtained through CPUID inside hypervisor. Users can fetch the MBA_MAX value
using the `psr-hwinfo` xl command.
# Technical details
MBA is a member of Intel PSR features, it shares the base PSR infrastructure
in Xen.
## Hardware perspective
MBA defines a range of MSRs to support specifying a delay value (Thrtl) per
COS, with details below.
```
+----------------------------+----------------+
| MSR (per socket) | Address |
+----------------------------+----------------+
| IA32_L2_QOS_Ext_BW_Thrtl_0 | 0xD50 |
+----------------------------+----------------+
| ... | ... |
+----------------------------+----------------+
| IA32_L2_QOS_Ext_BW_Thrtl_n | 0xD50+n |
+----------------------------+----------------+
```
When context switch happens, the COS ID of domain is written to per-hyper-
thread MSR `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation
according to the throttling value stored in the Thrtl MSR register.
## The relationship between MBA and CAT/CDP
Generally speaking, MBA is completely independent of CAT/CDP, and any
combination may be applied at any time, e.g. enabling MBA with CAT
disabled.
But it needs to be noticed that MBA shares COS infrastructure with CAT,
although MBA is enumerated by different CPUID leaf from CAT (which
indicates that the max COS of MBA may be different from CAT). In some
cases, a domain is permitted to have a COS that is beyond one (or more)
of PSR features but within the others. For instance, let's assume the max
COS of MBA is 8 but the max COS of L3 CAT is 16, when a domain is assigned
9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for MBA,
the HW works as default value is set since COS 9 is beyond the max COS (8)
of MBA.
## Design Overview
* Core COS/Thrtl association
When enforcing Memory Bandwidth Allocation, all cores of domains have
the same default Thrtl MSR (COS0) which stores the same Thrtl (0). The
default Thrtl MSR is used only in hypervisor and is transparent to tool stack
and user.
System administrators can change PSR allocation policy at runtime by
using the tool stack. Since MBA shares COS ID with CAT/CDP, a COS ID
corresponds to a 2-tuple, like [CBM, Thrtl] with only-CAT enabled, when CDP
is enabled, the COS ID corresponds to a 3-tuple, like [Code_CBM, Data_CBM,
Thrtl]. If neither CAT nor CDP is enabled, things are easier, since one COS
ID corresponds to one Thrtl.
* VCPU schedule
This part reuses CAT COS infrastructure.
* Multi-sockets
Different sockets may have different MBA capabilities (like max COS)
although it is consistent on the same socket. So the capability
of per-socket MBA is specified.
This part reuses CAT COS infrastructure.
## Implementation Description
* Hypervisor interfaces:
1. Boot line param: "psr=mba" to enable the feature.
2. SYSCTL:
- XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
3. DOMCTL:
- XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get throttling for a domain.
- XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set throttling for a domain.
* xl interfaces:
1. psr-mba-show [domain-id]
Show system/domain runtime MBA throttling value. For linear mode,
it shows the decimal value. For non-linear mode, it shows hexadecimal
value.
=> XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
2. psr-mba-set [OPTIONS] <domain-id> <throttling>
Set bandwidth throttling for a domain.
=> XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
3. psr-hwinfo
Show PSR HW information, including L3 CAT/CDP/L2 CAT/MBA.
=> XEN_SYSCTL_PSR_MBA_get_info
* Key data structure:
1. Feature HW info
```
struct {
unsigned int thrtl_max;
bool linear;
} mba;
- Member `thrtl_max`
`thrtl_max` is the max throttling value to be set, i.e. MBA_MAX.
- Member `linear`
`linear` means the response of delay value is linear or not.
As mentioned above, MBA is a member of Intel PSR features, it shares the
base PSR infrastructure in Xen. For example, the 'cos_max' is a common HW
property for all features. So, for other data structure details, please
refer to 'intel_psr_cat_cdp.pandoc'.
# Limitations
MBA can only work on HW which supports it (check CPUID).
# Testing
We can execute these commands to verify MBA on different HWs supporting them.
For example:
1. User can get the MBA hardware info through 'psr-hwinfo' command. From
result, user can know if this hardware works under linear mode or non-
linear mode, the max throttling value (MBA_MAX) and so on.
root@:~$ xl psr-hwinfo --mba
Memory Bandwidth Allocation (MBA):
Socket ID : 0
Linear Mode : Enabled
Maximum COS : 7
Maximum Throttling Value: 90
Default Throttling Value: 0
2. Then, user can set a throttling value to a domain. For example, set '10',
i.e 10% delay.
root@:~$ xl psr-mba-set 1 10
3. User can check the current configuration of the domain through
'psr-mab-show'. For linear mode, the decimal value is shown.
root@:~$ xl psr-mba-show 1
Socket ID : 0
Default THRTL : 0
ID NAME THRTL
1 ubuntu14 10
# Areas for improvement
N/A
# Known issues
N/A
# References
"INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES" [Intel 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
# History
------------------------------------------------------------------------
Date Revision Version Notes
---------- -------- -------- -------------------------------------------
2017-01-10 1.0 Xen 4.9 Design document written
2017-07-10 1.1 Xen 4.10 Changes:
1. Modify data structure according to latest
codes;
2. Add content for 'Areas for improvement';
3. Other minor changes.
2017-08-09 1.2 Xen 4.10 Changes:
1. Remove a special character to avoid error when
building pandoc.
2017-08-15 1.3 Xen 4.10 Changes:
1. Add terminology 'HW'.
2. Change 'COS ID of VCPU' to 'COS ID of domain'.
3. Change 'COS register' to 'Thrtl MSR'.
4. Explain the value shown for 'psr-mba-show' under
different modes.
5. Remove content in 'Areas for improvement'.
2017-08-16 1.4 Xen 4.10 Changes:
1. Add '<>' for mandatory argument.
2017-08-30 1.5 Xen 4.10 Changes:
1. Modify words in 'Overview' to make it easier to
understand.
2. Explain 'linear/non-linear' modes before mention
them.
3. Explain throttling value more accurate.
4. Explain 'MBA_MAX'.
5. Correct some words in 'Design Overview'.
6. Change 'mba_info' to 'mba' according to code
changes. Also, modify contents of it.
7. Add context in 'Testing' part to make things
more clear.
8. Remove 'n<64' to avoid out-of-sync.
2017-09-21 1.6 Xen 4.10 Changes:
1. Add 'domain-name' as parameter of 'psr-mba-show/
psr-mba-set'.
2. Fix some wordings.
3. Explain how user can know the MBA_MAX.
4. Move the description of 'Linear mode/Non-linear
mode' into section of 'psr-mba-show'.
5. Change 'per-thread' to 'per-hyper-thread'.
2017-09-29 1.7 Xen 4.10 Changes:
1. Correct some words.
2. Change 'xl psr-mba-set 1 0xa' to
'xl psr-mba-set 1 10'
2017-10-08 1.8 Xen 4.10 Changes:
1. Correct some words.
---------- -------- -------- -------------------------------------------
Reference in new issue View Git Blame Copy Permalink