mirror of
https://github.com/xen-project/xen.git
synced 2026-10-28 02:01:24 +01:00
Andrew Cooper
d661611d08
c/s a3a99df44 "docs/cmdline: Rewrite the cpuid_mask_* section" completely
forgot about how markdown gets rendered to HTML (as opposed to PDF), because
we use different translators depending on the destination format.
markdown and pandoc are very similar markup languages, but a couple of details
about pandoc cause it to have far more user-friendly inline markup.
Switch all markdown documents to be pandoc (so we are using a single
translator, and therefore a single flavour of markdown), which fixes the
rendered docs on xenbits.xen.org/docs.
While changing the format, fix the remainder of the escaped underscores in the
same mannor as the previous patch. The two problem cases here are __LINE__
and __FILE__ where the first underscore still needs escaping.
In addition, dmop.markdown and dom0less.markdown didn't used to get processed,
as only .markdown files in the misc/ directory got considered.
dom0less.pandoc gets picked up automatically now, due to being in the
features/ directory, but designs/ needs adding to the pandoc directory list
for dmop.pandoc to get processed.
While edting in appropriate areas, take the opportunity to fix some markup to
the surrounding style, and drop trailing whitespace.
No change in content - only formatting. This results in the text being easier
to read and grep.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
255 lines
10 KiB
Plaintext
255 lines
10 KiB
Plaintext
# Intel Platform Shared Resource Monitoring/Control in xl
|
|
|
|
This document introduces Intel Platform Shared Resource Monitoring/Control
|
|
technologies, their basic concepts and the xl interfaces.
|
|
|
|
## Cache Monitoring Technology (CMT)
|
|
|
|
Cache Monitoring Technology (CMT) is a new feature available on Intel Haswell
|
|
and later server platforms that allows an OS or Hypervisor/VMM to determine
|
|
the usage of cache (currently only L3 cache supported) by applications running
|
|
on the platform. A Resource Monitoring ID (RMID) is the abstraction of the
|
|
application(s) that will be monitored for its cache usage. The CMT hardware
|
|
tracks cache utilization of memory accesses according to the RMID and reports
|
|
monitored data via a counter register.
|
|
|
|
For more detailed information please refer to Intel SDM chapter
|
|
"Platform Shared Resource Monitoring: Cache Monitoring Technology".
|
|
|
|
In Xen's implementation, each domain in the system can be assigned a RMID
|
|
independently, while RMID=0 is reserved for monitoring domains that don't
|
|
have CMT service attached. RMID is opaque for xl/libxl and is only used in
|
|
hypervisor.
|
|
|
|
### xl interfaces
|
|
|
|
A domain is assigned a RMID implicitly by attaching it to CMT service:
|
|
|
|
`xl psr-cmt-attach <domid>`
|
|
|
|
After that, cache usage for the domain can be shown by:
|
|
|
|
`xl psr-cmt-show cache-occupancy <domid>`
|
|
|
|
Once monitoring is not needed any more, the domain can be detached from the
|
|
CMT service by:
|
|
|
|
`xl psr-cmt-detach <domid>`
|
|
|
|
An attach may fail because of no free RMID available. In such case unused
|
|
RMID(s) can be freed by detaching corresponding domains from CMT service.
|
|
|
|
Maximum RMID and supported monitor types in the system can be obtained by:
|
|
|
|
`xl psr-hwinfo --cmt`
|
|
|
|
## Memory Bandwidth Monitoring (MBM)
|
|
|
|
Memory Bandwidth Monitoring(MBM) is a new hardware feature available on Intel
|
|
Broadwell and later server platforms which builds on the CMT infrastructure to
|
|
allow monitoring of system memory bandwidth. It introduces two new monitoring
|
|
event type to monitor system total/local memory bandwidth. The same RMID can
|
|
be used to monitor both cache usage and memory bandwidth at the same time.
|
|
|
|
For more detailed information please refer to Intel SDM chapter
|
|
"Overview of Cache Monitoring Technology and Memory Bandwidth Monitoring".
|
|
|
|
In Xen's implementation, MBM shares the same set of underlying monitoring
|
|
service with CMT and can be used to monitor memory bandwidth on a per domain
|
|
basis.
|
|
|
|
The xl interfaces are the same with that of CMT. The difference is the
|
|
monitor type is corresponding memory monitoring type (local-mem-bandwidth/
|
|
total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach`:
|
|
|
|
`xl psr-cmt-show local-mem-bandwidth <domid>`
|
|
|
|
`xl psr-cmt-show total-mem-bandwidth <domid>`
|
|
|
|
## Cache Allocation Technology (CAT)
|
|
|
|
Cache Allocation Technology (CAT) is a new feature available on Intel
|
|
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
|
|
partition cache allocation (i.e. L3/L2 cache) based on application priority or
|
|
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
|
|
which represent cache capacity and indicate the degree of overlap and
|
|
isolation between classes. System cache resource is divided into numbers of
|
|
minimum portions which is then made up into subset for cache partition. Each
|
|
portion corresponds to a bit in CBM and the set bit represents the
|
|
corresponding cache portion is available.
|
|
|
|
For example, assuming a system with 8 portions and 3 domains:
|
|
|
|
* A CBM of 0xff for every domain means each domain can access the whole cache.
|
|
This is the default.
|
|
|
|
* Giving one domain a CBM of 0x0f and the other two domain's 0xf0 means that
|
|
the first domain gets exclusive access to half of the cache (half of the
|
|
portions) and the other two will share the other half.
|
|
|
|
* Giving one domain a CBM of 0x0f, one 0x30 and the last 0xc0 would give the
|
|
first domain exclusive access to half the cache, and the other two exclusive
|
|
access to one quarter each.
|
|
|
|
For more detailed information please refer to Intel SDM chapter
|
|
"Platform Shared Resource Control: Cache Allocation Technology".
|
|
|
|
In Xen's implementation, CBM can be configured with libxl/xl interfaces but
|
|
COS is maintained in hypervisor only. The cache partition granularity is per
|
|
domain, each domain has COS=0 assigned by default, the corresponding CBM is
|
|
all-ones, which means all the cache resource can be used by default.
|
|
|
|
### xl interfaces
|
|
|
|
System CAT information such as maximum COS and CBM length can be obtained by:
|
|
|
|
`xl psr-hwinfo --cat`
|
|
|
|
The simplest way to change a domain's CBM from its default is running:
|
|
|
|
`xl psr-cat-set [OPTIONS] <domid> <cbm>`
|
|
|
|
where cbm is a number to represent the corresponding cache subset can be used.
|
|
A cbm is valid only when:
|
|
|
|
* Set bits only exist in the range of [0, cbm_len), where cbm_len can be
|
|
obtained with `xl psr-hwinfo --cat`.
|
|
* All the set bits are contiguous.
|
|
|
|
In a multi-socket system, the same cbm will be set on each socket by default.
|
|
Per socket cbm can be specified with the `--socket SOCKET` option.
|
|
|
|
In different systems, the different cache level is supported, e.g. L3 cache or
|
|
L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
|
|
|
|
Setting the CBM may not be successful if insufficient COS is available. In
|
|
such case unused COS(es) may be freed by setting CBM of all related domains to
|
|
its default value(all-ones).
|
|
|
|
Per domain CBM settings can be shown by:
|
|
|
|
`xl psr-cat-show [OPTIONS] <domid>`
|
|
|
|
In different systems, the different cache level is supported, e.g. L3 cache or
|
|
L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
|
|
|
|
## Code and Data Prioritization (CDP)
|
|
|
|
Code and Data Prioritization (CDP) Technology is an extension of CAT, which
|
|
is available on Intel Broadwell and later server platforms. CDP enables
|
|
isolation and separate prioritization of code and data fetches to the L3
|
|
cache in a software configurable manner, which can enable workload
|
|
prioritization and tuning of cache capacity to the characteristics of the
|
|
workload. CDP extends Cache Allocation Technology (CAT) by providing
|
|
separate code and data masks per Class of Service (COS).
|
|
|
|
CDP can be enabled by adding `psr=cdp` to Xen command line.
|
|
|
|
When CDP is enabled,
|
|
|
|
* the CAT masks are re-mapped into interleaved pairs of masks for data or
|
|
code fetches.
|
|
|
|
* the range of COS for CAT is re-indexed, with the lower-half of the COS
|
|
range available for CDP.
|
|
|
|
CDP allows the OS or Hypervisor to partition cache allocation in a more
|
|
fine-grained manner. Code cache and data cache can be specified independently.
|
|
With CDP enabled, one COS corresponds to two CBMs (code CBM & data CBM),
|
|
since the sum of CBMs is fixed, that means the number of available COSes
|
|
will reduce by half when CDP is on.
|
|
|
|
For more detailed information please refer to Intel SDM chapter
|
|
"Platform Shared Resource Control: Cache Allocation Technology".
|
|
|
|
The xl interfaces are the same with that of CAT. The difference is that
|
|
CBM type can be passed as option to set code CBM or data CBM.
|
|
|
|
When CDP is enabled, `-c` or `--code` option is available to set code CBM
|
|
for the domain.
|
|
|
|
When CDP is enabled, `-d` or `--data` option is available to set data CBM
|
|
for the domain.
|
|
|
|
If neither `-c` nor `-d` option is specified when CDP is on, the same code
|
|
CBM and data CBM will be set for the domain. Passing both `-c` and `-d`
|
|
options is invalid.
|
|
|
|
Example:
|
|
|
|
Setting code CBM for a domain:
|
|
`xl psr-cat-set -c <domid> <cbm>`
|
|
|
|
Setting data CBM for a domain:
|
|
`xl psr-cat-set -d <domid> <cbm>`
|
|
|
|
Setting the same code and data CBM for a domain:
|
|
`xl psr-cat-set <domid> <cbm>`
|
|
|
|
## Memory Bandwidth Allocation (MBA)
|
|
|
|
Memory Bandwidth Allocation (MBA) is a new feature available on Intel
|
|
Skylake and later server platforms that allows an OS or Hypervisor/VMM to
|
|
slow misbehaving apps/VMs by using a credit-based throttling mechanism. To
|
|
enforce bandwidth on a specific domain, just set throttling value (THRTL)
|
|
into Class of Service (COS). MBA provides two THRTL mode. One is linear mode
|
|
and the other is non-linear mode.
|
|
|
|
In the linear mode the input precision is defined as 100-(THRTL_MAX). Values
|
|
not an even multiple of the precision (e.g., 12%) will be rounded down (e.g.,
|
|
to 10% delay by the hardware).
|
|
|
|
If linear values are not supported then input delay values are powers-of-two
|
|
from zero to the THRTL_MAX value from CPUID. In this case any values not a power
|
|
of two will be rounded down the next nearest power of two.
|
|
|
|
For example, assuming a system with 2 domains:
|
|
|
|
* A THRTL of 0x0 for every domain means each domain can access the whole cache
|
|
without any delay. This is the default.
|
|
|
|
* Linear mode: Giving one domain a THRTL of 0xC and the other domain's 0 means
|
|
that the first domain gets 10% delay to access the cache and the other one
|
|
without any delay.
|
|
|
|
* Non-linear mode: Giving one domain a THRTL of 0xC and the other domain's 0
|
|
means that the first domain gets 8% delay to access the cache and the other
|
|
one without any delay.
|
|
|
|
For more detailed information please refer to Intel SDM chapter
|
|
"Introduction to Memory Bandwidth Allocation".
|
|
|
|
In Xen's implementation, THRTL can be configured with libxl/xl interfaces but
|
|
COS is maintained in hypervisor only. The cache partition granularity is per
|
|
domain, each domain has COS=0 assigned by default, the corresponding THRTL is
|
|
0, which means all the cache resource can be accessed without delay.
|
|
|
|
### xl interfaces
|
|
|
|
System MBA information such as maximum COS and maximum THRTL can be obtained by:
|
|
|
|
`xl psr-hwinfo --mba`
|
|
|
|
The simplest way to change a domain's THRTL from its default is running:
|
|
|
|
`xl psr-mba-set [OPTIONS] <domid> <thrtl>`
|
|
|
|
In a multi-socket system, the same thrtl will be set on each socket by default.
|
|
Per socket thrtl can be specified with the `--socket SOCKET` option.
|
|
|
|
Setting the THRTL may not be successful if insufficient COS is available. In
|
|
such case unused COS(es) may be freed by setting THRTL of all related domains to
|
|
its default value(0).
|
|
|
|
Per domain THRTL settings can be shown by:
|
|
|
|
`xl psr-mba-show [OPTIONS] <domid>`
|
|
|
|
For linear mode, it shows the decimal value. For non-linear mode, it shows
|
|
hexadecimal value.
|
|
|
|
## Reference
|
|
|
|
[1] Intel SDM
|
|
(http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html).
|