There is a bug in commit 5e4b419966 ("xen: credit2: only reset
credit on reset condition"). In fact, the aim of that commit was to
make sure that we do not perform too many credit reset operations
(which are not super cheap, and in an hot-path). But the check used
to determine whether a reset is necessary was the wrong one.
In fact, knowing just that some vCPUs have been skipped, while
traversing the runqueue (in runq_candidate()), is not enough. We
need to check explicitly whether the first vCPU in the runqueue
has a negative amount of credit.
Since a trace record is changed, this patch updates xentrace format file
and xenalyze as well
This should be backported.
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
There have been report of stalls of guest vCPUs, when Credit2 was used.
It seemed like these vCPUs were not getting scheduled for very long
time, even under light load conditions (e.g., during dom0 boot).
Investigations led to the discovery that --although rarely-- it can
happen that a vCPU manages to run for very long timeslices. In Credit2,
this means that, when runtime accounting happens, the vCPU will lose a
large quantity of credits. This in turn may lead to the vCPU having less
credits than the idle vCPUs (-2^30). At this point, the scheduler will
pick the idle vCPU, instead of the ready to run vCPU, for a few
"epochs", which often times is enough for the guest kernel to think the
vCPU is not responding and crashing.
An example of this situation is shown here. In fact, we can see d0v1
sitting in the runqueue while all the CPUs are idle, as it has
-1254238270 credits, which is smaller than -2^30 = −1073741824:
(XEN) Runqueue 0:
(XEN) ncpus = 28
(XEN) cpus = 0-27
(XEN) max_weight = 256
(XEN) pick_bias = 22
(XEN) instload = 1
(XEN) aveload = 293391 (~111%)
(XEN) idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
(XEN) tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
(XEN) fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
[...]
(XEN) Runqueue 0:
(XEN) CPU[00] runq=0, sibling=00,..., core=00,...
(XEN) CPU[01] runq=0, sibling=00,..., core=00,...
[...]
(XEN) CPU[26] runq=0, sibling=00,..., core=00,...
(XEN) CPU[27] runq=0, sibling=00,..., core=00,...
(XEN) RUNQ:
(XEN) 0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)
We certainly don't want, under any circumstance, this to happen.
Let's, therefore, define a minimum amount of credits a vCPU can have.
During accounting, we make sure that, for however long the vCPU has
run, it will never get to have less than such minimum amount of
credits. Then, we set the credits of the idle vCPU to an even
smaller value.
NOTE: investigations have been done about _how_ it is possible for a
vCPU to execute for so much time that its credits becomes so low. While
still not completely clear, there are evidence that:
- it only happens very rarely,
- it appears to be both machine and workload specific,
- it does not look to be a Credit2 (e.g., as it happens when
running with Credit1 as well) issue, or a scheduler issue.
This patch makes Credit2 more robust to events like this, whatever
the cause is, and should hence be backported (as far as possible).
Reported-by: Glen <glenbarney@gmail.com>
Reported-by: Tomas Mozes <hydrapolic@gmail.com>
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
This commit implements the Xen part of the cap mechanism for
Credit2.
A cap is how much, in terms of % of physical CPU time, a domain
can execute at most.
For instance, a domain that must not use more than 1/4 of
one physical CPU, must have a cap of 25%; one that must not
use more than 1+1/2 of physical CPU time, must be given a cap
of 150%.
Caps are per domain, so it is all a domain's vCPUs, cumulatively,
that will be forced to execute no more than the decided amount.
This is implemented by giving each domain a 'budget', and
using a (per-domain again) periodic timer. Values of budget
and 'period' are chosen so that budget/period is equal to the
cap itself.
Budget is burned by the domain's vCPUs, in a similar way to
how credits are.
When a domain runs out of budget, its vCPUs can't run any
longer. They can gain, when the budget is replenishment by
the timer, which event happens once every period.
Blocking the vCPUs because of lack of budget happens by
means of a new (_VPF_parked) pause flag, so that, e.g.,
vcpu_runnable() still works. This is similar to what is
done in sched_rtds.c, as opposed to what happens in
sched_credit.c, where vcpu_pause() and vcpu_unpause()
(which means, among other things, more overhead).
Note that, while adding new fields to csched2_vcpu and
csched2_dom, currently existing members are being moved
around, to achieve best placement inside cache lines.
Note also that xenalyze and tools/xentrace/format are being
updated too.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Soft-affinity support is usually implemented by means
of a two step "balancing loop", where:
- during the first step, we consider soft-affinity
(if the vcpu has one);
- during the second (if we get to it), we consider
hard-affinity.
In runq_tickle(), we need to do that for checking
whether we can execute the waking vCPU on an pCPU
that is idle. In fact, we want to be sure that, if
there is an idle pCPU in the vCPU's soft affinity,
we'll use it.
If there are no such idle pCPUs, though, and we
have to check non-idle ones, we can avoid the loop
and to both hard and soft-affinity in one pass.
In fact, we can we scan runqueue and compute a
"score" for each vCPU which is running on each pCPU.
The idea is, since we may have to preempt someone:
- try to make sure that the waking vCPU will run
inside its soft-affinity,
- try to preempt someone that is running outside
of its own soft-affinity.
The value of the score is added to a trace record,
so xenalyze's code and tools/xentrace/formats are
updated accordingly.
Suggested-by: George Dunlap <george.dunlap@citrix.com>
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Including whether we actually tried stealing a vCPU from
a given pCPU, or we skipped that one, because of lock
contention.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
We record trace information about the next timeslice when
switching to a different vcpu, but not when continuing to
run the same cpu:
csched2:schedule cpu 9, rq# 1, idle, SMT idle, tickled
csched2:runq_candidate d0v3, 0 vcpus skipped, cpu 9 was tickled
sched_switch prev d32767v9, run for 991.186us
sched_switch next d0v3, was runnable for 2.515us, next slice 10000.0us
sched_switch prev d32767v9 next d0v3 ^^^^^^^^^^^^^^^^^^^^
runstate_change d32767v9 running->runnable
...
csched2:schedule cpu 2, rq# 0, busy, not tickled
csched2:burn_credits d1v5, credit = 9996950, delta = 502913
csched2:runq_candidate d1v5, 0 vcpus skipped, no cpu was tickled
runstate_continue d1v5 running->running
?????????????
This information is quite useful; so add a trace including
that information on the 'continue_running' path as well,
like this:
csched2:schedule cpu 1, rq# 0, busy, not tickled
csched2:burn_credits d0v8, credit = 9998645, delta = 12104
csched2:runq_candidate d0v8, credit = 9998645, 0 vcpus skipped, no cpu was tickled
sched_switch continue d0v8, run for 1125.820us, next slice 9998.645us
runstate_continue d0v8 running->running ^^^^^^^^^^^^^^^^^^^^^
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
When traversing a Credit2 runqueue to select the
best candidate vCPU to be run next, show in the
trace which vCPUs we consider.
A bit verbose, but quite useful, considering that
we may end up looking at, but then discarding, one
of more vCPU. This will help understand which ones
are skipped and why.
Also, add how much credits the chosen vCPU has
(in the TRC_CSCHED2_RUNQ_CANDIDATE record). And,
while there, fix a bug in tools/xentrace/formats
(still in the output of TRC_CSCHED2_RUNQ_CANDIDATE).
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
There are some scheduling related trace records that
are not being taken care of (and hence only dumped as
raw records).
Some of them are being introduced in this series, while
other were just neglected by previous patches.
Add support for them.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Right now, two out of the three events related to
context switch (that is TRC_SCHED_SWITCH_INFPREV and
TRC_SCHED_SWITCH_INFNEXT) only report the domain id,
and not the vcpu id.
That's omitting a useful piece of information, and
even if we be figured that out by looking at other
records, that's unnecessarily complicated (especially
if working on a trace from a sctipt).
This changes both the tracing code in Xen and the parsing
code in tools at once, to avoid introducing transitional
regressions.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
more specifically, with: TICKLE_NEW, RUNQ_MAX_WEIGHT,
MIGRATE, LOAD_CHECK, LOAD_BALANCE and PICKED_CPU, and
in both both xenalyze and formats (for xentrace_format).
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Add the shift used for the precision of the integer
arithmetic to the trace records, and update both xenalyze
and xentrace_format to make use of/print it.
In particular, in xenalyze, we are can now show the
load as a (easier to interpreet) percentage.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
to include the vcpu IDs, in a way that matches
how the "dom:vcpu" couple is displayed in other
events (runstate changes).
Also add the trace for TRC_SCHED_SHUTDOWN_CODE which
was missing and was done via SCHEDOP_shutdown_code hypercall.
(TRC_SCHED_SHUTDOWN trace was present).
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Olaf Hering <olaf@aepfle.de>
---
Changes from v1:
* enhanced changelog, as suggested during review.
This add a set of trace events that track the setup of various
emulated devices related to timers in domU.
This set is hpet, pit (i8253, i8254), rtc (MC146818), apic (lapic),
and pic (i8259). The pmtimer is not traced since it does not have a
changeable rate.
Signed-off-by: Don Slutz <dslutz@verizon.com>
Reviewed-by: Tim Deegan <tim@xen.org>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
This expands the format to include the class of TRC_HW_IRQ.
This means that instead of:
CPU28 1753521436727 (+ 3252) unknown (0x0000000000802008) [ 0x0000006c 0x4605709c 0x4605b682 0x00000000 0x00000000 0x00000000 0x00000000 ]
we now see:
CPU28 1753521436727 (+ 3252) do_irq [ irq = 108, began = 1174761628us, ended = 1174779522us ]
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Add a trace record for every hypercall inside a multicall. These use
a new event ID (with a different sub-class ) so they may be filtered
out if only the calls into hypervisor are of interest.
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Committed-by: Keir Fraser <keir@xen.org>
Trace hypercalls using a more useful trace record format.
The EIP field is removed (it was always somewhere in the hypercall
page) and include selected hypercall arguments (e.g., the number of
calls in a multicall, and the number of PTE updates in an mmu_update
etc.). 12 bits in the first extra word are used to indicate which
arguments are present in the record and what size they are (32 or
64-bit).
This is an incompatible record format so a new event ID is used so
tools can distinguish between the two formats.
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Committed-by: Keir Fraser <keir@xen.org>
We want to add additional sub-classes for TRC_PV tracepoints and to be
able to only capture these new sub-classes. This cannot currently be
done as the existing tracepoints all use a sub-class of 0xf.
So, redefine the PV events to use a new sub-class. All the current
tracepoints are tracing entry points to the hypervisor so the
sub-class is named TRC_PV_ENTRY.
This change does not affect xenalyze as that only looks at the main
class and the event number and does not use the sub-class field.
Signed-off-by: Frediano Ziglio <frediano.ziglio@citrix.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Committed-by: Keir Fraser <keir@xen.org>
xentrace_format cannot use "0x016x" format as we expect.
It show only %(N) as "0x016x" format, not as "%(N+1)08x%(N)08x".
So I fixed tools/xentrace/formats by using "%(N+1)08x%(N)08x" format.
Also I added some TRC_PV entries.
Signed-off-by: Akio Takebe <takebe_akio@jp.fujitsu.com>
The reason is that tsc stops and it causes the inaccuracy.
And later we can write some scripts based on this patch.
Signed-off-by: Guanqun Lu <guanqun.lu@intel.com>
Xen tracing some times ago used to put values of type 'long' into the
trace buffer. This has changed to uint32_t. Some trace points log
virtual addresses, which get cropped to 32bit in this case. There were
some inline functions to handle at least PF_XEN and VMEXIT, which
caused a lot of code duplication. The attached patch fixes several
issues:
1. fix and extend tools/xentrace/formats
2. Fix xentrace_format to handle up to 7 parameters
3. create convenience macros to properly log long values
4. remove the inline functions in hvm/trace.h and replace them by macros
5. Change the CPUID trace to work correctly
6. group HVM trace points enable mechanism
I used a similar approach as in PV tracing with bit 8 indicating 64bit
pointers.
Signed-off-by: Andre Przywara <andre.przywara@amd.com>
Additionally a default line is added (key 0x00000000) which is used
when no specific entry in the format file is found.
Signed-off-by: Thomas Friebel <thomas.friebel@amd.com>
be enabled and disabled dynamically while the system is running. The
existing DOM0_TBUFCONTROL hypercall is used, and three new operations
have been added: enable, disable, and set_size. This is to address
concerns that use of the trace buffers impacts system performance.
The initial value of opt_tbuf_size is set to zero, so that trace
buffers remain disabled by default. They can be enabled as they used
to be, by adding tbuf_size= to the xen command line at boot time. They
can also be turned on and off any time when the system is running. The
size of the trace buffer allocation can also be changed dynamically
while the system is running. The trace buffers may be completely
deallocated by setting the size to zero. No change in buffer size can
be done while tracing is enabled;
Other Changes made:
- Update the constants in tools/xentrace/formats to match those in
public/trace.h
- libxc functions for enable/disable, get/set size
Still left to do:
- remove all ifdef's for trace buffers and the 'trace' build option
- xm command to do enable/disable trace buffers and get/set size
Signed-off-by: Rob Gardner <rob.gardner@hp.com>
The low level asm instrumentation is needed to get precise TSC counts.
trace=n should have precisely 0 overhead because of this patch.
Signed-off-by: Yunfeng Zhai <yunfeng.zhao@intel.com>
Signed-off-by: Arun Sharma <arun.sharma@intel.com>
New Xen trace buffer implementation that avoids blocking
interrupts. Add support for trace buffer CPU mask and
event mask. Add support for trace classes. New DOM0 trace
buffer hypercall. Fix 'xentrace' signal handler bug.
Signed-off-by: bin.ren@cl.cam.ac.uk
Dario Faggioli
Dario Faggioli
George Dunlap
Don Slutz
Konrad Rzeszutek Wilk
David Vrabel
Keir Fraser
Christoph Egger
Keir Fraser
Keir Fraser
kfraser@localhost.localdomain
kaf24@firebug.cl.cam.ac.uk
adsharma@los-vmm.sc.intel.com
bren@br260.wolfson.cam.ac.uk