We warn when we see data for a vcpu moving into a non-RUNNING state,
just so that people know why we're ignoring it. On full traces, this
happens only once. However, if the trace was limited to a subset of
pcpus, then this will happen every time the domain in question is
woken on that pcpu.
Add a 'delayed_init' flag to the vcpu struct to indicate when a vcpu
has experienced a delayed init. Print a warning message once when
entering the state, and once when leaving it.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
EIP lists are generalized across several use cases. For many of them,
it make sense to have a cycle per sample; but not really for interrupt
EIP lists. For this reason, it normally just passes 0 as for the tsc
value, which will in turn down at the bottom of update_cycles(),
update only the summary.event_count, but nothing else.
The dump_eip() function attempted to handle this by calling the generic
cycle print handler if the summary contained *any* cycles, and by collecting
and printing its own stats, based solely on counts, if not.
Unfortunately, it used the wrong element for this: it collected the
total from samples.count rather samples.event_count; in the case that
there are no cycles, this will always be zero. It then divided by
this zero value. This results in output that looked like this:
```
ffff89d29656 : 0 -nan%
ffff89d298b6 : 0 -nan%
ffff89d298c0 : 0 -nan%
```
It's better than nothing, but a lot less informative than one would
like.
Use event_count rather than count for collecting the total, and the
reporting when there are no cycles in the summary information. This results
in output that looks like this:
```
ffff89d29656 : 2 1.21%
ffff89d298b6 : 1 0.61%
ffff89d298c0 : 1 0.61%
```
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
If a vmentry/exit arc unexpectedly doesn't have a handler, we throw an
error, and then log the information under HVM event 0; thus those
particular traces within the vmexit reason will have stats gathered,
and will show up with "(no handler)". This is useful in the event
that there are unusual paths through the hypervisor which don't have
trace points.
However, if there are more than one of these, then although we detect and warn
that this is happening, we subsequently continue to stuff data about all exits
into that one exit, even though we only show it in one place.
Instead, refator things to only allow a single exit reason to be
accumulated into any given event.
Also put a comment explaining what's going on, and how to fix it.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
These defines are not used by xenalyze.
NR_CPUS will not match what was used for building Xen itself.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Reviewed-by: George Dunlap <george.dunlap@cloud.com>
Invoking the --help option of any tool should not return with an error,
if that tool does have a documented and implemented help option.
Adjust the usage() function to exit with either error or success.
Handle the existing entry in the option table to call usage accordingly.
Adjust the getopt value for help. The char '?' is returned for unknown
options. Returning 'h' instead of '?' allows to handle --help.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Reviewed-by: George Dunlap <george.dunlap@cloud.com>
A recent xentrace highlighted an unhandled corner case in the vcpu
"start-of-day" logic, if the trace starts after the last running ->
non-running transition, but before the first non-running -> running
transition. Because start-of-day wasn't handled, vcpu_next_update()
was expecting p->current to be NULL, and tripping out with the
following error message when it wasn't:
vcpu_next_update: FATAL: p->current not NULL! (d32768dv$p, runstate RUNSTATE_INIT)
where 32768 is the DEFAULT_DOMAIN, and $p is the pcpu number.
Instead of calling vcpu_start() piecemeal throughout
sched_runstate_process(), call it at the top of the function if the
vcpu in question is still in RUNSTATE_INIT, so that we can handle all
the cases in one place.
Sketch out at the top of the function all cases which we need to
handle, and what to do in those cases. Some transitions tell us where
v is running; some transitions tell us about what is (or is not)
running on p; some transitions tell us neither.
If a transition tells us where v is now running, update its state;
otherwise leave it in INIT, in order to avoid having to deal with TSC
skew on start-up.
If a transition tells us what is or is not running on p, update
p->current (either to v or NULL). Otherwise leave it alone.
If neither, do nothing.
Reifying those rules:
- If we're continuing to run, set v to RUNNING, and use p->first_tsc
as the runstate time.
- If we're starting to run, set v to RUNNING, and use ri->tsc as the
runstate time.
- If v is being deschedled, leave v in the INIT state to avoid dealing
with TSC skew; but set p->current to NULL so that whatever is
scheduled next won't trigger the assert in vcpu_next_update().
- If a vcpu is waking up (switching from one non-runnable state to
another non-runnable state), leave v in INIT, and p in whatever
state it's in (which may be the default domain, or some other vcpu
which has already run).
While here, fix the comment above vcpu_start; it's called when the
vcpu state is INIT, not when current is the default domain.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
The program is structured so that fatal errors cause exit() to be
called directly, rather than being passed up the stack; returning a
value here may mislead people into believing otherwise.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Reviewed-by: George Dunlap <george.dunlap@cloud.com>
The output file is optional. In case it is missing, xentrace is supposed
to write to stdout - unless it is a tty, which is checked prior using it.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: George Dunlap <george.dunlap@cloud.com>
clang complains about the signed type:
implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
The potential ABI change in libxenvchan is covered by the Xen version based SONAME.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Reviewed-by: Juergen Gross <jgross@suse.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Four TRC_MEM_* records supply custom structures with tail padding, leaking
stack rubble into the trace buffer. Three of the records were fine in 32-bit
builds of Xen, due to the relaxed alignment of 64-bit integers, but
POD_SUPERPAGE_SPLITER was broken right from the outset.
We could pack the datastructures to remove the padding, but xentrace_format
has no way of rendering the upper half of a 16-bit field. Instead, expand all
16-bit fields to 32-bit.
For POD_SUPERPAGE_SPLINTER, introduce an order field as it is relevant
information, and to match DECREASE_RESERVATION, and so it doesn't require a
__packed attribute to drop tail padding.
Update xenalyze's structures to match, and introduce xentrace_format rendering
which was absent previously.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Dario Faggioli <dfaggioli@suse.com>
In order to better reflect the contents of the header and to make it
more appropriate to use it for different runtime environments like
programs, libraries, and firmware, rename the libs.h include file to
common-macros.h. Additionally add a comment pointing out the need to be
self-contained.
Suggested-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Acked-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> # tools/python/xen/lowlevel/xc/xc.c
Acked-by: Christian Lindig <christian.lindig@cloud.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Remove "build" targets.
Use "$(TARGETS)" to list binary to be built.
Cleanup "clean" rule.
Also drop conditional install of $(BIN) and $(LIBBIN) as those two
variables are now always populated.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
The first thing XEN_SYSCTL_physinfo does is zero op->u.physinfo.
Do not copy-in. It's pointless, and most callers don't initialise their
xc_physinfo_t buffer to begin with. Remove the redundant zeroing from the
remaining callers.
Spotted by Coverity.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Wei Chen <Wei.Chen@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
918b8842a8 changed CPSR and SPSR to be stored as 64bit values.
This is fixing the print size in some tools to use 64bit type.
Fixes: 918b8842a8 ("arm64: Change type of hsr, cpsr, spsr_el1 to uint64_t")
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@arm.com>
Tested-by: Michal Orzel <michal.orzel@arm.com>
literal strings are not meant to be modified. So we should use const
char * rather than char * when we want to store a pointer to them.
Signed-off-by: Julien Grall <jgrall@amazon.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Use unsigned char variable, or cast to (unsigned char), for
tolower()/islower() and friends. Fix compiler error
array subscript has type 'char' [-Werror=char-subscripts]
Signed-off-by: Manuel Bouyer <bouyer@netbsd.org>
Reviewed-by: Ian Jackson <ian.jackson@eu.citrix.com>
Release-Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
There is a bug in commit 5e4b419966 ("xen: credit2: only reset
credit on reset condition"). In fact, the aim of that commit was to
make sure that we do not perform too many credit reset operations
(which are not super cheap, and in an hot-path). But the check used
to determine whether a reset is necessary was the wrong one.
In fact, knowing just that some vCPUs have been skipped, while
traversing the runqueue (in runq_candidate()), is not enough. We
need to check explicitly whether the first vCPU in the runqueue
has a negative amount of credit.
Since a trace record is changed, this patch updates xentrace format file
and xenalyze as well
This should be backported.
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
There have been report of stalls of guest vCPUs, when Credit2 was used.
It seemed like these vCPUs were not getting scheduled for very long
time, even under light load conditions (e.g., during dom0 boot).
Investigations led to the discovery that --although rarely-- it can
happen that a vCPU manages to run for very long timeslices. In Credit2,
this means that, when runtime accounting happens, the vCPU will lose a
large quantity of credits. This in turn may lead to the vCPU having less
credits than the idle vCPUs (-2^30). At this point, the scheduler will
pick the idle vCPU, instead of the ready to run vCPU, for a few
"epochs", which often times is enough for the guest kernel to think the
vCPU is not responding and crashing.
An example of this situation is shown here. In fact, we can see d0v1
sitting in the runqueue while all the CPUs are idle, as it has
-1254238270 credits, which is smaller than -2^30 = −1073741824:
(XEN) Runqueue 0:
(XEN) ncpus = 28
(XEN) cpus = 0-27
(XEN) max_weight = 256
(XEN) pick_bias = 22
(XEN) instload = 1
(XEN) aveload = 293391 (~111%)
(XEN) idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
(XEN) tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
(XEN) fully idle cores: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
[...]
(XEN) Runqueue 0:
(XEN) CPU[00] runq=0, sibling=00,..., core=00,...
(XEN) CPU[01] runq=0, sibling=00,..., core=00,...
[...]
(XEN) CPU[26] runq=0, sibling=00,..., core=00,...
(XEN) CPU[27] runq=0, sibling=00,..., core=00,...
(XEN) RUNQ:
(XEN) 0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 (~100%)
We certainly don't want, under any circumstance, this to happen.
Let's, therefore, define a minimum amount of credits a vCPU can have.
During accounting, we make sure that, for however long the vCPU has
run, it will never get to have less than such minimum amount of
credits. Then, we set the credits of the idle vCPU to an even
smaller value.
NOTE: investigations have been done about _how_ it is possible for a
vCPU to execute for so much time that its credits becomes so low. While
still not completely clear, there are evidence that:
- it only happens very rarely,
- it appears to be both machine and workload specific,
- it does not look to be a Credit2 (e.g., as it happens when
running with Credit1 as well) issue, or a scheduler issue.
This patch makes Credit2 more robust to events like this, whatever
the cause is, and should hence be backported (as far as possible).
Reported-by: Glen <glenbarney@gmail.com>
Reported-by: Tomas Mozes <hydrapolic@gmail.com>
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
On Arm64, system registers are always 64-bit including SCTLR_EL1.
However, Xen is assuming this is 32-bit because earlier revision of
Armv8 had the top 32-bit RES0 (see ARM DDI0595.b).
>From Armv8.5, some bits in [63:32] will be defined and allowed to be
modified by the guest. So we would effectively reset those bits to 0
after each context switch. This means the guest may not function
correctly afterwards.
Rather than resetting to 0 the bits [63:32], preserve them across
context switch.
Note that the corresponding register on Arm32 (i.e SCTLR) is always
32-bit. So we need to use register_t anywhere we deal the SCTLR{,_EL1}.
Outside interface is switched to use 64-bit to allow ABI compatibility
between 32-bit and 64-bit.
[Stefano: fix typo in commit message]
Signed-off-by: Julien Grall <julien.grall@arm.com>
Signed-off-by: Stefano Stabellini <stefanos@xilinx.com>
Reviewed-by: Volodymyr Babchuk <volodymyr.babchuk@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Modified to provide building of the xenalyze binary for both ARM and
x86 platforms. The xenalyze binary is now built as part of the BIN
list for both platforms.
Signed-off-by: Benjamin Sanda <ben.sanda@dornerworks.com>
Signed-off-by: Julien Grall <julien.grall@arm.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
The per-cpu buffers for Xentrace are addressed by cpu-id, but the info
array for the buffers is sized only by number of online cpus. This
might lead to crashes when using Xentrace with smt=0.
The t_info structure has to be sized based on nr_cpu_ids.
Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Modify the xentrace utility to allow sparse cpu list resulting in not
all possible cpus having a trace buffer allocated.
Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
xen-tools/libs.h currently contains a shared BUILD_BUG_ON() implementation and
is used by some tools. Extend this to include ARRAY_SIZE and clean up all the
opencoding.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
With gcc optimization enabled by:
tools: detect appropriate debug optimization level
b435014517
-Wformat-truncation throws warnings
gcc version 7.3.0
xenalyze.c: In function 'find_symbol':
xenalyze.c:382:36: error: 'snprintf' output may be truncated before the last format character [-Werror=format-truncation=]
snprintf(name, 128, "(%s +%llx)",
^
xenalyze.c:382:5: note: 'snprintf' output between 6 and 144 bytes into a destination of size 128
snprintf(name, 128, "(%s +%llx)",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lastname, offset);
~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Signed-off-by: John Thomson <git@johnthomson.fastmail.com.au>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Recent changes in xenalyze introduced INT_MIN without also adding the
required header, fix this by adding the header.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
Release-acked-by: Julien Grall <julien.grall@linaro.org>
This commit implements the Xen part of the cap mechanism for
Credit2.
A cap is how much, in terms of % of physical CPU time, a domain
can execute at most.
For instance, a domain that must not use more than 1/4 of
one physical CPU, must have a cap of 25%; one that must not
use more than 1+1/2 of physical CPU time, must be given a cap
of 150%.
Caps are per domain, so it is all a domain's vCPUs, cumulatively,
that will be forced to execute no more than the decided amount.
This is implemented by giving each domain a 'budget', and
using a (per-domain again) periodic timer. Values of budget
and 'period' are chosen so that budget/period is equal to the
cap itself.
Budget is burned by the domain's vCPUs, in a similar way to
how credits are.
When a domain runs out of budget, its vCPUs can't run any
longer. They can gain, when the budget is replenishment by
the timer, which event happens once every period.
Blocking the vCPUs because of lack of budget happens by
means of a new (_VPF_parked) pause flag, so that, e.g.,
vcpu_runnable() still works. This is similar to what is
done in sched_rtds.c, as opposed to what happens in
sched_credit.c, where vcpu_pause() and vcpu_unpause()
(which means, among other things, more overhead).
Note that, while adding new fields to csched2_vcpu and
csched2_dom, currently existing members are being moved
around, to achieve best placement inside cache lines.
Note also that xenalyze and tools/xentrace/format are being
updated too.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
DEPS_INCLUDE is currently the same as DEPS, so no functional change.
This patch is the result of this perl rune:
git-grep -l 'include.*DEPS' | xargs perl -i -pe 'next unless m/^-?include/; s/\bDEPS\b/DEPS_INCLUDE/'
I have verified that I haven't missed anything, with this rune:
git-grep '\bDEPS\b'
Reported-by: Jan Beulich <JBeulich@suse.com>
CC: Wei Liu <wei.liu2@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
DEPS_RM is currently the same as DEPS, so no functional change.
This patch is the result of two perl runes:
git-grep -l 'rm.*DEPS' | xargs perl -i~ -pe 'next unless m/^\t+rm\b/; s/\bDEPS\b/DEPS_RM/;'
git-grep -l 'RM.*DEPS' | xargs perl -i~ -pe 'next unless m/^\t+\$\(RM\)/; s/\bDEPS\b/DEPS_RM/;'
And editing tools/xenstat/libxenstat/Makefile by hand.
I verified that I didn't miss anything with this rune:
git-grep '\bDEPS\b' | grep -v include |less
Reported-by: Jan Beulich <JBeulich@suse.com>
CC: Wei Liu <wei.liu2@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Soft-affinity support is usually implemented by means
of a two step "balancing loop", where:
- during the first step, we consider soft-affinity
(if the vcpu has one);
- during the second (if we get to it), we consider
hard-affinity.
In runq_tickle(), we need to do that for checking
whether we can execute the waking vCPU on an pCPU
that is idle. In fact, we want to be sure that, if
there is an idle pCPU in the vCPU's soft affinity,
we'll use it.
If there are no such idle pCPUs, though, and we
have to check non-idle ones, we can avoid the loop
and to both hard and soft-affinity in one pass.
In fact, we can we scan runqueue and compute a
"score" for each vCPU which is running on each pCPU.
The idea is, since we may have to preempt someone:
- try to make sure that the waking vCPU will run
inside its soft-affinity,
- try to preempt someone that is running outside
of its own soft-affinity.
The value of the score is added to a trace record,
so xenalyze's code and tools/xentrace/formats are
updated accordingly.
Suggested-by: George Dunlap <george.dunlap@citrix.com>
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Running "make uninstall" does not remove all installed files, a
situation which might cause link related issues if xen is re-installed
in a different location.
In order to make uninstall correctly remove the files it is best
the process should be done recursively by mirroring each "install"
target with an "uninstall" who removes the installed files.
An exception to this rule is uninstalling the files produced by
"qemu-xen-dir-remote" and "qemu-xen-traditional-dir", which are external
to the project. These projects do not implement an "uninstall" target so
the files have to be removed manually.
Signed-off-by: Petre Pircalabu <ppircalabu@bitdefender.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Including whether we actually tried stealing a vCPU from
a given pCPU, or we skipped that one, because of lock
contention.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
We record trace information about the next timeslice when
switching to a different vcpu, but not when continuing to
run the same cpu:
csched2:schedule cpu 9, rq# 1, idle, SMT idle, tickled
csched2:runq_candidate d0v3, 0 vcpus skipped, cpu 9 was tickled
sched_switch prev d32767v9, run for 991.186us
sched_switch next d0v3, was runnable for 2.515us, next slice 10000.0us
sched_switch prev d32767v9 next d0v3 ^^^^^^^^^^^^^^^^^^^^
runstate_change d32767v9 running->runnable
...
csched2:schedule cpu 2, rq# 0, busy, not tickled
csched2:burn_credits d1v5, credit = 9996950, delta = 502913
csched2:runq_candidate d1v5, 0 vcpus skipped, no cpu was tickled
runstate_continue d1v5 running->running
?????????????
This information is quite useful; so add a trace including
that information on the 'continue_running' path as well,
like this:
csched2:schedule cpu 1, rq# 0, busy, not tickled
csched2:burn_credits d0v8, credit = 9998645, delta = 12104
csched2:runq_candidate d0v8, credit = 9998645, 0 vcpus skipped, no cpu was tickled
sched_switch continue d0v8, run for 1125.820us, next slice 9998.645us
runstate_continue d0v8 running->running ^^^^^^^^^^^^^^^^^^^^^
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
When traversing a Credit2 runqueue to select the
best candidate vCPU to be run next, show in the
trace which vCPUs we consider.
A bit verbose, but quite useful, considering that
we may end up looking at, but then discarding, one
of more vCPU. This will help understand which ones
are skipped and why.
Also, add how much credits the chosen vCPU has
(in the TRC_CSCHED2_RUNQ_CANDIDATE record). And,
while there, fix a bug in tools/xentrace/formats
(still in the output of TRC_CSCHED2_RUNQ_CANDIDATE).
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
For Credit2, in both the trace records, inside Xen,
and in their parsing, in xenalyze.
In fact, as it is quite a bit better, in order to
understand how much negative credits have gone for
a certain vCPU, to see an actual negative number,
as compared to a wrapped around unsigned!
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>