Commit 37f074a338 ("x86/msr: introduce guest_rdmsr()") introduced a
function to combine the MSR_READ handling between PV and HVM.
Unfortunately, by returning directly, it skipped the trace generation,
leading to gaps in the trace record, as well as xenalyze errors like
this:
hvm_generic_postprocess: d2v0 Strange, exit 7c(VMEXIT_MSR) missing a handler
Replace the `return` with `goto out`.
Fixes: 37f074a338 ("x86/msr: introduce guest_rdmsr()")
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Remain a Reviewer on the golang bindings and scheduler for now (using
a xenproject.org alias), since there may be architectural decisions I
can shed light on.
Remove the XENTRACE section entirely, as there's no obvious candidate
to take it over; having the respective parts fall back to the tools
and The Rest seems the most reasonable option.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Juergen Gross <jgross@suse.com>
Acked-by: Bertrand Marquis <bertrand.marquis@arm.com>
xenalyze sets argp_program_bug_address to my old Citrix address. This
was done before xenalyze was in the xen.git tree; and it's the only
program in the tree which does so.
Now that xenalyze is part of the normal Xen distribution, it should be
obvious where to report bugs.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-Acked-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
To unify certain common sanity checks, checks are done very early in
processing based only on the top-level type.
Unfortunately, when TRC_HVM_EMUL was introduced, it broke some of the
assumptions about how the top-level types worked. Namely, traces of
this type will show up outside of HVM contexts: in idle domains and in
PV domains.
Make an explicit exception for TRC_HVM_EMUL types in a number of places:
- Pass the record info pointer to toplevel_assert_check, so that it
can exclude TRC_HVM_EMUL records from idle and vcpu data_mode
checks
- Don't attempt to set the vcpu data_type in hvm_process for
TRC_HVM_EMUL records.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
A long-standing usability sub-optimality with xenalyze is the
necessity to specify `--svm-mode` when analyzing AMD processors. This
fundamentally comes about because the same trace event ID is used for
both VMX and SVM, but the contents of the trace must be interpreted
differently.
Instead, allocate separate trace events for VMX and SVM vmexits in
Xen; this will allow all readers to properly interpret the meaning of
the vmexit reason.
In xenalyze, first remove the redundant call to init_hvm_data();
there's no way to get to hvm_vmexit_process() without it being already
initialized by the set_vcpu_type call in hvm_process().
Replace this with set_hvm_exit_reson_data(), and move setting of
hvm->exit_reason_* into that function.
Modify hvm_process and hvm_vmexit_process to handle all four potential
values appropriately.
If SVM entries are encountered, set opt.svm_mode so that other
SVM-specific functionality is triggered.
Remove the `--svm-mode` command-line option, since it's now redundant.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
xentrace_format was always of limited utility, since trace records
across pcpus were processed out of order; it was superseded by xenalyze
over a decade ago.
But for several releases, the `formats` file it has depended on for
proper operation has not even been included in `make install` (which
generally means it doesn't get picked up by distros either); yet
nobody has seemed to complain.
Simple remove xentrace_format, and point people to xenalyze instead.
NB that there is no man page for xenalyze, so the "see also" on the
xentrace man page is simply removed for now.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Olaf Hering <olaf@aepfle.de>
Go has always interpreted a comment directly before a function as a
docstring, so having the C function prototypes which the golang method
is meant to wrap in the comment before was always a bit non-standard.
However, recent versions of `go fmt` now attempt to normalize these
docstrings as well, leading to strange changes, particularly if `go
fmt` is run on save.
Go through and put a space between non-docstring comments and methods,
so that `go fmt` leaves the comments alone.
No functional change.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Nick Rosbrook <rosbrookn@gmail.com>
In a handful of cases, it was decided to return a pointer to a
structure rather than the plain structure itself, due to the size.
However, in these cases the structure was never allocated, leading to
a nil pointer exception when calling the relevant `fromC` method.
Allocate structures before attempting to fill them in.
Fixes: 453713b175 ("golang/xenlight: Add host-related functionality")
Reported-by: Tobias Fitschen <tobias.fitschen@posteo.de>
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Tested-by: Tobias Fitschen <tobias.fitschen@posteo.de>
Acked-by: Nick Rosbrook <rosbrookn@gmail.com>
In order to make implementation and testing tractable, we will require
specific host functionality. Add a nested_virt bit to hvm_funcs.caps,
and return an error if a domain is created with nested virt and this
bit isn't set. Create VMX and SVM callbacks to be executed from
nestedhvm_setup(), which is guaranteed to execute after all
command-line options have been procesed.
For VMX, start with always enabling it if HAP is present; this
shouldn't change current behvior.
For SVM, require some basic functionality, adding a document
explaining the rationale.
NB that only SVM CPUID bits 0-7 have been considered. Bits 10-16 may
be considered in a follow-up patch.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
The primary purpose of TSC scaling, from our perspective, is to
maintain the fiction of an "invariant TSC" across migrates between
platforms with different clock speeds.
On AMD, the TscRateMSR CPUID bit is unconditionally enabled in the
"host cpuid", even if the hardware doesn't actually support it.
According to c/s fd14a1943c ("nestedsvm: Support TSC Rate MSR"),
testing showed that emulating TSC scaling in an L1 was more expensive
than emulating TSC scaling on an L0 (due to extra sets of vmexit /
vmenter).
However, the current implementation seems to be broken.
First of all, the final L2 scaling ratio should be a composition of
the L0 scaling ratio and the L1 scaling ratio; there's no indication
this is being done anywhere.
Secondly, it's not clear that the L1 tsc scaling ratio actually
affects the L0 tsc scaling ratio. The stored value (ns_tscratio) is
used to affect the tsc *offset*, but doesn't seem to actually be
factored into d->hvm.tsc_scaling_ratio. (Which shouldn't be
per-domain anyway, but per-vcpu.) Having the *offset* scaled
according to the nested scaling without the actual RDTSC itself also
being scaled has got to produce inconsistent results.
For now, just disable the functionality entirely until we can
implement it properly:
- Don't set TSCRATEMSR in the HVM max CPUID policy
- Remove MSR_AMD64_TSC_RATIO emulation handling, so that the guest
guests a #GP if it tries to access them (as it should when
TSCRATEMSR is clear)
- Remove ns_tscratio from struct nestedhvm, and all code that touches
it
Unfortunately this means ripping out the scaling calculation stuff as
well, since it's only used in the nested case; it's there in the git
tree if we need it for reference when we re-introduce it.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Currently (nested) SVM features we're willing to expose to the guest
are defined in calculate_host_policy, and stored in host_cpu_policy.
This is the wrong place for this; move it into
calculate_hvm_max_policy(), and store it in hvm_max_cpu_policy.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Changeset ef3e8db806 ("x86/hvm: Corrections and improvements to
unhandled vmexit logging") introduced a printk to the default path of
the switch statement in nestedsvm_check_intercepts(), complaining of
an unknown exit reason.
Unfortunately, the "core" switch statement which is meant to handle
all vmexit reasons is in nsvm_vmcb_guest_intercepts_exitcode(); the
switch statement in nestedsvm_check_intercepts() is only meant to
superimpose on top of that some special-casing for how to interaction
between L1 and L0 vmexits.
Remove the printk, and add a comment to prevent future confusion.
Fixes: ef3e8db806 ("x86/hvm: Corrections and improvements to unhandled vmexit logging")
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Moving them all together has several advantages:
* Collects them all in one part of the struct
* The `caps` field means that we can drop the "_supported" suffix, as it's
clear what is meant.
While here, replace some 0/1 values with false/true.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
The "effective type" of the cpu_has_svm_feature macro is effectively
an unsigned log with one bit set (or not); at least one place someone
felt compelled to do a !! to make sure that they got a boolean out of
it.
Ideally the whole of this would be folded into the cpufeature.h
infrastructure. But for now, duplicate the more type-safe static
inlines in that file, and remove the !!.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
hvm_function_table is an internal structure; rather than manually
|-ing and &-ing bits, just make it a boolean bitfield and let the
compiler do all the work. This makes everything easier to read, and
presumably allows the compiler more flexibility in producing efficient
code.
No functional change intended.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Commit fc2b57c9a ("xenstored: send an evtchn notification on
introduce_domain") introduced the sending of an event channel to the
guest when first introduced, so that dom0less domains waiting for the
connection would know that xenstore was ready to use.
Unfortunately, it was introduced in introduce_domain(), which 1) is
called by other functions, where such functionality is unneeded, and
2) after the main XS_INTRODUCE call, calls domain_conn_reset(). This
introduces a race condition, whereby if xenstored is delayed, a domain
can wake up, send messages to the buffer, only to have them deleted by
xenstore before finishing its processing of the XS_INTRODUCE message.
Move the connect-and-notfy call into do_introduce() instead, after the
domain_conn_rest(); predicated on the state being in the
XENSTORE_RECONNECT state.
(We don't need to check for "restoring", since that value is always
passed as "false" from do_domain_introduce()).
Also take the opportunity to add a missing wmb barrier after resetting
the indexes of the ring in domain_conn_reset.
This change will also remove an extra event channel notification for
dom0 (because the notification is now done by do_introduce which is not
called for dom0.) The extra dom0 event channel notification was only
introduced by fc2b57c9a and was never present before. It is not needed
because dom0 is the one to tell xenstored the connection parameters, so
dom0 has to know that the ring page is setup correctly by the time
xenstored starts looking at it. It is dom0 that performs the ring page
init.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
CC: jgross@suse.com
CC: julien@xen.org
CC: wl@xen.org
We warn when we see data for a vcpu moving into a non-RUNNING state,
just so that people know why we're ignoring it. On full traces, this
happens only once. However, if the trace was limited to a subset of
pcpus, then this will happen every time the domain in question is
woken on that pcpu.
Add a 'delayed_init' flag to the vcpu struct to indicate when a vcpu
has experienced a delayed init. Print a warning message once when
entering the state, and once when leaving it.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
EIP lists are generalized across several use cases. For many of them,
it make sense to have a cycle per sample; but not really for interrupt
EIP lists. For this reason, it normally just passes 0 as for the tsc
value, which will in turn down at the bottom of update_cycles(),
update only the summary.event_count, but nothing else.
The dump_eip() function attempted to handle this by calling the generic
cycle print handler if the summary contained *any* cycles, and by collecting
and printing its own stats, based solely on counts, if not.
Unfortunately, it used the wrong element for this: it collected the
total from samples.count rather samples.event_count; in the case that
there are no cycles, this will always be zero. It then divided by
this zero value. This results in output that looked like this:
```
ffff89d29656 : 0 -nan%
ffff89d298b6 : 0 -nan%
ffff89d298c0 : 0 -nan%
```
It's better than nothing, but a lot less informative than one would
like.
Use event_count rather than count for collecting the total, and the
reporting when there are no cycles in the summary information. This results
in output that looks like this:
```
ffff89d29656 : 2 1.21%
ffff89d298b6 : 1 0.61%
ffff89d298c0 : 1 0.61%
```
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
If a vmentry/exit arc unexpectedly doesn't have a handler, we throw an
error, and then log the information under HVM event 0; thus those
particular traces within the vmexit reason will have stats gathered,
and will show up with "(no handler)". This is useful in the event
that there are unusual paths through the hypervisor which don't have
trace points.
However, if there are more than one of these, then although we detect and warn
that this is happening, we subsequently continue to stuff data about all exits
into that one exit, even though we only show it in one place.
Instead, refator things to only allow a single exit reason to be
accumulated into any given event.
Also put a comment explaining what's going on, and how to fix it.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Following a conversation with Bob Eschleman, it was agreed that
Bobby would prefer to return to being a Reviewer.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
Acked-by: Julien Grall <jgrall@amazon.com>
On large systems with many vcpus yielding due to spinlock priority
inversion, it's not uncommon for a vcpu to yield its timeslice, only
to be immediately stolen by another pcpu looking for higher-priority
work.
To prevent this:
* Keep the YIELD flag until a vcpu is removed from a runqueue
* When looking for work to steal, skip vcpus which have yielded
NB that this does mean that sometimes a VM is inserted into an empty
runqueue; handle that case.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
The credit scheduler tries as hard as it can to ensure that it always
runs scheduling units with positive credit (PRI_TS_UNDER) before
running those with negative credit (PRI_TS_OVER). If the next
runnable scheduling unit is of priority OVER, it will always run the
load balancer, which will scour the system looking for another
scheduling unit of the UNDER priority.
Unfortunately, as the number of cores on a system has grown, the cost
of the work-stealing algorithm has dramatically increased; a recent
trace on a system with 128 cores showed this taking over 50
microseconds.
Add a parameter, load_balance_ratelimit, to limit the frequency of
load balance operations on a given pcpu. Default this to 1
millisecond.
Invert the load balancing conditional to make it more clear, and line
up more closely with the comment above it.
Overall it might be cleaner to have the last_load_balance checking
happen inside csched_load_balance(), but that would require either
passing both now and spc into the function, or looking them up again;
both of which seemed to be worse than simply checking and setting the
values before calling it.
On a system with a vcpu:pcpu ratio of 2:1, running Windows guests
(which will end up calling YIELD during spinlock contention), this
patch increased performance significantly.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Release-acked-by: Henry Wang <Henry.Wang@arm.com>
A recent xentrace highlighted an unhandled corner case in the vcpu
"start-of-day" logic, if the trace starts after the last running ->
non-running transition, but before the first non-running -> running
transition. Because start-of-day wasn't handled, vcpu_next_update()
was expecting p->current to be NULL, and tripping out with the
following error message when it wasn't:
vcpu_next_update: FATAL: p->current not NULL! (d32768dv$p, runstate RUNSTATE_INIT)
where 32768 is the DEFAULT_DOMAIN, and $p is the pcpu number.
Instead of calling vcpu_start() piecemeal throughout
sched_runstate_process(), call it at the top of the function if the
vcpu in question is still in RUNSTATE_INIT, so that we can handle all
the cases in one place.
Sketch out at the top of the function all cases which we need to
handle, and what to do in those cases. Some transitions tell us where
v is running; some transitions tell us about what is (or is not)
running on p; some transitions tell us neither.
If a transition tells us where v is now running, update its state;
otherwise leave it in INIT, in order to avoid having to deal with TSC
skew on start-up.
If a transition tells us what is or is not running on p, update
p->current (either to v or NULL). Otherwise leave it alone.
If neither, do nothing.
Reifying those rules:
- If we're continuing to run, set v to RUNNING, and use p->first_tsc
as the runstate time.
- If we're starting to run, set v to RUNNING, and use ri->tsc as the
runstate time.
- If v is being deschedled, leave v in the INIT state to avoid dealing
with TSC skew; but set p->current to NULL so that whatever is
scheduled next won't trigger the assert in vcpu_next_update().
- If a vcpu is waking up (switching from one non-runnable state to
another non-runnable state), leave v in INIT, and p in whatever
state it's in (which may be the default domain, or some other vcpu
which has already run).
While here, fix the comment above vcpu_start; it's called when the
vcpu state is INIT, not when current is the default domain.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Anthony PERARD <anthony.perard@citrix.com>
The output file is optional. In case it is missing, xentrace is supposed
to write to stdout - unless it is a tty, which is checked prior using it.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: George Dunlap <george.dunlap@cloud.com>
There was a question raised recently about the requirements for
checking in a patch which was originally written by one maintainer,
then picked up and modified by a second maintainer, and which they now both
agree should be checked in.
It was proposed that in that case, the following set of tags would suffice:
Signed-off-by: First Author <...>
Signed-off-by: Second Author <...>
Reviewed-by: First Author <...>
The rationale was as follows:
1. The patch will be a mix of code, whose copyright is owned by the
various authors (or the companies they work for). It's important to
keep this information around in the event, for instance, of a license
change or something else requiring knowledge of the copyright owner.
2. The Signed-off-by of the Second Author approves not only their own
code, but First Author's code; the Reviewed-by of the First Author
approves not only their own code, but the Second Author's code. Thus
all the code has been approved by a maintainer, as well as someone who
was not the author.
In support of this, several arguments were put forward:
* We shouldn't make it harder for maintainers to get their code in
than for non-maintainers
* The system we set up should not add pointless bureaucracy; nor
discourage collaboration; nor encourage contributors to get around
the rules by dropping important information. (For instance, by
removing the first SoB, so that the patch appears to have been
written entirely by Second Author.)
Concerns were raised about two maintainers from the same company
colluding to get a patch in from their company; but such maintainers
could already collude, by working on the patch in secret, and posting
it publicly with only a single author's SoB, and having the other
person review it.
There's also something slightly strange about adding "Reviewed-by" to
code that you've written; but in the end you're reviewing not only the
code itself, but the final arrangement of it. There's no need to
overcomplicate things.
Encode this in MAINTAINERS as follows:
* Refine the wording of requirement #2 in the check-in policy; such
that *each change* must have approval from someone other than *the
person who wrote it*.
* Add a paragraph explicitly stating that the multiple-SoB-approval
system satisfies the requirements, and why.
Signed-off-by: George Dunlap <george.dunlap@cloud.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Julien Grall <jgrall@amazon.com>
George Dunlap