Update the libxc spec to indicate more sternly that TOOLSTACK records
should no longer be used.
Also, trim further toolstack infrastructure which should have gone in
c/s 39bf4e9 "tools/libxl: Drop all knowledge of toolstack callbacks"
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Currently, libxl__xc_domain_{save,restore}() have specific knowledge
of where the libxl__save_helper_state lives inside a
libxl__domain_{create,save}_state object.
In later changes, the logical ownership of the
libxl__save_helper_state will change and will no longer be
d{c,s}s->shs.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v3
With migration v2, there are several moving parts needing to be
juggled at once. This requires the error handling logic to be able to
query the state of each moving part, possibly before they have been
started, and be able to cancel them.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Don't force _init() to be mandatory
v3: Adjust helper_{stop,failed,done} to use libxl__save_helper_inuse()
v2: Add an _init() function which allows _inuse() to be safe to call even
before the save helper has started.
In a future patch, we shall support automatically converting a legacy
stream to a v2 stream, in which case libxc needs to read from a
different fd.
Simply overwriting restore_fd does not work; the two fd's have
different circumstances. The restore_fd needs to be returned to its
original state before libxl_domain_create_restore() returns, while in
the converted case, the fd needs allocating and deallocating
appropriately.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v2
Shortly more parameters will appear, and this saves unboxing each one.
libxl_domain_restore_params is mandatory for restore streams, and
ignored for plain creation. The old 'checkpointed_stream' was
incorrectly identified as a private parameter when it was infact
public.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
Reviewed-by: Yang Hongyang <yanghy@cn.fujitsu.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Since v1:
* Gate validity on restore_fd being valid.
as a wrapper to kill(2), and use it in preference to sendsig in
libxl_save_callout.c.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v3: Fix typo, add _hidden
Logically new in v2 - split out from a v1 change which was itself a
cherrypick-and-modify from the AO Abort series
If we already have an rc (eg from ao abort), keep it.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v4: New patch in this version of the series.
Register the the save/restore helper interface with the abort
machinery. When we are informed that save/restore should be aborted,
we make a note of the that in our rc variable, and send the helper a
SIGTERM. It will die in due course.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New in this version of the series.
Two unrelated non-functional changes, broken out into a pre-patch for
easier review:
Break out a function sendsig() in libxl_save_callout.c.
Move io_fd to be a global variable in libxl_save_helper.c.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New in this version of the series.
The current usage of LIBEXEC is bogus. It describes the directory for
private xen executables. Other places create their own, similar
libexecdir path as $prefix/lib/xen/*.
Additional two other variables are used to describe similar paths:
PRIVATE_BINDIR and PRIVATE_PREFIX
The autoconf documentation refers to libexec as a directory for
executables and stuff which is called by other programs, not by the
user.
Adjust all places that want libexecdir as a target path. LIBEXEC refers
now to the base directory. Three convenience variables are used to refer
to paths to private binaries, libs and include files.
In the systemd files LIBEXEC_BIN is substituted, so this variable has to
be present in autoconf. All other variables are expanded in Paths.mk
because they are only used in Makefiles.
Most users of LIBEXEC are updated to use LIBEXEC_BIN because that is
what they want.
Users of PRIVATE_BINDIR are updated to use LIBEXEC_BIN because that is
what they want. PRIVATE_BINDIR and PRIVATE_PREFIX usage is removed by
this patch, in favour of LIBXEC_BIN and LIBEXEC
An internal libxl function was removed. A single helper to retrieve
LIBEXEC_BIN remains.
As suggested by the autoconf documentation, configure appends the
package name to LIBEXEC to make sure the provided directory really
refers to xen. This makes sure "make uninstall" preserves the real
libexecdir.
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Samuel Thibault <samuel.thibault@ens-lyon.org>
Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
[ ijc -- ran autogen.sh as requested, updated QEMU_TRADITIONAL_REVISION to
pickup version which uses LIBEXEC_BIN ]
The VM generation ID support in libxc/libxl was based on a draft
specification which subsequently changed considerably. Remove much of
the code in anticipation of introducing something simpler that
conforms to the current specification from Microsoft.
Switch to using a HVM param (HVM_PARAM_VM_GENERATION_ID_ADDR) instead
of the hvmloader/generation-id-address XenStore key. This simplifies
save/restore since it only needs to transfer the value of this param.
There are no changes to the migration stream format or the public
libxl API.
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
[ ijc -- updated xc_nomigrate.c for interface change ]
toolstack_save data is written to a temporary file in libxl and read
back in libxl-save-helper. The file position must be reset prior to
reading the file, which is done in libxl-save-helper with lseek.
lseek is unsupported for pipes and sockets, so a wrapper passing such an
fd to libxl-save-helper fails the lseek. Moving the lseek to libxl
avoids the error, allowing the save to continue.
Signed-off-by: Jason Andryuk <andryuk@aero.org>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
If libxl_pipe fails we leave the carefd locked, which translates to
the atfork lock remaining held. This would probably cause the process
to deadlock shortly afterwards.
Of course libxl_pipe is very unlikely to fail unless things are
already going very badly. This bug has not been observed anywhere as
far as we are aware.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: George Dunlap <george.dunlap@eu.citrix.com>
Commit 00a4b65f85 Sep 7 2010
"libxc: provide notification of final checkpoint to restore end"
broke migration from any version of Xen using tools from prior to that commit
Older tools have no idea about an XC_SAVE_ID_LAST_CHECKPOINT, causing newer
tools xc_domain_restore() to start reading the qemu save record, as
ctx->last_checkpoint is 0.
The failure looks like:
xc: error: Max batch size exceeded (1970103633). Giving up.
where 1970103633 = 0x756d6551 = *(uint32_t*)"Qemu"
With this fix in place, the behaviour for normal migrations is reverted to how
it was before the regression; the migration is considered non-checkpointed
right from the start. A XC_SAVE_ID_LAST_CHECKPOINT chunk seen in the
migration stream is a nop. For checkpointed migrations the behaviour is
unchanged.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Shriram Rajagopalan <rshriram@cs.ubc.ca> (Remus bits)
in current xen x86_64, the default libexec directory is /usr/lib/xen/bin,
while the private binder is /usr/lib64/xen/bin. but some commands(pygrub,
libxl-save-helper) located in private binder directory is called from
libexec directory which lead to the following error:
1, for pygrub bootloader:
libxl: debug: libxl_bootloader.c:429:bootloader_disk_attached_cb: /usr/lib/xen/bin/pygrub doesn't exist, falling back to config path
2, for libxl-save-helper:
libxl: cannot execute /usr/lib/xen/bin/libxl-save-helper: No such file or directory
libxl: error: libxl_utils.c:363:libxl_read_exactly: file/stream truncated reading ipc msg header from domain 3 save/restore helper stdout pipe
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: domain 3 save/restore helper [10222] exited with error status 255
there are two ways to fix above error. the first one is make such command
store in the /usr/lib/xen/bin and /usr/lib64/xen/bin(symbol link to
previous), e.g. qemu-dm. The second way is using private binder dir
instead of libexec dir. e.g. xenconsole.
For these cases, the latter one is suitable.
Signed-off-by: Bamvor Jian Zhang <bjzhang@suse.com>
Committed-by: Ian Campbell <ian.campbell@citrix.com>
The current migration code in libxl instructs qemu to start or stop
logdirty, but it does not wait for an acknowledgement from qemu before
continuing. This might lead to memory corruption (!)
Fix this by waiting for qemu to acknowledge the command.
Unfortunately the necessary ao arrangements for waiting for this
command are unique because qemu has a special protocol for this
particular operation.
Also, this change means that the switch_qemu_logdirty callback
implementation in libxl can no longer synchronously produce its return
value, as it now needs to wait for xenstore. So we tell the
marshalling code generator that it is a message which does not need a
reply. This turns the callback function called by the marshaller into
one which returns void; the callback function arranges to later
explicitly sends the reply to the helper, when the xs watch triggers
and the appropriate value is read from xenstore.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
libxenctrl expects to be able to simply run the save or restore
operation synchronously. This won't work well in a process which is
trying to handle multiple domains.
The options are:
- Block such a whole process (eg, the whole of libvirt) while
migration completes (or until it fails).
- Create a thread to run xc_domain_save and xc_domain_restore on.
This is quite unpalatable. Multithreaded programming is error
prone enough without generating threads in libraries, particularly
if the thread does some very complex operation.
- Fork and run the operation in the child without execing. This is
no good because we would need to negotiate with the caller about
fds we would inherit (and we might be a very large process).
- Fork and exec a helper.
Of these options the latter is the most palatable.
Consequently:
* A new helper program libxl-save-helper (which does both save and
restore). It will be installed in /usr/lib/xen/bin. It does not
link against libxl, only libxc, and its error handling does not
need to be very advanced. It does contain a plumbing through of
the logging interface into the callback stream.
* A small ad-hoc protocol between the helper and libxl which allows
log messages and the libxc callbacks to be passed up and down.
Protocol doc comment is in libxl_save_helper.c.
* To avoid a lot of tedium the marshalling boilerplate (stubs for the
helper and the callback decoder for libxl) is generated with a
small perl script.
* Implement new functionality to spawn the helper, monitor its
output, provide responses, and check on its exit status.
* The functions libxl__xc_domain_restore_done and
libxl__xc_domain_save_done now turn out to want be called in the
same place. So make their state argument a void* so that the two
functions are type compatible.
The domain save path still writes the qemu savefile synchronously.
This will need to be fixed in a subsequent patch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Change the internal and external APIs for domain save (suspend) to be
capable of asynchronous operation. The implementation remains
synchronous. The interfaces surrounding device model saving are still
synchronous.
Public API changes:
* libxl_domain_save takes an ao_how.
* libxl_domain_remus_start takes an ao_how. If the
libxl_domain_remus_info is NULL, we abort rather than returning an
error.
* The `suspend_callback' function passed to libxl_domain_save is
never called by the existing implementation in libxl. Abolish it.
* libxl_domain_save takes its flags parameter as an argument.
Thus libxl_domain_suspend_info is abolished.
* XL_SUSPEND_* flags renamed to LIBXL_SAVE_*.
* Callers in xl updated.
Internal code restructuring:
* libxl__domain_suspend_state member types and names rationalised.
* libxl__domain_suspend renamed from libxl__domain_suspend_common.
(_common here actually meant "internal function").
* libxl__domain_suspend takes a libxl__domain_suspend_state, which
where the parameters to the operation are filled in by the caller.
* xc_domain_save is now called via libxl__xc_domain_save which can
itself become asynchronous.
* Consequently, libxl__domain_suspend is split into two functions at
the callback boundary; the second half is
libxl__xc_domain_save_done.
* libxl__domain_save_device_model is now called by the actual
implementation rather than by the public wrapper. It is already in
its proper place in the domain save execution sequence. So
officially make it part of that execution sequence, renaming it to
domain_save_device_model.
* Effectively, rewrite the public wrapper functions
libxl_domain_suspend and libxl_domain_remus_start.
* Remove a needless #include <xenctrl.h>
* libxl__domain_suspend aborts on unexpected domain types rather
than mysteriously returning EINVAL.
* struct save_callbacks moved from the stack to the dss.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
We are going to arrange that libxl, instead of calling
xc_domain_restore, calls a stub function which forks and execs a
helper program, so that restore can be asynchronous rather than
blocking the whole toolstack.
This stub function will be called libxl__xc_domain_restore.
However, its prospective call site is unsuitable for a function which
needs to make a callback, and is buried in two nested single-call-site
functions which are logically part of the domain creation procedure.
So we first abolish those single-call-site functions, integrate their
contents into domain creation in their proper temporal order, and
break out libxl__xc_domain_restore ready for its reimplementation.
No functional change - just the following reorganisation:
* Abolish libxl__domain_restore_common, as it had only one caller.
Move its contents into (what was) domain_restore.
* There is a new stage function domcreate_rebuild_done containing what
used to be the bulk of domcreate_bootloader_done, since
domcreate_bootloader_done now simply starts the restore (or does the
rebuild) and arranges to call the next stage.
* Move the contents of domain_restore into its correct place in the
domain creation sequence. We put it inside
domcreate_bootloader_done, which now either calls
libxl__xc_domain_restore which will call the new function
domcreate_rebuild_done, or calls domcreate_rebuild_done directly.
* Various general-purpose local variables (`i' etc.) and convenience
alias variables need to be shuffled about accordingly.
* Consequently libxl__toolstack_restore needs to gain external linkage
as it is now in a different file to its user.
* Move the xc_domain_save callbacks struct from the stack into
libxl__domain_create_state.
In general the moved code remains almost identical. Two returns in
what used to be libxl__domain_restore_common have been changed to set
the return value and "goto out", and the call sites for the abolished
and new functions have been adjusted.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Andrew Cooper
Ian Jackson
Olaf Hering
David Vrabel
Jason Andryuk
Bamvor Jian Zhang