Secondary vm is running in colo mode. So we will do
the following things again and again:
1. Resume secondary vm
a. Send CHECKPOINT_SVM_READY to master.
b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
c. If it is the first resume(resume right after live migration),
- call libxl__xc_domain_restore_done() to build the secondary vm.
- enable secondary vm's logdirty.
- call libxl__domain_resume() to resume secondary vm.
- call libxl__checkpoint_devices_setup() to setup checkpoint devices.
d. Send CHECKPOINT_SVM_RESUMED to master.
2. Wait a new checkpoint
a. Call libxl__checkpoint_devices_commit().
b. Read CHECKPOINT_NEW from master.
3. Suspend secondary vm
a. Suspend secondary vm.
b. Call libxl__checkpoint_devices_postsuspend().
c. Send CHECKPOINT_SVM_SUSPENDED to master.
4. Checkpoint
a. Read emulator xenstore data and emulator context
b. REC_TYPE_CHECKPOINT_END
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
In COLO mode, both VMs are running, and are considered in sync if the
visible network traffic is identical. After some time, they fall out of
sync.
At this point, the two VMs have definitely diverged. Lets call the
primary dirty bitmap set A, while the secondary dirty bitmap set B.
Sets A and B are different.
Under normal migration, the page data for set A will be sent from the
primary to the secondary.
However, the set difference B - A (the one in B but not in A, lets
call this C) is out-of-date on the secondary (with respect to the
primary) and will not be sent by the primary (to secondary), as it
was not memory dirtied by the primary. The secondary needs C page data
to reconstruct an exact copy of the primary at the checkpoint.
The secondary cannot calculate C as it doesn't know A. Instead, the
secondary must send B to the primary, at which point the primary
calculates the union of A and B (lets call this D) which is all the
pages dirtied by both the primary and the secondary, and sends all page
data covered by D.
In the general case, D is a superset of both A and B. Without the
backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
copy of the primary.
We transfer the dirty bitmap on libxc side, so we need to introduce back
channel to libxc.
Note: it is different from the paper. We change the original design to
the current one, according to our following concerns:
1. The original design needs extra memory on Secondary host. When there's
multiple backups on one host, the memory cost is high.
2. The memory cache code will be another 1k+, it will make the review
more time consuming.
Note: this patch merely adds new parameters to various prototypes and
functions. The new parameters are used in later patch called
"libxc/restore: send dirty pfn list to primary when checkpoint under
COLO".
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
We change the xen_version libxc code to use the new hypercall.
Which of course means every user in the code base has to
be changed over.
It is important to note that the xc_version_op has a different
return semantic than the previous one. It returns negative
values on error (like the old one), but it also returns
an positive value on success (unlike the old one). The positive
value is the number of bytes copied in.
Note that both Ocaml and xenstat use tabs instead of four
spaces so they look quite odd.
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Tested-by: Andrew Cooper <andrew.cooper3@citrix.com> [for the Ocaml stubs]
Acked-by: George Dunlap <george.dunlap@eu.citrix.com> [xenctx bits]
Acked-by: Wei Liu <wei.liu2@citrix.com>
xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
By having the "num" argument before the page and error arrays we can
potentially use a variable-length-array argument ("int pages[num]") in
the function prototype.
However VLAs are a C99 feature and we are currently targetting C89 and
later, so we don't actually make use of this here, merely arrange that
we can switch to VLAs in the future without changing the function ABI.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
And require it be used instead of direct munmap.
This will allow e.g. Valgrind hooks to help track incorrect use of
foreign mappings.
Switch all uses of xenforeignmemory_map to use
xenforeignmemory_unmap, not that foreign mappings via the libxc compat
xc_map_foreign_* interface will not take advantage of this and will
need converting.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
libxenforeignmemory will provide a stable API and ABI for mapping
foreign domain memory (subject to appropriate privileges).
The new library exposes an interface equivalent to
xc_map_foreign_memory_bulk, which all the other
xc_map_foreign_memory_* functions (which remain in libxc) are
implemented in terms of.
Upon request (via #define XC_WANT_COMPAT_MAP_FOREIGN_API) libxenctrl
will provide a compat API for the old names. This is used by qemu-xen
and qemu-trad as well as various in tree things (which required
de-dupping various #includes in some too to get the #define before the
first).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
[ ijc -- updated MINIOS_UPSTREAM_REVISION and QEMU_TRADITIONAL_REVISION ]
With support of the virtual mapped linear p2m list for migration it is
now possible to detect structural changes of the p2m list which before
would either lead to a crashing or otherwise wrong behaving domU.
A guest supporting the linear p2m list will increment the
p2m_generation counter located in the shared info page before and after
each modification of a mapping related to the p2m list. A change of
that counter can be detected by the tools and reacted upon.
As such a change should occur only very rarely once the domU is up the
most simple reaction is to cancel migration in such an event.
Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
For migration the last used pfn of a guest is needed to size the
logdirty bitmap and as an upper bound of the page loop. Unfortunately
there are pv-kernels advertising a much higher maximum pfn as they
are really using in order to support memory hotplug. This will lead
to allocation of much more memory in Xen tools during migration as
really needed.
Try to find the last used guest pfn of a pv-domu by scanning the p2m
tree from the last entry towards it's start and search for an entry
not being invalid.
Normally the mid pages of the p2m tree containing all invalid entries
are being reused, so we can just scan the top page for identical
entries and skip them but the first one.
Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
[ ijc -- added errno = E2BIG to one error path ]
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Rather than dirtying a page when establishing a (permanent) mapping,
dirty it when the page gets unmapped, or - if still mapped - on the
final iteration of a save operation (or in other cases where the guest
is paused or already shut down). (Transient mappings continue to get
dirtied upon getting mapped, to avoid the overhead of tracking.)
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
The prefix "Memory" is confusing because the numbers shown after that
are referring to frames.
Change a bunch of prefixes from "Memory" to "Frames". Also rename
send_memory_verify to verify_frames.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
All of these are UNUSED_VALUE defects where a default value is
unconditionally overwritten. They are not particularly interesting,
bug wise, but keeping these defects at bay helps prevent real bugs
going unnoticed in the volume.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Add further instructions to the libxc "Future Extensions" section, and
provide such a section for libxl.
In addition, drop the "In experimental __func__" IPRINTF()s from the
libxc implementations.
Finally, a correction to libxl's "Not Yet Included" section which
should have been amended in c/s 7eaec00 when libxl Remus support was
introduced into the protocol.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Originally there was only one counter to keep track of pages. It was
used erroneously to keep track of how many pages were mapped and how
many pages needed to be sent. In the end munmap(2) always had 0 as the
length argument, which resulted in leaking the mapping.
This problem was discovered on 32bit toolstack because 32bit applications
have notably smaller address space. In fact this bug affects 64bit
toolstack too.
Use a separate counter to keep track of the number of mapped pages to
solve this problem.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
The old code checks the callbacks "postcopy & checkpoint", if the
callbacks exists, it will call them. However this is unreliable, so
add this flag to explicitly indicate a checkpointed stream in the new
code. This is backward compatible with the legacy migration just don't
know this flag and will ignore it.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Yang Hongyang <yanghy@cn.fujitsu.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
[ ijc -- added discussion of legacy compat ]
There are some records which should only be sent once in the stream, and not
repeated for each checkpoint. {start,end}_of_stream() become per-checkpoint,
and a new start_of_stream() is introduced.
There is no resulting change record order, but the X86_PV_INFO record is
identified as once per stream. Currently the X86_PV_P2M_FRAMES record is as
well, but this is because of an implementation bug and can move back to being
on an as-needed basis when fixed.
In addition, a few minor adjustments of comments and layout.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Yang Hongyang <yanghy@cn.fujitsu.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
Save a domain, calling domain type specific function at the appropriate
points. This implements the xc_domain_save2() API function which is
equivalent to the existing xc_domain_save().
This writes the image and domain headers, and writes all the PAGE_DATA records
using a "live" process.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
For testing purposes, the environmental variable "XG_MIGRATION_V2" allows the
two save/restore codepaths to coexist, and have a runtime switch.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
Wei Liu
Wen Congyang
Konrad Rzeszutek Wilk
Ian Campbell
Juergen Gross
Jan Beulich
Andrew Cooper
Yang Hongyang