xen/docs/misc/xenstore.txt

Xenstore protocol specification
-------------------------------

Xenstore implements a database which maps filename-like pathnames
(also known as `keys') to values.  Clients may read and write values,
watch for changes, and set permissions to allow or deny access.  There
is a rudimentary transaction system.

While xenstore and most tools and APIs are capable of dealing with
arbitrary binary data as values, this should generally be avoided.
Data should generally be human-readable for ease of management and
debugging; xenstore is not a high-performance facility and should be
used only for small amounts of control plane data.  Therefore xenstore
values should normally be 7-bit ASCII text strings containing bytes
0x20..0x7f only, and should not contain a trailing nul byte.  (The
APIs used for accessing xenstore generally add a nul when reading, for
the caller's convenience.)

A separate specification will detail the keys and values which are
used in the Xen system and what their meanings are.  (Sadly that
specification currently exists only in multiple out-of-date versions.)


Paths are /-separated and start with a /, just as Unix filenames.

We can speak of two paths being <child> and <parent>, which is the
case if they're identical, or if <parent> is /, or if <parent>/ is an
initial substring of <child>.  (This includes <path> being a child of
itself.)

If a particular path exists, all of its parents do too.  Every
existing path maps to a possibly empty value, and may also have zero
or more immediate children.  There is thus no particular distinction
between directories and leaf nodes.  However, it is conventional not
to store nonempty values at nodes which also have children.

The permitted character for paths set is ASCII alphanumerics and plus
the four punctuation characters -/_@ (hyphen slash underscore atsign).
@ should be avoided except to specify special watches (see below).
Doubled slashes and trailing slashes (except to specify the root) are
forbidden.  The empty path is also forbidden.  Paths longer than 3072
bytes are forbidden; clients specifying relative paths should keep
them to within 2048 bytes.  (See XENSTORE_*_PATH_MAX in xs_wire.h.)


Each node has one or multiple permission entries.  Permissions are
granted by domain-id, the first permission entry of each node specifies
the owner of the node, who always has full access to the node (read and
write permission).  The access rights of the first entry specify the
allowed access for all domains not having a dedicated permission entry
(the default is "n", removing access for all domains not explicitly
added via additional permission entries).  Permissions of a node can be
changed by the owner of the node, the owner can only be modified by the
control domain (usually domain id 0).  Other permissions can be setup to
allow read and/or write access for other domains.  When a domain is
being removed from Xenstore nodes owned by that domain will be removed
together with all of those nodes' children.


Communication with xenstore is via either sockets, or event channel
and shared memory, as specified in io/xs_wire.h: each message in
either direction is a header formatted as a struct xsd_sockmsg
followed by xsd_sockmsg.len bytes of payload.

The payload syntax varies according to the type field.  Generally
requests each generate a reply with an identical type, req_id and
tx_id.  However, if an error occurs, a reply will be returned with
type ERROR, and only req_id and tx_id copied from the request.

A caller who sends several requests may receive the replies in any
order and must use req_id (and tx_id, if applicable) to match up
replies to requests.  (The current implementation always replies to
requests in the order received but this should not be relied on.)

The payload length (len field of the header) is limited to 4096
(XENSTORE_PAYLOAD_MAX) in both directions.  If a client exceeds the
limit, its xenstored connection will be immediately killed by
xenstored, which is usually catastrophic from the client's point of
view.  Clients (particularly domains, which cannot just reconnect)
should avoid this.

Existing clients do not always contain defences against overly long
payloads.  Increasing xenstored's limit is therefore difficult; it
would require negotiation with the client, and obviously would make
parts of xenstore inaccessible to some clients.  In any case passing
bulk data through xenstore is not recommended as the performance
properties are poor.

---------- Defined Xenstore message types ----------

Below is a table with all defined Xenstore message types (type name
and its associated numerical value).

Some types are optional to be supported by a specific Xenstore
implementation.  If an optional type is not supported by a Xenstore
implementation, Xen tools will continue to work, maybe with slightly
reduced functionality.  A mandatory type not being supported will
result in severely reduced functionality, like inability to create
domains.  In case a type is optional, this is stated in the table with
the lost functionality in case Xenstore doesn't support that type.
Any not supported type sent to Xenstore will result in an error response
with the "ENOSYS" error.

CONTROL               0    optional
    If not supported, xenstore-control command will not work.
    DEBUG is a deprecated alias of CONTROL.
DIRECTORY             1
READ                  2
GET_PERMS             3
WATCH                 4
UNWATCH               5
TRANSACTION_START     6
TRANSACTION_END       7
INTRODUCE             8
RELEASE               9
GET_DOMAIN_PATH      10
WRITE                11
MKDIR                12
RM                   13
SET_PERMS            14
WATCH_EVENT          15
    Not valid in client sent messages.
    Only valid in Xenstore replies.
ERROR                16
    Not valid in client sent messages.
    Only valid in Xenstore replies.
IS_DOMAIN_INTRODUCED 17
RESUME               18
SET_TARGET           19
RESTRICT             20    no longer supported
    RESTRICT has been removed, the type value 20 is invalid.
RESET_WATCHES        21
DIRECTORY_PART       22    optional
    If not supported, the output of xenstore-ls might be incomplete
    with a node's sub-node list exceeding the maximum payload size
    (e.g. the "/local/domain" node with more than ca. 1000 domains
    active).
GET_FEATURE          23    optional
SET_FEATURE          24    optional
    SET_FEATURE requires GET_FEATURE to be supported.
    If unsupported, setting availability of Xenstore features per
    domain is not possible.
GET_QUOTA            25    optional
SET_QUOTA            26    optional
    SET_QUOTA requires GET_QUOTA to be supported.
    If unsupported, setting of Xenstore quota per domain is not
    possible.
INVALID           65535
    Guaranteed invalid type (never supported).

---------- Xenstore protocol details - introduction ----------

The payload syntax and semantics of the requests and replies are
described below.  In the payload syntax specifications we use the
following notations:

 |		A nul (zero) byte.
 <foo>		A string guaranteed not to contain any nul bytes.
 <foo|>		Binary data (which may contain zero or more nul bytes)
 <foo>|*	Zero or more strings each followed by a trailing nul
 <foo>|+	One or more strings each followed by a trailing nul
 ?		Reserved value (may not contain nuls)
 ??		Reserved value (may contain nuls)

Except as otherwise noted, reserved values are believed to be sent as
empty strings by all current clients.  Clients should not send
nonempty strings for reserved values; those parts of the protocol may
be used for extension in the future.


Error replies are as follows:

ERROR						E<something>|
	Where E<something> is the name of an errno value
	listed in io/xs_wire.h.  Note that the string name
	is transmitted, not a numeric value.


Where no reply payload format is specified below, success responses
have the following payload:
						OK|

Values commonly included in payloads include:

    <path>
	Specifies a path in the hierarchical key structure.
	If <path> starts with a / it simply represents that path.

	<path> is allowed not to start with /, in which case the
	caller must be a domain (rather than connected via a socket)
	and the path is taken to be relative to /local/domain/<domid>
	(eg, `x/y' sent by domain 3 would mean `/local/domain/3/x/y').

    <domid>
	Integer domid, represented as decimal number 0..65535.
	Parsing errors and values out of range generally go
	undetected.  The special DOMID_... values (see xen.h) are
	represented as integers; unless otherwise specified it
	is an error not to specify a real domain id.



The following are the actual type values, including the request and
reply payloads as applicable:


---------- Database read, write and permissions operations ----------

READ			<path>|			<value|>
WRITE			<path>|<value|>
	Store and read the octet string <value> at <path>.
	WRITE creates any missing parent paths, with empty values.

MKDIR			<path>|
	Ensures that the <path> exists, by necessary by creating
	it and any missing parents with empty values.  If <path>
	or any parent already exists, its value is left unchanged.

RM			<path>|
	Ensures that the <path> does not exist, by deleting
	it and all of its children.  It is not an error if <path> does
	not exist, but it _is_ an error if <path>'s immediate parent
	does not exist either.

DIRECTORY		<path>|			<child-leaf-name>|*
	Gives a list of the immediate children of <path>, as only the
	leafnames.  The resulting children are each named
	<path>/<child-leaf-name>.

DIRECTORY_PART		<path>|<offset>		<gencnt>|<child-leaf-name>|*
	Same as DIRECTORY, but to be used for children lists longer than
	XENSTORE_PAYLOAD_MAX. Input are <path> and the byte offset into
	the list of children to return. Return values are the generation
	count <gencnt> of the node (to be used to ensure the node hasn't
	changed between two reads: <gencnt> being the same for multiple
	reads guarantees the node hasn't changed) and the list of children
	starting at the specified <offset> of the complete list.

GET_PERMS	 	<path>|			<perm-as-string>|+
SET_PERMS		<path>|<perm-as-string>|+?
	<perm-as-string> is one of the following
		w<domid>	write only
		r<domid>	read only
		b<domid>	both read and write
		n<domid>	no access
	See https://wiki.xen.org/wiki/XenBus section
	`Permissions' for details of the permissions system.
	It is possible to set permissions for the special watch paths
	"@introduceDomain" and "@releaseDomain" to enable receiving those
	watches in unprivileged domains.

---------- Watches ----------

WATCH			<wpath>|<token>|[<depth>|]?
	Adds a watch.

	When a <path> is modified (including path creation, removal,
	contents change or permissions change) this generates an event
	on the changed <path>.  Changes made in transactions cause an
	event only if and when committed.  Each occurring event is
	matched against all the watches currently set up, and each
	matching watch results in a WATCH_EVENT message (see below).

	The event's path matches the watch's <wpath> if it is an child
	of <wpath>. This match can be limited by specifying <depth> (a
	decimal value of 0 or larger): it denotes the directory levels
	below <wpath> to consider for a match ("0" would not match for
	a child of <wpath>, "1" would match only for a direct child,
	etc.).  The depth specification is possible only, if the
	XENSTORE_SERVER_FEATURE_WATCHDEPTH feature is available.

	<wpath> can be a <path> to watch or @<wspecial>.  In the
	latter case <wspecial> may have any syntax but it matches
	(according to the rules above) only the following special
	events which are invented by xenstored:
	    @introduceDomain	occurs on INTRODUCE
	    @releaseDomain 	occurs on any domain crash or
				shutdown, and also on RELEASE
				and domain destruction
	<wspecial> events are sent to privileged callers or explicitly
	via SET_PERMS enabled domains only. The semantics for a
	specification of <depth> differ for generating <wspecial>
	events: specifying "1" will report the related domid by using
	@<wspecial>/<domid> for the reported path. Other <depth>
	values are not supported.
	For @releaseDomain it is possible to watch only for a specific
	domain by specifying @releaseDomain/<domid> for the path.
	This requires the XENSTORE_SERVER_FEATURE_WATCHDEPTH to be
	available.

	When a watch is first set up it is triggered once straight
	away, with <path> equal to <wpath>.  Watches may be triggered
	spuriously.  The tx_id in a WATCH request is ignored.

	Watches are supposed to be restricted by the permissions
	system but in practice the implementation is imperfect.
	Applications should not rely on being sent a notification for
	paths that they cannot read; however, an application may rely
	on being sent a watch when a path which it _is_ able to read
	is deleted even if that leaves only a nonexistent unreadable
	parent.  A notification may omitted if a node's permissions
	are changed so as to make it unreadable, in which case future
	notifications may be suppressed (and if the node is later made
	readable, some notifications may have been lost).

WATCH_EVENT					<epath>|<token>|
	Unsolicited `reply' generated for matching modification events
	as described above.  req_id and tx_id are both 0.

	<epath> is the event's path, ie the actual path that was
	modified; however if the event was the recursive removal of an
	parent of <wpath>, <epath> is just
	<wpath> (rather than the actual path which was removed).  So
	<epath> is a child of <wpath>, regardless.

	Iff <wpath> for the watch was specified as a relative pathname,
	the <epath> path will also be relative (with the same base,
	obviously).

UNWATCH			<wpath>|<token>|?

RESET_WATCHES		|
	Reset all watches and transactions of the caller.

---------- Transactions ----------

TRANSACTION_START	|			<transid>|
	<transid> is an opaque uint32_t allocated by xenstored
	represented as unsigned decimal.  After this, transaction may
	be referenced by using <transid> (as 32-bit binary) in the
	tx_id request header field.  When transaction is started whole
	db is copied; reads and writes happen on the copy.
	It is not legal to send non-0 tx_id in TRANSACTION_START.

TRANSACTION_END		T|
TRANSACTION_END		F|
	tx_id must refer to existing transaction.  After this
 	request the tx_id is no longer valid and may be reused by
	xenstore.  If F, the transaction is discarded.  If T,
	it is committed: if there were any other intervening writes
	then our END gets get EAGAIN.

	The plan is that in the future only intervening `conflicting'
	writes cause EAGAIN, meaning only writes or other commits
	which changed paths which were read or written in the
	transaction at hand.

---------- Domain management and xenstored communications ----------

INTRODUCE		<domid>|<gfn>|<evtchn>|?
	Notifies xenstored to communicate with this domain.

	INTRODUCE is currently only used by xen tools (during domain
	startup and various forms of restore and resume), and
	xenstored prevents its use other than by dom0.

	<domid> must be a real domain id (not 0 and not a special
	DOMID_... value).  <gfn> must be a page in that domain
	represented in signed decimal (!).  <evtchn> is an unbound
	event channel in <domid> (likewise in decimal), on which
	xenstored will call bind_interdomain.

	Violations of these rules may result in undefined behaviour;
	for example passing a high-bit-set 32-bit gfn as an unsigned
	decimal will attempt to use 0x7fffffff instead (!).

	The <gfn> field is used by xenstoreds which use foreign
	mapping to access the ring page.

	Alternatively, Grant 1 (GNTTAB_RESERVED_XENSTORE) is reserved
	for the same purpose, and is populated by the domain builder
	on behalf of the guest.  This mechanism is preferred because
	it reduces the permissions that xenstored needs in order to
	function.

	Both <gfn> and Grant 1 need to agree, because implementations
	of xenstored will use one and ignore the other.

RELEASE			<domid>|
	Manually requests that xenstored disconnect from the domain.
	The event channel is unbound at the xenstored end and the page
	unmapped.  If the domain is still running it won't be able to
	communicate with xenstored.  NB that xenstored will in any
	case detect domain destruction and disconnect by itself.
	xenstored prevents the use of RELEASE other than by dom0.

GET_DOMAIN_PATH		<domid>|		<path>|
	Returns the domain's base path, as is used for relative
	transactions: ie, /local/domain/<domid> (with <domid>
	normalised).  The answer will be useless unless <domid> is a
	real domain id.

IS_DOMAIN_INTRODUCED	<domid>|		T| or F|
	Returns T if xenstored is in communication with the domain:
	ie, if INTRODUCE for the domain has not yet been followed by
	domain destruction or explicit RELEASE.

RESUME			<domid>|

	Arranges that @releaseDomain events will once more be
	generated when the domain becomes shut down.  This might have
	to be used if a domain were to be shut down (generating one
	@releaseDomain) and then subsequently restarted, since the
	state-sensitive algorithm in xenstored will not otherwise send
	further watch event notifications if the domain were to be
	shut down again.

	This command will be issued in place such as resume because
	Xen will "shutdown" the domain on suspend.

	xenstored prevents the use of RESUME other than by dom0.


SET_TARGET		<domid>|<tdomid>|
	Notifies xenstored that domain <domid> is targeting domain
	<tdomid>. This grants domain <domid> full access to paths
	owned by <tdomid>. Domain <domid> also inherits all
	permissions granted to <tdomid> on all other paths. This
	allows <domid> to behave as if it were dom0 when modifying
	paths related to <tdomid>.

	xenstored prevents the use of SET_TARGET other than by dom0.

GET_FEATURE		[<domid>|]		<value>|
SET_FEATURE		<domid>|<value>|
	Returns or sets the contents of the "feature" field copied to
	offset 2064 of the Xenstore ring page of the domain specified by
	<domid>. <value> is a decimal number being a logical or of the
	feature bits as defined in docs/misc/xenstore-ring.txt. Trying
	to set a bit for a feature not being supported by the running
	Xenstore will be denied. Providing no <domid> with the
	GET_FEATURE command will return the features which are supported
	by Xenstore for the domain issuing the command.

	SET_FEATURE for a domain will be rejected after the INTRODUCE
	command for this domain has been sent to xenstored. This has the
	effect that dom0 will always have all supported features enabled.

	xenstored prevents the use of GET_FEATURE with a domid specified
	and SET_FEATURE other than by dom0.

GET_QUOTA		[[<domid>|]<quota>|]	<value>|
SET_QUOTA		[<domid>|]<quota>|<value>|
	Returns or sets a quota value for the domain being specified by
	<domid>. Omitting <domid> will return or set the global quota
	values, which are the default values for new domains. <quota> is
	 one of "nodes", "watches", "transactions", "node-size",
	"permissions", or any other implementation defined value. For
	GET_QUOTA it is possible to omit the <quota> parameter together
	with the <domid> parameter, which will return a single string of
	all supported <quota> values separated by blanks. <value> is a
	decimal number specifying the quota value, with "0" having the
	special meaning of quota checks being disabled. The initial quota
	settings for a domain are the global ones of Xenstore.

	xenstored prevents the use of GET_QUOTA and SET_QUOTA other
	than by dom0.

---------- Miscellaneous ----------

CONTROL			<command>|[<parameters>|]
	Send a control command <command> with optional parameters
	(<parameters>) to Xenstore daemon.

	The set of commands and their semantics is implementation
	specific and is likely to change from one Xen version to the
	next.  Out-of-tree users will encounter compatibility issues.

	Current commands are:
	check
		checks xenstored innards
	live-update|<params>|+
		perform a live-update of the Xenstore daemon, only to
		be used via xenstore-control command.
		<params> are implementation specific and are used for
		different steps of the live-update processing. Currently
		supported <params> are:
		-f <file>  specify new daemon binary
		-b <size>  specify size of new stubdom binary
		-d <chunk-size> <binary-chunk>  transfer chunk of new
			stubdom binary
		-c <pars>  specify new command line to use
		-s [-t <sec>] [-F]  start live update process (-t specifies
			timeout in seconds to wait for active transactions
			to finish, default is 60 seconds; -F will force
			live update to happen even with running transactions
			after timeout elapsed)
		-a  abort live update handling
		All sub-options will return "OK" in case of success or an
		error string in case of failure. -s can return "BUSY" in case
		of an active transaction, a retry of -s can be done in that
		case.
	log|[on|off|+<switch>|-<switch>]
		without parameters: show possible log switches
		on: turn xenstore logging on
		off: turn xenstore logging off
		+<switch>: activates log entries for <switch>,
		-<switch>: deactivates log entries for <switch>
	logfile|<file-name>
		log to specified file
	memreport|[<file-name>]
		print memory statistics to logfile (no <file-name>
		specified) or to specific file
	print|<string>
		print <string> to syslog (xenstore runs as daemon) or
		to console (xenstore runs as stubdom)
	quota|[set <name> <val>|<domid>|max [-r]]
		without parameters: print the current quota settings
		with "set <name> <val>": set the quota <name> to new value
		<val> (The admin should make sure all the domain usage is
		below the quota. If it is not, then Xenstored may continue to
		handle requests from the domain as long as the resource
		violating the new quota setting isn't increased further)
		with "<domid>": print quota related accounting data for
		the domain <domid>
		with "max [-r]": show global per-domain maximum values of all
		unprivileged domains, optionally reset the values by adding
		"-r"
	quota-soft|[set <name> <val>]
		like the "quota" command, but for soft-quota.
	help			<supported-commands>
		return list of supported commands for CONTROL

DEBUG
	Deprecated, now named CONTROL