Stefano Stabellini
1a0ab02e34
PV Calls is a paravirtualized protocol that allows the implementation of a set of POSIX functions in a different domain. The PV Calls frontend sends POSIX function calls to the backend, which implements them and returns a value to the frontend and acts on the function call. This version of the document covers networking function calls, such as connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but the protocol is meant to be easily extended to cover different sets of calls. Unimplemented commands return ENOTSUP. Signed-off-by: Stefano Stabellini <stefano@aporeto.com> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
36 KiB
PV Calls Protocol version 1
Glossary
The following is a list of terms and definitions used in the Xen community. If you are a Xen contributor you can skip this section.
-
PV
Short for paravirtualized.
-
Dom0
First virtual machine that boots. In most configurations Dom0 is privileged and has control over hardware devices, such as network cards, graphic cards, etc.
-
DomU
Regular unprivileged Xen virtual machine.
-
Domain
A Xen virtual machine. Dom0 and all DomUs are all separate Xen domains.
-
Guest
Same as domain: a Xen virtual machine.
-
Frontend
Each DomU has one or more paravirtualized frontend drivers to access disks, network, console, graphics, etc. The presence of PV devices is advertized on XenStore, a cross domain key-value database. Frontends are similar in intent to the virtio drivers in Linux.
-
Backend
A Xen paravirtualized backend typically runs in Dom0 and it is used to export disks, network, console, graphics, etcs, to DomUs. Backends can live both in kernel space and in userspace. For example xen-blkback lives under drivers/block in the Linux kernel and xen_disk lives under hw/block in QEMU. Paravirtualized backends are similar in intent to virtio device emulators.
-
VMX and SVM
On Intel processors, VMX is the CPU flag for VT-x, hardware virtualization support. It corresponds to SVM on AMD processors.
Rationale
PV Calls is a paravirtualized protocol that allows the implementation of a set of POSIX functions in a different domain. The PV Calls frontend sends POSIX function calls to the backend, which implements them and returns a value to the frontend and acts on the function call.
This version of the document covers networking function calls, such as connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but the protocol is meant to be easily extended to cover different sets of calls. Unimplemented commands return ENOTSUP.
PV Calls provide the following benefits:
- full visibility of the guest behavior on the backend domain, allowing for inexpensive filtering and manipulation of any guest calls
- excellent performance
Specifically, PV Calls for networking offer these advantages:
- guest networking works out of the box with VPNs, wireless networks and any other complex configurations on the host
- guest services listen on ports bound directly to the backend domain IP addresses
- localhost becomes a secure host wide network for inter-VMs communications
Design
Why Xen?
PV Calls are part of an effort to create a secure runtime environment for containers (Open Containers Initiative images to be precise). PV Calls are based on Xen, although porting them to other hypervisors is possible. Xen was chosen because of its security and isolation properties and because it supports PV guests, a type of virtual machines that does not require hardware virtualization extensions (VMX on Intel processors and SVM on AMD processors). This is important because PV Calls is meant for containers and containers are often run on top of public cloud instances, which do not support nested VMX (or SVM) as of today (early 2017). Xen PV guests are lightweight, minimalist, and do not require machine emulation: all properties that make them a good fit for this project.
Xenstore
The frontend and the backend connect via xenstore to exchange information. The toolstack creates front and back nodes with state of XenbusStateInitialising. The protocol node name is pvcalls. There can only be one PV Calls frontend per domain.
Frontend XenBus Nodes
version Values:
Protocol version, chosen among the ones supported by the backend
(see **versions** under [Backend XenBus Nodes]). Currently the
value must be "1".
port Values: <uint32_t>
The identifier of the Xen event channel used to signal activity
in the command ring.
ring-ref Values: <uint32_t>
The Xen grant reference granting permission for the backend to map
the sole page in a single page sized command ring.
Backend XenBus Nodes
versions Values:
List of comma separated protocol versions supported by the backend.
For example "1,2,3". Currently the value is just "1", as there is
only one version.
max-page-order Values: <uint32_t>
The maximum supported size of a memory allocation in units of
log2n(machine pages), e.g. 1 = 2 pages, 2 == 4 pages, etc. It must
be 1 or more.
function-calls Values: <uint32_t>
Value "0" means that no calls are supported.
Value "1" means that socket, connect, release, bind, listen, accept
and poll are supported.
State Machine
Initialization:
*Front* *Back*
XenbusStateInitialising XenbusStateInitialising
- Query virtual device - Query backend device
properties. identification data.
- Setup OS device instance. - Publish backend features
- Allocate and initialize the and transport parameters
request ring. |
- Publish transport parameters |
that will be in effect during V
this connection. XenbusStateInitWait
|
|
V
XenbusStateInitialised
- Query frontend transport parameters.
- Connect to the request ring and
event channel.
|
|
V
XenbusStateConnected
- Query backend device properties.
- Finalize OS virtual device
instance.
|
|
V
XenbusStateConnected
Once frontend and backend are connected, they have a shared page, which will is used to exchange messages over a ring, and an event channel, which is used to send notifications.
Shutdown:
*Front* *Back*
XenbusStateConnected XenbusStateConnected
|
|
V
XenbusStateClosing
- Unmap grants
- Unbind event channels
|
|
V
XenbusStateClosing
- Unbind event channels
- Free rings
- Free data structures
|
|
V
XenbusStateClosed
- Free remaining data structures
|
|
V
XenbusStateClosed
Commands Ring
The shared ring is used by the frontend to forward POSIX function calls
to the backend. We shall refer to this ring as commands ring to
distinguish it from other rings which can be created later in the
lifecycle of the protocol (see [Indexes Page and Data ring]). The grant
reference for shared page for this ring is shared on xenstore (see
[Frontend XenBus Nodes]). The ring format is defined using the familiar
DEFINE_RING_TYPES macro (xen/include/public/io/ring.h). Frontend
requests are allocated on the ring using the RING_GET_REQUEST macro.
The list of commands below is in calling order.
The format is defined as follows:
#define PVCALLS_SOCKET 0
#define PVCALLS_CONNECT 1
#define PVCALLS_RELEASE 2
#define PVCALLS_BIND 3
#define PVCALLS_LISTEN 4
#define PVCALLS_ACCEPT 5
#define PVCALLS_POLL 6
struct xen_pvcalls_request {
uint32_t req_id; /* private to guest, echoed in response */
uint32_t cmd; /* command to execute */
union {
struct xen_pvcalls_socket {
uint64_t id;
uint32_t domain;
uint32_t type;
uint32_t protocol;
#ifdef CONFIG_X86_32
uint8_t pad[4];
#endif
} socket;
struct xen_pvcalls_connect {
uint64_t id;
uint8_t addr[28];
uint32_t len;
uint32_t flags;
grant_ref_t ref;
uint32_t evtchn;
#ifdef CONFIG_X86_32
uint8_t pad[4];
#endif
} connect;
struct xen_pvcalls_release {
uint64_t id;
uint8_t reuse;
#ifdef CONFIG_X86_32
uint8_t pad[7];
#endif
} release;
struct xen_pvcalls_bind {
uint64_t id;
uint8_t addr[28];
uint32_t len;
} bind;
struct xen_pvcalls_listen {
uint64_t id;
uint32_t backlog;
#ifdef CONFIG_X86_32
uint8_t pad[4];
#endif
} listen;
struct xen_pvcalls_accept {
uint64_t id;
uint64_t id_new;
grant_ref_t ref;
uint32_t evtchn;
} accept;
struct xen_pvcalls_poll {
uint64_t id;
} poll;
/* dummy member to force sizeof(struct xen_pvcalls_request) to match across archs */
struct xen_pvcalls_dummy {
uint8_t dummy[56];
} dummy;
} u;
};
The first two fields are common for every command. Their binary layout is:
0 4 8
+-------+-------+
|req_id | cmd |
+-------+-------+
-
req_id is generated by the frontend and is a cookie used to identify one specific request/response pair of commands. Not to be confused with any command id which are used to identify a socket across multiple commands, see Socket.
-
cmd is the command requested by the frontend:
PVCALLS_SOCKET: 0PVCALLS_CONNECT: 1PVCALLS_RELEASE: 2PVCALLS_BIND: 3PVCALLS_LISTEN: 4PVCALLS_ACCEPT: 5PVCALLS_POLL: 6
Both fields are echoed back by the backend. See [Socket families and address format] for the format of the addr field of connect and bind. The maximum size of command specific arguments is 56 bytes. Any future command that requires more than that will need a bump the version of the protocol.
Similarly to other Xen ring based protocols, after writing a request to
the ring, the frontend calls RING_PUSH_REQUESTS_AND_CHECK_NOTIFY and
issues an event channel notification when a notification is required.
Backend responses are allocated on the ring using the RING_GET_RESPONSE macro.
The format is the following:
struct xen_pvcalls_response {
uint32_t req_id;
uint32_t cmd;
int32_t ret;
uint32_t pad;
union {
struct _xen_pvcalls_socket {
uint64_t id;
} socket;
struct _xen_pvcalls_connect {
uint64_t id;
} connect;
struct _xen_pvcalls_release {
uint64_t id;
} release;
struct _xen_pvcalls_bind {
uint64_t id;
} bind;
struct _xen_pvcalls_listen {
uint64_t id;
} listen;
struct _xen_pvcalls_accept {
uint64_t id;
} accept;
struct _xen_pvcalls_poll {
uint64_t id;
} poll;
struct _xen_pvcalls_dummy {
uint8_t dummy[8];
} dummy;
} u;
};
The first four fields are common for every response. Their binary layout is:
0 4 8 12 16
+-------+-------+-------+-------+
|req_id | cmd | ret | pad |
+-------+-------+-------+-------+
- req_id: echoed back from request
- cmd: echoed back from request
- ret: return value, identifies success (0) or failure (see [Error numbers] in further sections). If the cmd is not supported by the backend, ret is ENOTSUP.
- pad: padding
After calling RING_PUSH_RESPONSES_AND_CHECK_NOTIFY, the backend checks whether
it needs to notify the frontend and does so via event channel.
A description of each command, their additional request and response fields follow.
Socket
The socket operation corresponds to the POSIX socket function. It creates a new socket of the specified family, type and protocol. id is freely chosen by the frontend and references this specific socket from this point forward. See [Socket families and address format] to see which ones are supported by different versions of the protocol.
Request fields:
- cmd value: 0
- additional fields:
- id: generated by the frontend, it identifies the new socket
- domain: the communication domain
- type: the socket type
- protocol: the particular protocol to be used with the socket, usually 0
Request binary layout:
8 12 16 20 24 28
+-------+-------+-------+-------+-------+
| id |domain | type |protoco|
+-------+-------+-------+-------+-------+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+-------+--------+
| id |
+-------+--------+
Return value:
- 0 on success
- See the POSIX socket function for error names; see [Error numbers] in further sections.
Connect
The connect operation corresponds to the POSIX connect function. It connects a previously created socket (identified by id) to the specified address.
The connect operation creates a new shared ring, which we'll call data ring. The data ring is used to send and receive data from the socket. The connect operation passes two additional parameters: evtchn and ref. evtchn is the port number of a new event channel which will be used for notifications of activity on the data ring. ref is the grant reference of the indexes page: a page which contains shared indexes that point to the write and read locations in the data ring. The indexes page also contains the full array of grant references for the data ring. When the frontend issues a connect command, the backend:
- finds its own internal socket corresponding to id
- connects the socket to addr
- maps the grant reference ref, the indexes page, see struct pvcalls_data_intf
- maps all the grant references listed in
struct pvcalls_data_intfand uses them as shared memory for the data ring - bind the evtchn
- replies to the frontend
The [Indexes Page and Data ring] format will be described in the following section. The data ring is unmapped and freed upon issuing a release command on the active socket identified by id. A frontend state change can also cause data rings to be unmapped.
Request fields:
- cmd value: 0
- additional fields:
- id: identifies the socket
- addr: address to connect to, see [Socket families and address format]
- len: address length up to 28 octets
- flags: flags for the connection, reserved for future usage
- ref: grant reference of the indexes page
- evtchn: port number of the evtchn to signal activity on the data ring
Request binary layout:
8 12 16 20 24 28 32 36 40 44
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| id | addr |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| len | flags | ref |evtchn |
+-------+-------+-------+-------+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+-------+-------+
| id |
+-------+-------+
Return value:
- 0 on success
- See the POSIX connect function for error names; see [Error numbers] in further sections.
Release
The release operation closes an existing active or a passive socket.
When a release command is issued on a passive socket, the backend releases it and frees its internal mappings. When a release command is issued for an active socket, the data ring and indexes page are also unmapped and freed:
- frontend sends release command for an active socket
- backend releases the socket
- backend unmaps the data ring
- backend unmaps the indexes page
- backend unbinds the event channel
- backend replies to frontend with an ret value
- frontend frees data ring, indexes page and unbinds event channel
Request fields:
- cmd value: 1
- additional fields:
- id: identifies the socket
- reuse: an optimization hint for the backend. The field is ignored for passive sockets. When set to 1, the frontend lets the backend know that it will reuse exactly the same set of grant pages (indexes page and data ring) and event channel when creating one of the next active sockets. The backend can take advantage of it by delaying unmapping grants and unbinding the event channel. The backend is free to ignore the hint. Reused data rings are found by ref, the grant reference of the page containing the indexes.
Request binary layout:
8 12 16 17
+-------+-------+-----+
| id |reuse|
+-------+-------+-----+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+-------+-------+
| id |
+-------+-------+
Return value:
- 0 on success
- See the POSIX shutdown function for error names; see [Error numbers] in further sections.
Bind
The bind operation corresponds to the POSIX bind function. It assigns the address passed as parameter to a previously created socket, identified by id. Bind, listen and accept are the three operations required to have fully working passive sockets and should be issued in that order.
Request fields:
- cmd value: 2
- additional fields:
- id: identifies the socket
- addr: address to connect to, see [Socket families and address format]
- len: address length up to 28 octets
Request binary layout:
8 12 16 20 24 28 32 36 40 44
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| id | addr |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| len |
+-------+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+-------+-------+
| id |
+-------+-------+
Return value:
- 0 on success
- See the POSIX bind function for error names; see [Error numbers] in further sections.
Listen
The listen operation marks the socket as a passive socket. It corresponds to the POSIX listen function.
Reuqest fields:
- cmd value: 3
- additional fields:
- id: identifies the socket
- backlog: the maximum length to which the queue of pending connections may grow in number of elements
Request binary layout:
8 12 16 20
+-------+-------+-------+
| id |backlog|
+-------+-------+-------+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+-------+-------+
| id |
+-------+-------+
Return value:
- 0 on success
- See the POSIX listen function for error names; see [Error numbers] in further sections.
Accept
The accept operation extracts the first connection request on the queue of pending connections for the listening socket identified by id and creates a new connected socket. The id of the new socket is also chosen by the frontend and passed as an additional field of the accept request struct (id_new). See the POSIX accept function as reference.
Similarly to the connect operation, accept creates new [Indexes Page and Data ring]. The data ring is used to send and receive data from the socket. The accept operation passes two additional parameters: evtchn and ref. evtchn is the port number of a new event channel which will be used for notifications of activity on the data ring. ref is the grant reference of the indexes page: a page which contains shared indexes that point to the write and read locations in the data ring. The indexes page also contains the full array of grant references for the data ring.
The backend will reply to the request only when a new connection is successfully accepted, i.e. the backend does not return EAGAIN or EWOULDBLOCK.
Example workflow:
- frontend issues an accept request
- backend waits for a connection to be available on the socket
- a new connection becomes available
- backend accepts the new connection
- backend creates an internal mapping from id_new to the new socket
- backend maps the grant reference ref, the indexes page, see struct pvcalls_data_intf
- backend maps all the grant references listed in
struct pvcalls_data_intfand uses them as shared memory for the new data ring in and out arrays - backend binds to the evtchn
- backend replies to the frontend with a ret value
Request fields:
- cmd value: 4
- additional fields:
- id: id of listening socket
- id_new: id of the new socket
- ref: grant reference of the indexes page
- evtchn: port number of the evtchn to signal activity on the data ring
Request binary layout:
8 12 16 20 24 28 32
+-------+-------+-------+-------+-------+-------+
| id | id_new | ref |evtchn |
+-------+-------+-------+-------+-------+-------+
Response additional fields:
- id: id of the listening socket, echoed back from request
Response binary layout:
16 20 24
+-------+-------+
| id |
+-------+-------+
Return value:
- 0 on success
- See the POSIX accept function for error names; see [Error numbers] in further sections.
Poll
In this version of the protocol, the poll operation is only valid for passive sockets. For active sockets, the frontend should look at the indexes on the indexes page. When a new connection is available in the queue of the passive socket, the backend generates a response and notifies the frontend.
Request fields:
- cmd value: 5
- additional fields:
- id: identifies the listening socket
Request binary layout:
8 12 16
+-------+-------+
| id |
+-------+-------+
Response additional fields:
- id: echoed back from request
Response binary layout:
16 20 24
+--------+--------+
| id |
+--------+--------+
Return value:
- 0 on success
- See the POSIX poll function for error names; see [Error numbers] in further sections.
Expanding the protocol
It is possible to introduce new commands without changing the protocol ABI. Naturally, a feature flag among the backend xenstore nodes should advertise the availability of a new set of commands.
If a new command requires parameters in struct xen_pvcalls_request larger than 56 bytes, which is the current size of the request, then the protocol version should be increased. One way to implement the large request structure without disrupting the current ABI is to introduce a new command, such as PVCALLS_CONNECT_EXTENDED, and a flag to specify that the request uses two request slots, for a total of 112 bytes.
Error numbers
The numbers corresponding to the error names specified by POSIX are:
[EPERM] -1
[ENOENT] -2
[ESRCH] -3
[EINTR] -4
[EIO] -5
[ENXIO] -6
[E2BIG] -7
[ENOEXEC] -8
[EBADF] -9
[ECHILD] -10
[EAGAIN] -11
[EWOULDBLOCK] -11
[ENOMEM] -12
[EACCES] -13
[EFAULT] -14
[EBUSY] -16
[EEXIST] -17
[EXDEV] -18
[ENODEV] -19
[EISDIR] -21
[EINVAL] -22
[ENFILE] -23
[EMFILE] -24
[ENOSPC] -28
[EROFS] -30
[EMLINK] -31
[EDOM] -33
[ERANGE] -34
[EDEADLK] -35
[EDEADLOCK] -35
[ENAMETOOLONG] -36
[ENOLCK] -37
[ENOTEMPTY] -39
[ENOSYS] -38
[ENODATA] -61
[ETIME] -62
[EBADMSG] -74
[EOVERFLOW] -75
[EILSEQ] -84
[ERESTART] -85
[ENOTSOCK] -88
[EOPNOTSUPP] -95
[EAFNOSUPPORT] -97
[EADDRINUSE] -98
[EADDRNOTAVAIL] -99
[ENOBUFS] -105
[EISCONN] -106
[ENOTCONN] -107
[ETIMEDOUT] -110
[ENOTSUP] -524
Socket families and address format
The following definitions and explicit sizes, together with POSIX
sys/socket.h and netinet/in.h define socket families and
address format. Please be aware that only the domain AF_INET, type
SOCK_STREAM and protocol 0 are supported by this version of the
specification, others return ENOTSUP.
#define AF_UNSPEC 0
#define AF_UNIX 1 /* Unix domain sockets */
#define AF_LOCAL 1 /* POSIX name for AF_UNIX */
#define AF_INET 2 /* Internet IP Protocol */
#define AF_INET6 10 /* IP version 6 */
#define SOCK_STREAM 1
#define SOCK_DGRAM 2
#define SOCK_RAW 3
/* generic address format */
struct sockaddr {
uint16_t sa_family_t;
char sa_data[26];
};
struct in_addr {
uint32_t s_addr;
};
/* AF_INET address format */
struct sockaddr_in {
uint16_t sa_family_t;
uint16_t sin_port;
struct in_addr sin_addr;
char sin_zero[20];
};
Indexes Page and Data ring
Data rings are used for sending and receiving data over a connected socket. They are created upon a successful accept or connect command. The sendmsg and recvmsg calls are implemented by sending data and receiving data from a data ring, and updating the corresponding indexes on the indexes page.
Firstly, the indexes page is shared by a connect or accept
command, see ref parameter in their sections. The content of the
indexes page is represented by struct pvcalls_ring_intf, see
below. The structure contains the list of grant references which
constitute the in and out buffers of the data ring, see ref[]
below. The backend maps the grant references contiguously. Of the
resulting shared memory, the first half is dedicated to the in array
and the second half to the out array. They are used as circular
buffers for transferring data, and, together, they are the data ring.
+---------------------------+ Indexes page | Command ring: | +----------------------+ | @0: xen_pvcalls_connect: | |@0 pvcalls_data_intf: | | @44: ref +-------------------------------->+@76: ring_order = 1 | | | |@80: ref[0]+ | +---------------------------+ |@84: ref[1]+ | | | | | | | +----------------------+ | v (data ring) +-------+-----------+ | @0->4098: in | | ref[0] | |-------------------| | @4099->8196: out | | ref[1] | +-------------------+
Indexes Page Structure
typedef uint32_t PVCALLS_RING_IDX;
struct pvcalls_data_intf {
PVCALLS_RING_IDX in_cons, in_prod;
int32_t in_error;
uint8_t pad[52];
PVCALLS_RING_IDX out_cons, out_prod;
int32_t out_error;
uint8_t pad[52];
uint32_t ring_order;
grant_ref_t ref[];
};
/* not actually C compliant (ring_order changes from socket to socket) */
struct pvcalls_data {
char in[((1<<ring_order)<<PAGE_SHIFT)/2];
char out[((1<<ring_order)<<PAGE_SHIFT)/2];
};
- ring_order
It represents the order of the data ring. The following list of grant
references is of
(1 << ring_order)elements. It cannot be greater than max-page-order, as specified by the backend on XenBus. It has to be one at minimum. - ref[]
The list of grant references which will contain the actual data. They are
mapped contiguosly in virtual memory. The first half of the pages is the
in array, the second half is the out array. The arrays must
have a power of two size. Together, their size is
(1 << ring_order) * PAGE_SIZE. - in is an array used as circular buffer It contains data read from the socket. The producer is the backend, the consumer is the frontend.
- out is an array used as circular buffer It contains data to be written to the socket. The producer is the frontend, the consumer is the backend.
- in_cons and in_prod Consumer and producer indexes for data read from the socket. They keep track of how much data has already been consumed by the frontend from the in array. in_prod is increased by the backend, after writing data to in. in_cons is increased by the frontend, after reading data from in.
- out_cons, out_prod Consumer and producer indexes for the data to be written to the socket. They keep track of how much data has been written by the frontend to out and how much data has already been consumed by the backend. out_prod is increased by the frontend, after writing data to out. out_cons is increased by the backend, after reading data from out.
- in_error and out_error They signal errors when reading from the socket (in_error) or when writing to the socket (out_error). 0 means no errors. When an error occurs, no further reads or writes operations are performed on the socket. In the case of an orderly socket shutdown (i.e. read returns 0) in_error is set to ENOTCONN. in_error and out_error are never set to EAGAIN or EWOULDBLOCK (the data is written to the ring as soon as it is available).
The binary layout of struct pvcalls_data_intf follows:
0 4 8 12 64 68 72 76
+---------+---------+---------+-----//-----+---------+---------+---------+
| in_cons | in_prod |in_error | padding |out_cons |out_prod |out_error|
+---------+---------+---------+-----//-----+---------+---------+---------+
76 80 84 88 4092 4096
+---------+---------+---------+----//---+---------+
|ring_orde| ref[0] | ref[1] | | ref[N] |
+---------+---------+---------+----//---+---------+
N.B For one page, N is maximum 991 ((4096-132)/4), but given that N needs to be a power of two, actually max N is 512 (ring_order = 9).
Data Ring Structure
The binary layout of the data ring follow:
0 ((1<<ring_order)<<PAGE_SHIFT)/2 ((1<<ring_order)<<PAGE_SHIFT)
+------------//-------------+------------//-------------+
| in | out |
+------------//-------------+------------//-------------+
Why ring.h is not needed
Many Xen PV protocols use the macros provided by ring.h to manage their shared ring for communication. PVCalls does not, because the [Data Ring Structure] actually comes with two rings: the in ring and the out ring. Each of them is mono-directional, and there is no static request size: the producer writes opaque data to the ring. On the other end, in ring.h they are combined, and the request size is static and well-known. In PVCalls:
in -> backend to frontend only out-> frontend to backend only
In the case of the in ring, the frontend is the consumer, and the backend is the producer. Everything is the same but mirrored for the out ring.
The producer, the backend in this case, never reads from the in ring. In fact, the producer doesn't need any notifications unless the ring is full. This version of the protocol doesn't take advantage of it, leaving room for optimizations.
On the other end, the consumer always requires notifications, unless it is already actively reading from the ring. The producer can figure it out, without any additional fields in the protocol, by comparing the indexes at the beginning and the end of the function. This is similar to what ring.h does.
Workflow
The in and out arrays are used as circular buffers:
0 sizeof(array) == ((1<<ring_order)<<PAGE_SHIFT)/2
+-----------------------------------+
|to consume| free |to consume |
+-----------------------------------+
^ ^
prod cons
0 sizeof(array)
+-----------------------------------+
| free | to consume | free |
+-----------------------------------+
^ ^
cons prod
The following function is provided to calculate how many bytes are currently left unconsumed in an array:
#define _MASK_PVCALLS_IDX(idx, ring_size) ((idx) & (ring_size-1))
static inline PVCALLS_RING_IDX pvcalls_ring_unconsumed(PVCALLS_RING_IDX prod,
PVCALLS_RING_IDX cons,
PVCALLS_RING_IDX ring_size)
{
PVCALLS_RING_IDX size;
if (prod == cons)
return 0;
prod = _MASK_PVCALLS_IDX(prod, ring_size);
cons = _MASK_PVCALLS_IDX(cons, ring_size);
if (prod == cons)
return ring_size;
if (prod > cons)
size = prod - cons;
else {
size = ring_size - cons;
size += prod;
}
return size;
}
The producer (the backend for in, the frontend for out) writes to the array in the following way:
- read [in|out]_cons, [in|out]_prod, [in|out]_error from shared memory
- general memory barrier
- return on [in|out]_error
- write to array at position [in|out]_prod up to [in|out]_cons, wrapping around the circular buffer when necessary
- write memory barrier
- increase [in|out]_prod
- notify the other end via evtchn
The consumer (the backend for out, the frontend for in) reads from the array in the following way:
- read [in|out]_prod, [in|out]_cons, [in|out]_error from shared memory
- read memory barrier
- return on [in|out]_error
- read from array at position [in|out]_cons up to [in|out]_prod, wrapping around the circular buffer when necessary
- general memory barrier
- increase [in|out]_cons
- notify the other end via evtchn
The producer takes care of writing only as many bytes as available in the buffer up to [in|out]_cons. The consumer takes care of reading only as many bytes as available in the buffer up to [in|out]_prod. [in|out]_error is set by the backend when an error occurs writing or reading from the socket.