mirror of
https://github.com/xen-project/xen.git
synced 2026-10-29 02:30:03 +01:00
Edwin Török
9284ae0c40
Introduce a limit on the number of outstanding reply packets in the xenbus queue. This limits the number of in-flight requests: when the output queue is full we'll stop processing inputs until the output queue has room again. To avoid a busy loop on the Unix socket we only add it to the watched input file descriptor set if we'd be able to call `input` on it. Even though Dom0 is trusted and exempt from quotas a flood of events might cause a backlog where events are produced faster than daemons in Dom0 can consume them, which could lead to an unbounded queue size and OOM. Therefore the xenbus queue limit must apply to all connections, Dom0 is not exempt from it, although if everything works correctly it will eventually catch up. This prevents a malicious guest from sending more commands while it has outstanding watch events or command replies in its input ring. However if it can cause the generation of watch events by other means (e.g. by Dom0, or another cooperative guest) and stop reading its own ring then watch events would've queued up without limit. The xenstore protocol doesn't have a back-pressure mechanism, and doesn't allow dropping watch events. In fact, dropping watch events is known to break some pieces of normal functionality. This leaves little choice to safely implement the xenstore protocol without exposing the xenstore daemon to out-of-memory attacks. Implement the fix as pipes with bounded buffers: * Use a bounded buffer for watch events * The watch structure will have a bounded receiving pipe of watch events * The source will have an "overflow" pipe of pending watch events it couldn't deliver Items are queued up on one end and are sent as far along the pipe as possible: source domain -> watch -> xenbus of target -> xenstore ring/socket of target If the pipe is "full" at any point then back-pressure is applied and we prevent more items from being queued up. For the source domain this means that we'll stop accepting new commands as long as its pipe buffer is not empty. Before we try to enqueue an item we first check whether it is possible to send it further down the pipe, by attempting to recursively flush the pipes. This ensures that we retain the order of events as much as possible. We might break causality of watch events if the target domain's queue is full and we need to start using the watch's queue. This is a breaking change in the xenstore protocol, but only for domains which are not processing their incoming ring as expected. When a watch is deleted its entire pending queue is dropped (no code is needed for that, because it is part of the 'watch' type). There is a cache of watches that have pending events that we attempt to flush at every cycle if possible. Introduce 3 limits here: * quota-maxwatchevents on watch event destination: when this is hit the source will not be allowed to queue up more watch events. * quota-maxoustanding which is the number of responses not read from the ring: once exceeded, no more inputs are processed until all outstanding replies are consumed by the client. * overflow queue on the watch event source: all watches that cannot be stored on destination are queued up here, a single command can trigger multiple watches (e.g. due to recursion). The overflow queue currently doesn't have an upper bound, it is difficult to accurately calculate one as it depends on whether you are Dom0 and how many watches each path has registered and how many watch events you can trigger with a single command (e.g. a commit). However these events were already using memory, this just moves them elsewhere, and as long as we correctly block a domain it shouldn't result in unbounded memory usage. Note that Dom0 is not excluded from these checks, it is important that Dom0 is especially not excluded when it is the source, since there are many ways in which a guest could trigger Dom0 to send it watch events. This should protect against malicious frontends as long as the backend follows the PV xenstore protocol and only exposes paths needed by the frontend, and changes those paths at most once as a reaction to guest events, or protocol state. The queue limits are per watch, and per domain-pair, so even if one communication channel would be "blocked", others would keep working, and the domain itself won't get blocked as long as it doesn't overflow the queue of watch events. Similarly a malicious backend could cause the frontend to get blocked, but this watch queue protects the frontend as well as long as it follows the PV protocol. (Although note that protection against malicious backends is only a best effort at the moment) This is part of XSA-326 / CVE-2022-42318. Signed-off-by: Edwin Török <edvin.torok@citrix.com> Acked-by: Christian Lindig <christian.lindig@citrix.com>