7. Viability & Safe Value Functions

Viability kernels, learnable safety measures, the penalty threshold theorem, entropy regularisation and uncertainty-aware safe RL

Book contents · Apply this chapter to a real decision · Glossary

Modules 4–6 kept a learner safe by never evaluating a parameter whose safety was not certified. This module changes the object of study: which states are safe, and what it takes for an optimal controller to respect that. The central question, posed by Massiani, Heim, Solowjow and Trimpe, is one every RL practitioner answers by reflex when writing a reward function: how large must the failure penalty be? The answer is a theorem with an exponential in it.

1. Failure Sets, Viable Sets and the Viability Kernel

Safety in this module is a state constraint: there is a set of failure states that the system must never visit. The difficulty is that "not failed yet" is not the same as "safe". A car sliding on ice towards a wall has not crashed, but no input can prevent the crash any more. Viability theory (Aubin, Viability Theory, 1991/2009) names the states from which failure can still be avoided forever; that set, and not the complement of the failure set, is the right notion of a safe set.

Set notation used below (Primer 0): $X\setminus Y$ is the set of points of $X$ that are not in $Y$; $X\times U$ is the set of all pairs $(x,u)$ (for $X=\{0,1\}$ and $U=\{-1,1\}$ it has four elements); $\exists$ reads "there exists" and $\forall$ "for every"; an intersection $\bigcap_{x\in X_V}\mathcal U_{\mathrm{safe}}(x)$ keeps the elements that belong to every set $\mathcal U_{\mathrm{safe}}(x)$. The order of quantifiers matters: $\exists u\ \forall t$ asks for one controller that works for the entire trajectory, whereas $\forall t\ \exists u$ would allow a different controller for each horizon $t$.

Computing the kernel on a finite space

On a finite state-input space the kernel is a greatest fixed point and can be computed by a decreasing iteration. This is what the explorer below runs, and it is also the algorithm behind shield synthesis (Section 7).

Why it works. By induction (Primer 0), $X^k$ is the set of states from which failure can be avoided for $k$ steps. The sequence is decreasing, so on a finite space it stabilises after at most $|X|$ iterations, and the limit is the largest set $Y \subseteq X \setminus X_F$ with the invariance property $\forall x \in Y\ \exists u : f(x,u) \in Y$, which is precisely $X_V$. On a continuous state space Hamilton–Jacobi reachability (see Primer D) approximates this object on a grid: $X_U$ is the unavoidable-failure backward-reachable tube of $X_F$, and $X_V$ is its complement (Bansal, Chen, Herbert & Tomlin, CDC 2017; Module 10).

The tube meant here is the unavoidable-failure tube: $\{x:\forall u\in\mathcal U,\ \exists t<\infty,\ \varphi_x^u(t)\in X_F\}$. It is not the set from which some controller can reach failure. For example, a shelf state can be driven off the edge, but remains viable because another controller can keep it safe.

2. A Learnable Safety Measure

A binary label "viable / unviable" hides how close to failure a state is. Heim, von Rohr, Trimpe & Badri-Spröwitz, CoRL 2019 propose a graded notion: count the viable actions. This section uses their notation, $s' = T(s, a)$ with $q = (s, a) \in Q = S \times A$.

Learning the measure from samples, and why failures are informative

The goal is a large, conservative estimate $\hat Q_V \subseteq Q_V$ (see Module 4) obtained model-free. The agent picks an action $a$ in state $s$ and receives the tuple $(s', \mathrm{failed})$. Two kinds of samples arrive: a failure is the only direct observation of the ground truth, $\Lambda_Q(q) = 0$; a success only yields the current estimate $\hat\Lambda(s')$ of the next state's measure, which is itself computed from $\hat\Lambda_Q$. The learning signal therefore flows from failures outwards.

Why the two conditions. Non-failure samples only ever copy estimates and failures only ever lower them, so a viable pair that is wrongly assigned $0$ can be repaired only through a successor state whose estimate is still positive; once the zeros close a loop they are permanent. Optimism is needed so that errors are only ever in the direction that failures can fix, and it has to hold pair by pair: the invariant $Q_V \subseteq \hat Q_V$ survives every update (a viable pair's successor $s' \in S_V$ keeps at least its $\Lambda(s') \ge 1$ viable actions positive), and together with the theorem it forces $\hat Q_V = Q_V$, hence $\hat\Lambda = \Lambda$. The state-wise $\hat\Lambda \ge \Lambda$ does not suffice: take one viable state $s$ with a safe self-loop $a_1$ and a failing action $a_2$, initialised $\hat\Lambda_Q(s,a_1) = 0$, $\hat\Lambda_Q(s,a_2) = 1$, so that $\hat\Lambda(s) = 1 = \Lambda(s)$; if $a_2$ is sampled first, $\hat\Lambda(s)$ drops to $0$ and the self-loop copies $0$ forever. Infinite sampling is what lets every unviable pair eventually be observed as a failure, directly or through the chain $\hat\Lambda_Q(q) = \hat\Lambda(T(q))$. Both are the usual conditions of model-free learning and both are impractical, since optimistic starts encourage visits to the failure set. The practical algorithm trades the guarantee for confidence bounds.

"A random state of $\hat Q_{\mathrm{caut}}$" means a state of its projection onto the state space, $\{s:\exists a,\ (s,a)\in\hat Q_{\mathrm{caut}}\}$. The rule presupposes that this set is nonempty and that the experiment can be reset to a chosen state; otherwise an externally supplied reset state is needed.

Results reported. On a continuous version of the hovership with a deliberately poor prior mean, the estimate is close to the ground truth after 250 samples with an 8% failure rate; the confidence thresholds are raised linearly over the run as a heuristic. On a spring-loaded inverted pendulum (SLIP) running model, controlled once per step at the apex, a conservative prior converges to a nearly maximal conservative approximation of $Q_V$ with an 8% failure rate after 500 samples. Where the measure has a non-smooth edge (an infeasibility constraint) the GP's smoothness assumption is violated and more failures are sampled; at smooth borders the shrinking measure lets the border be inferred with few or no failures.

Exploration requirements: constraints need not be accurate everywhere

If the goal is not to know $Q_V$ but to make a given nominal policy $\pi$ safe, much less has to be learned. Massiani, Heim & Trimpe, L4DC 2021 define the constrained policy as the projection of $\pi$ onto a constraint set $K \subset Q$ (projections: Primer B),

with the paper's default cost $J(s,a) = \|a - \pi(s)\|_2^2$ (see Primer A), where $\mathrm{OPT}(K)(s)$ denotes the set of minimisers, and ask which closed sets $K$ produce the same policy as the true viable set, $\mathrm{OPT}(K) = \mathrm{OPT}(Q_V)$ on $S_V$. For this quadratic distance cost, nonempty closed action slices guarantee attainment (see Primer 0); for $\mathrm{OPT}(Q_V)$ itself the paper assumes that the dynamics generate a closed viable set.

Why: pick any feasible $a_0$ in the slice; actions with $\|a-\pi(s)\|_2\gt\|a_0-\pi(s)\|_2$ cannot be optimal, so the minimisation can be restricted to the slice intersected with a closed ball around $\pi(s)$, a compact set on which the continuous $J$ attains its minimum. For a general cost, attainment must be assumed separately (e.g. continuity plus compact slices). In the critical-set formula below, $J(s,\mathrm{OPT}(Q_V)(s))$ is shorthand for the common optimal cost $\min_{a\in Q_V[s]}J(s,a)$, not $J$ evaluated at a set.

In words: an admissible constraint must contain the optimal viable policy and exclude the critical set, and may otherwise contain unviable pairs that are never tempting. Two consequences follow. First, greed is good: following $\pi_{\hat K}(s) = \mathrm{OPT}(\hat K)(s)$ when feasible (and $\pi$ otherwise) aims exploration at $Q_{\mathrm{crit}}$ by itself, and the paper concludes that it keeps visiting $Q_{\mathrm{crit}}$ until the learner has excluded it, so the exploration requirement is fulfilled by the greedy on-policy strategy, with no off-policy exploration. That is an argument, not a theorem (Theorem 6 is a static statement about sets): the greedy policy only probes the critical pairs at states its trajectories actually reach, so a viable region they never enter is never explored, and each probed pair must then really be removed by the learner's update. In the experiments the coverage comes from starting every episode (at most 10 steps) in a random state of the projection of $\hat K$. Second, this is why failures are informative here too: as long as $\hat K$ contains $\mathrm{OPT}(Q_V)$, an unviable action that the greedy policy picks at a viable state costs at most $J(s, \mathrm{OPT}(Q_V)(s))$, so it is by definition a critical pair, and the failures it causes teach exactly what must be excluded. In their experiment with the affine nominal policy $a = 0.7 - 0.3 s$ on the hovership and the GP learner of Heim et al., 20 episodes (196 samples) gave a safe policy $\pi_{\hat K}$ over nearly the whole viability kernel.

3. Penalising Failure: The Setup

Back to control notation. The reward $r : X \times U \to \mathbb{R}$ is measurable and bounded. For a controller $u$ and initial state $x$ the return is

with the discount time constant $\tau \gt 0$; in discrete time the integral is a sum and $e^{-t/\tau} = \gamma^t$ with $\gamma = e^{-1/\tau}$. The smaller $\tau$, the shorter the agent's inner time horizon, although large delayed rewards can still matter. Because $r$ is bounded the return is finite for every $u$ and $x$. The (unconstrained) value function is $v(x) = \sup_{u \in \mathcal{U}} G(x,u)$ and satisfies the dynamic-programming identity (see Primer D), for all $t$,

The $\delta t$ bookkeeping is what produces the extra term in the discrete-time theorems. In discrete time the integral up to $t$ is the sum $\sum_{k=0}^{t}\gamma^k r(x_k,u_k)$, which already contains the reward at step $t$, so the "rest of the trajectory" starts one step later, at $x_{t+1}$, with weight $\gamma^{t+1}$; in continuous time the split point itself carries no reward and the continuation starts at $t$.

Three reward constants will control everything that follows. For discrete time, with $Q_V$ and $Q_U$ from Section 1,

$R_{Q_U}$ is the best one-step reward obtainable on a transition that leaves the kernel (or happens outside it), $R_{Q_V}$ the worst one-step reward of a viable transition, and $R_{X_U}$ the best reward rate available while doomed. In every example of this module the failure states carry zero reward; since $X_F \subset X_U$, this makes both suprema at least $0$. Only the sign of $R_{X_U}$ matters for the theory (Section 4).

4. Safe Value Functions: The Theorems

The main result has two layers. First, a condition on the value function itself that guarantees safety: unviable states must look worse than every viable state. Second, a guarantee that a large enough failure penalty always enforces that condition, with an explicit (if conservative) formula for "large enough".

In words. No state outside the kernel may be worth more than the worst state inside it. In discrete time the transition that leaves the kernel earns a one-step reward of at most $R_{Q_U}$, while the safe alternative at the same state earns at least $R_{Q_V}$; these two one-step rewards are collected before the discounted continuation, hence the extra terms and the factor $\gamma$ in (Zb). The condition is called "zeroth-order" because it compares values, not derivatives, of $V_p$ at different states.

Why each ingredient matters. The supremum on the left ranges over the whole unviability kernel, failure states included (but not the post-failure sink, which is bookkeeping and not a state of $X$), because an unsafe controller can wander anywhere in $X_U$ before failing. The infimum on the right is over the constrained value $V$, not $V_p$: the safe alternative we compare against must itself be safe. The inequality is strict so that unsafe actions are strictly suboptimal, which is what "all maximisers are safe" needs; with equality some optimal controller could be indifferent and fail.

The penalty should be big enough

The point of Lemma 2 is (ii): the supremum of $V_p$ on the unviability kernel can be pushed arbitrarily low by increasing $p$, while $\inf_{X_V} V$ does not depend on $p$ at all. So (Z) can always be enforced.

Reading the formula

Four things are now established: the threshold is finite; all maximisers of (P) for $p \gt p^\star$ are safe; they are also optimal for (C); and $p^\star$ depends explicitly on the time-to-failure, the reward and the discount. For $p \gt p^\star$ the penalised and the constrained problem are equivalent. The dependence has two parts.

• The exponential. $e^{T_f/\tau} = \gamma^{-T_f}$ is the inverse of the discount that the penalty suffers before it arrives: a penalty paid $T_f$ steps after the mistake is only worth $\gamma^{T_f} p$ at the moment the decision is made, so it must be inflated by $\gamma^{-T_f}$ to have any effect. If the agent is short-sighted, $\tau \ll T_f$, the required penalty is astronomically large. The paper's phrase: the ratio $T_f/\tau$ measures how the time-scale of failure compares to the agent's inner horizon.
• The bracket. $R_{X_U}\tau - \inf_{X_V} V$ (continuous time) is how attractive the doomed region can look: the best reward rate available while failing, integrated over the horizon $\tau$, minus the worst value that safety can guarantee. Positive rewards outside $X_V$ raise $R_{X_U}$ and $R_{Q_U}$; negative rewards inside $X_V$ lower $\inf_{X_V} V$ and $R_{Q_V}$; both raise $p^\star$.

Recovering the kernel and a barrier function from an SVF

Proposition 3 applies this theorem with $C = X_V$. The definition of $X_V$ supplies the invariance, but the SVF does not supply the defining function. For $\alpha \lt \alpha_{\sup}$ the function $h = V_p|_{X_V} - \alpha$ is strictly positive on $\partial X_V$, so $\{h = 0\} \neq \partial X_V$, and every continuous extension of $h$ stays positive just outside $X_V$. The identity $X_V = \{h \ge 0\}$ holds only because $h$ is defined on $X_V$ alone. There, the CBF inequality $\sup_u [L_{f_0} h + L_g h\,u] \ge -\kappa(h)$ holds trivially, because $h$ is bounded away from $0$ on the compact $X_V$ and a steep extended class-$\mathcal{K}_\infty$ function $\kappa$ absorbs any decrease. At a true boundary point, where $h = 0$, the condition would demand $\dot h \ge 0$; here $h \gt 0$ on $\partial X_V$, so it allows $\dot h \lt 0$ there and need not exclude outward inputs, and the resulting filter need not keep the state in $X_V$. The natural extension $V_p - \alpha$ does have $X_V$ as its 0-superlevel set on all of $X$ (Proposition 2), but it jumps across $\partial X_V$ (see the caveat on (Z)) and is not $C^1$. A CBF that represents the whole kernel therefore needs more than the listed assumptions: a $C^1$ function on a neighbourhood of $X_V$ with $X_V = \{h \ge 0\}$ and $h = 0$ exactly on $\partial X_V$ (for $h = V_p|_{X_V} - \alpha$ this forces $\alpha = \alpha_{\sup}$, with $V$ equal to its minimum on all of $\partial X_V$ and larger inside), plus the regularity of the dynamics that the CBF theorems assume. When such an $h$ exists, note the asymmetry with usual CBF practice: CBFs certify a conservative subset of $X_V$ chosen a priori, whereas here the level set is the whole kernel. Choosing $\alpha \gt \alpha_{\sup}$ selects a strict subset that need not be controlled invariant at all (the paper's Fig. 6 shows such a conservative subset, for which no CBF is guaranteed).

What the numerical examples teach about reward shaping

The paper's second example is a "satellite" $\dot x_1 = x_2$, $\dot x_2 = -g/x_1^2 + \omega^2 x_1 + u$ with $u \in [-1, 1]$, $g = 10$, $\omega = 0.1$, failure set $\{x_1 \le 1\} \cup \{x_1 \ge 15\}$, discretised on a $401 \times 301$ grid with a one-second zero-order hold (the input is held constant over each one-second interval, Primer D; on the resulting finite grid Assumption 1 holds automatically, see below) and solved by value iteration (see Primer E) with $\gamma = 0.6$. Three reward functions for the task "reach the equilibrium $(10, 0)$" illustrate the three ways a reward can interact with $p^\star$.

Why Assumption 1 is automatic on a finite deterministic grid: a non-failure cycle reachable while remaining in $X_U$ would allow a controller to repeat that cycle forever, contradicting unviability. Thus the non-failure part of the unviable transition graph has no cycle, so every path through it visits each state at most once and reaches $X_F$ within $|X_U|$ steps, whatever the inputs: $T_f \le |X_U|$. The argument does not carry over unchanged to stochastic transitions.

5. Viability of Future Actions and Entropy Regularisation

Practitioners train constrained tasks with SAC (see Primer E), whose target policy is a Boltzmann (softmax) distribution over soft Q-values with full support: no action ever gets zero probability. That breaks the SVF story in an obvious way: a stochastic policy that sometimes picks an unviable action cannot be safe. Massiani, von Rohr, Haverbeck & Trimpe, ECML-PKDD 2025 ask two questions about the entropy-regularised constrained problem: in what sense is it a robust control problem, and can penalties still make it amenable to unconstrained algorithms? The theory is for discrete-time deterministic dynamics $x' = f(x,a)$ on finite state and action spaces, with stochastic policies $\pi(a \mid x)$ and discount $\gamma \in (0,1)$.

Why entropy prefers states with many viable actions

The whole mechanism rests on one elementary inequality. A safe policy at $x \in X_V$ can only put mass on viable actions, and the entropy of a distribution supported on $|Q_V[x]|$ points is at most $\ln|Q_V[x]|$:

where the last equality is the counting-measure safety measure of Section 2. $\bar S$ is the discounted sum of these entropies along trajectories, so an entropy-regularised safe optimiser is drawn away from states with few viable actions. It does not forbid the actions leading there, because a hard exclusion would propagate the same cap backwards to earlier states and cost immediate entropy; it merely lowers their probability, the more so the sooner the constrained states would be reached. The paper's interpretation: the mode of such a policy tends to minimise the long-term proportion of actions that are unavailable because of constraints, and therefore the chance that action noise picks one of them. This is offered as an empirical explanation of the robustness of SAC-style agents; a precise formalisation is explicitly left to future work.

Penalties again, but only $\delta$-safety

The constrained problem involves $Q_V$, which is unknown in a model-free setting. The Lagrangian relaxation of $\pi \in \Pi_V$ is the penalised problem

with $c$ a dynamic indicator and $p \ge 0$. For $\alpha = 0$ this is (P) and Theorem 2 of Section 4 applies. For $\alpha \gt 0$ the softmax gives $\pi^\star_{\alpha,p}(a \mid x) \gt 0$ for every pair, so, as soon as $Q_{\mathrm{crit}} \neq \emptyset$, $\pi^\star_{\alpha,p} \notin \Pi_V$ for every finite penalty. (If $Q_{\mathrm{crit}} = \emptyset$, every action at every viable state is viable and every policy is safe, so there is nothing to enforce.) What survives is approximate safety.

The proof is short enough to be worth doing in full, and it shows where the temperature enters the required penalty. The core is a two-sided bound on the soft Q-function (Lemma 2 of the paper, a generalisation of Lemma 2 in Section 4).

What the experiments show

On a cliff-walking grid with three cliff cells (failure set), a target column with invariant states, and reward $-1$ for every action outside cliff and target, the constrained "fenced" version offers only three actions next to the cliff, so the entropy cap bites there. Maximising entropy alone drives the mode away from the cliff towards the top corners, even giving up the target when it starts on the left; finite temperatures trade this robustness against reaching the target. With penalties, sufficient $p$ recovers the constrained solution, consistent with Theorem 2, but entropy and penalty compete: for any fixed $p$ a high enough $\alpha$ degrades safety, so the penalty must scale with the temperature (Fig. 3 of the paper: the minimum penalty for a safe mode increases with $\alpha$, from about $9$ at $\alpha = 1$ to about $17$ at $\alpha = 32$; the achievable $\delta$ decreases in $p$, and beyond $p \approx 15$ it is larger at higher $\alpha$, while at small penalties the order is reversed). On two benchmarks (a modified Pendulum-v1 and Hopper-v4, trained with SAC and penalties) the paper evaluates the mode of the learned policy under injected uniform action noise and reports that modes trained at higher temperature avoid failure under larger noise, which is the claimed robustness; it is an empirical finding, not a theorem. Two remarks from the paper are worth keeping: with the plain indicator as dynamic indicator, the minimum penalty scales exponentially with the longest trajectory in $X \setminus (X_V \cup X_C)$, the discrete cousin of $e^{T_f/\tau}$; and while theory says "as high as possible", very large penalties cause numerical instabilities with function approximators.

6. Uncertainty-Aware Safe RL at DSME (UPSi, Dyna-SAuR, CHEQ)

The theory above assumes the dynamics are known and deterministic. The group's recent algorithmic work keeps the viability idea but replaces the model by a learned, uncertainty-aware one, and replaces "stay in $Q_V$" by "stay in the part of $Q_V$ where the model can be trusted". This section is a factual summary; the filter mechanics are treated in Module 10.

UPSi: a predictive safety filter on a probabilistic ensemble

Frauenknecht, Kesper, Mayfrank, Hose & Trimpe, RLC 2026 build a predictive safety filter (MPC background: Primer D; the filter of Wabersich & Zeilinger in Module 10) on the probabilistic-ensemble (PE) neural dynamics model (see Primer E) used by Dyna-style MBRL, for dynamics $s_{t+1} = \mu(s_t,a_t) + L(s_t,a_t)w_t$ with bounded process noise $w_t \in W$. The filter solves, at every step,

where $R_{n|t}$ is the robust reachable set of the plan ($n|t$ reads "predicted $n$ steps ahead from the information at time $t$"; $N$ is the horizon, $j$ counts model-update episodes, and $R_{0|t} = \{s_t\}$), $v_{n|t} = u_{n|t} + K_{n|t}(s - z_{n|t})$ the input with ancillary feedback around the nominal state $z_{n|t}$ ($v_{0|t} = u_{0|t}$; because this input depends on the state, $R_{n|t} \times v_{n|t}$ is shorthand for the graph $\{(s, v_{n|t}(s)) : s \in R_{n|t}\}$, so every state the tube may contain must yield an admissible and certain pair), $T_j$ a robustly positive invariant terminal set (for some control law $\kappa$, every $s \in T_j$ satisfies $\mu(s,\kappa(s)) + L(s,\kappa(s))W \subseteq T_j$, the paper's Definition 2: the robust counterpart of controlled invariance, so the tube can end there and stay safe forever), and the only non-standard ingredient is the certainty constraint $C_j$. (The paper's compact statement (4) indexes the stage constraints from $n = 1$; its full formulation (11) enforces them from $n = 0$, which in particular requires the applied pair $(s_t, u_{0|t})$ to be admissible and certain.) The ensemble members $e = 1, \dots, E$ predict means $\mu_{\theta_e}$ and covariances $\Sigma_{\theta_e}$ (see Primer C); following the group's earlier work these are fused into an aleatoric variance $\bar\Sigma = \big(\tfrac{1}{E}\sum_e \Sigma_{\theta_e}^{-1}\big)^{-1}$, a nominal model $\bar\mu$ and an epistemic variance $\hat\Sigma = \tfrac{1}{E}\sum_e (\mu_{\theta_e} - \bar\mu)(\mu_{\theta_e} - \bar\mu)^\top$ (the spread of the members). The Kalman-gain-like matrix $G = \bar\Sigma(\bar\Sigma + \hat\Sigma)^{-1}$ (a gain matrix, not the return $G$ of Section 3) is the ratio of aleatoric to total predictive variance, and the certain set is

which, with $n_S$ the state dimension, approaches $1$ where epistemic uncertainty is negligible and $0$ under high model uncertainty (eq. 5 of the paper). Reachable sets are over-approximated by ellipsoidal tubes $\mathcal{P}_{n|t} = \{s : (s - z_{n|t})^\top P_{n|t}^{-1}(s - z_{n|t}) \le 1\}$: the centre follows $\bar\mu$, the shape is propagated through the averaged Jacobians with an ancillary feedback $K_{n|t}$, an outer ellipsoid of the Minkowski sum ($A \oplus B = \{a + b : a \in A,\ b \in B\}$; Primer D) with the truncated noise ellipsoid $\epsilon\bar\Sigma$, and an explicit inflation by a Lipschitz bound on the linearisation error. Theorem 1 of the paper (stated precisely below): under Assumptions 2, 3 and 5, $R_{n|t} \subseteq \mathcal{P}_{n|t}$ for all $n$. Assumption 2 (consistent estimator: $\Sigma_{\theta_e} \succeq \Sigma$ inside $C_j$) and Assumption 3 (unbiased inside $C_j$) only hold inside the certain set, which is why certainty must be enforced along the whole predicted tube; this is done by tightening $C_j$ with a Lipschitz margin $g_{n|t} = \ell_G\sqrt{\lambda_{\max}(P_{n|t}) + \lambda_{\max}(K_{n|t}P_{n|t}K_{n|t}^\top)}$. Assumption 4 (the noise estimate improves monotonically and $C_j \subseteq C_{j+1}$) makes the sampled terminal-set expansion valid across episodes; Assumption 5 asks for twice-differentiable nominal dynamics with Lipschitz gradient, noise scale and certainty measure.

Why $\tfrac{1}{n_S}\operatorname{tr}G\in[0,1]$ (Primer A): assume $\bar\Sigma\succ0$, $\hat\Sigma\succeq0$ and write $\Sigma_{\mathrm{tot}}=\bar\Sigma+\hat\Sigma$. Since $0\preceq\bar\Sigma\preceq\Sigma_{\mathrm{tot}}$, the symmetric matrix $\Sigma_{\mathrm{tot}}^{-1/2}\bar\Sigma\,\Sigma_{\mathrm{tot}}^{-1/2}$ lies between $0$ and $I$, so its $n_S$ eigenvalues lie in $[0,1]$; by cyclicity of the trace, $\operatorname{tr}G=\operatorname{tr}(\bar\Sigma\,\Sigma_{\mathrm{tot}}^{-1})=\operatorname{tr}(\Sigma_{\mathrm{tot}}^{-1/2}\bar\Sigma\,\Sigma_{\mathrm{tot}}^{-1/2})$ is their sum. ($G$ itself need not be symmetric, hence the detour; in one dimension $G=\bar\sigma^2/(\bar\sigma^2+\hat\sigma^2)$.) The inverses in the ensemble formulas need positive-definite member covariances or a stated regularisation.

Here $P_{n|t}$ is the shape matrix of the ellipsoid. At $n=0$ it is $0$ and the tube is the single point $s_t$ (for a singular shape read $\mathcal{P}=\{z+P^{1/2}q:\|q\|_2\le1\}$, since the inverse in the formula needs $P\succ0$). The paper's noise set is the ball $W=\{w:\|w\|_2^2\le\epsilon\}$ with $L$ diagonal, so $\epsilon$ is a noise radius (not the explorer's slip probability), and the noise $Lw$ lies in the ellipsoid with shape $\epsilon LL^\top=\epsilon\Sigma\preceq\epsilon\bar\Sigma$ under Assumption 2.

The certainty margin: write a state of the tube as $s=z+P^{1/2}q$ with $\|q\|_2\le1$. Then $\|s-z\|_2^2=q^\top Pq\le\lambda_{\max}(P)$ and $\|v(s)-u\|_2^2=\|KP^{1/2}q\|_2^2\le\lambda_{\max}(KPK^\top)$, so the pair $(s,v(s))$ lies within distance $\sqrt{\lambda_{\max}(P)+\lambda_{\max}(KPK^\top)}$ of $(z,u)$. A certainty measure with Lipschitz constant $\ell_G$ can therefore drop by at most the margin $g_{n|t}$ above, and requiring the nominal pair to exceed $\xi$ by $g_{n|t}$ certifies the whole tube.

Dyna-SAuR: a hyperplane safety filter learned inside a Dyna loop

Eisele, Frauenknecht, Solowjow & Trimpe, 2026 learn both a control policy and a scalable safety filter from an uncertainty-aware model, with minimal domain knowledge. Their viability notion is finite-horizon and probabilistic (see Primer C): $S_V = \{s : \exists\pi,\ \Pr_\pi[\forall t \le T,\ S_t \in S_S \mid S_0 = s] \ge 1 - \delta\}$ with $S_S$ the non-failure states (unbounded process noise rules out infinite-horizon statements), and it is intersected with a certain set $E = \{(s,a) : H(\hat S_{t+1}) \le \lambda_1\}$ defined by the entropy of the model's predictive distribution, giving a certain viability kernel $S_V(E)$ and the certain viable policies $\Pi_V(E)$. The filter is the projection onto a state-dependent half-space of actions (see Primer B),

with $A = [-1,1]^{n_A}$ ($n_A$ the action dimension), a filter policy $\mu$ with values in the punctured unit ball $\mathcal{U}_h = \{u \in \mathbb{R}^{n_A} : 0 \lt \|u\|_2 \le 1\}$ (the paper's $\mathcal{U}$; renamed here to avoid a clash with the controller set $\mathcal{U}$ of Sections 1–4), and the map $h(u) = (w, b)$, $w = u/\|u\|_2$, $b = (2\|u\|_2 - 1)\|w\|_1$. Theorem 5.1 calls $h$ a bijection onto the hyperplanes that intersect $A$. Precisely, $h$ is a bijection from $\mathcal{U}_h$ onto the oriented pairs $\{(w, b) : \|w\|_2 = 1,\ -\|w\|_1 \lt b \le \|w\|_1\}$, i.e. onto half-spaces $w^\top a \ge b$; the pairs $(w, b)$ and $(-w, -b)$ give the same hyperplane but opposite half-spaces. The endpoint $b = -\|w\|_1$ is missing. Its half-space contains all of $A$ (no filtering), and it would need $u = 0$, which the paper's definition (13) excludes and where $h$ is undefined; the appendix proof uses the closed ball, and puncturing it removes the division by zero at the cost of this endpoint. $\|u\|_2$ near $0$ classifies almost all actions as viable and near $1$ almost none, a minimal search space for $\mu$. The paper motivates the half-space by saying that for control-affine dynamics a state-dependent hyperplane discriminates viable from unviable actions (citing Lavanakul et al.). What Lavanakul et al. show is weaker. For continuous-time control-affine dynamics and a control-invariant set with $C^1$ boundary, the tangency (Nagumo/CBF) condition (see Primer D) is affine in the input, so a state-dependent half-space can serve as a sufficient, possibly conservative, safety constraint that separates certified from uncertified inputs. The exact set of viable actions need not be a half-space, least of all in the discrete-time setting used here: for $x' = 2a$ with $A = [-1, 1]$ and safe states $[-1, 1]$, the viable actions are $[-\tfrac12, \tfrac12]$, which no single half-space intersected with $A$ represents. The hyperplane filter is therefore in general an approximation of the viable actions, at best a conservative (inner) one. The filter policy is trained purely on model rollouts in a "safety filter MDP" with reward $+1$ per step inside the certain safe set, $-1/(1-\gamma_{\mathrm{SF}})$ and termination when leaving it, and a regulariser $c\|\mu(s)\|_2$ in its objective that pushes towards the least restrictive hyperplane; both $\mu$ and $\pi$ are retrained from scratch in each Dyna iteration, and a better model enlarges $E$ and hence makes the filter less conservative. Reported results: control performance matching or exceeding safe-RL baselines on goal-reaching CartPole (constraints on pole angle and cart position) and on par with PPO-Lagrangian on MuJoCo Walker, with accumulated training failures reduced by at least two orders of magnitude compared with the other methods.

Why these endpoints: over the box $A=[-1,1]^{n_A}$, $\max_{a\in A} w^\top a=\sum_i|w_i|=\|w\|_1$ (take each coordinate with the sign of $w_i$) and the minimum is $-\|w\|_1$, so the hyperplane $w^\top a=b$ meets $A$ iff $-\|w\|_1\le b\le\|w\|_1$. The inverse of $h$: for an allowed pair ($\|w\|_2=1$, $-\|w\|_1\lt b\le\|w\|_1$) put $r=(1+b/\|w\|_1)/2\in(0,1]$ and $u=rw$; then $\|u\|_2=r$, $u/\|u\|_2=w$ and $(2\|u\|_2-1)\|w\|_1=b$, so $h(u)=(w,b)$, and $u$ is the only preimage because $h(u)$ fixes both the direction and the norm of $u$.

CHEQ on hardware, and safe RL in uncertain contexts

Cramer, Jäschke & Trimpe, IFAC-PapersOnLine 2025 give the first hardware evaluation of CHEQ, an adaptive hybrid RL method that blends a control prior with an RL action, $a^{\mathrm{ref}}_t = (1 - \lambda^{\mathrm{RL}}_t)a^{\mathrm{prior}}_t + \lambda^{\mathrm{RL}}_t a^{\mathrm{RL}}_t$, where the weight $\lambda^{\mathrm{RL}}_t$ is a clipped linear map of the parametric uncertainty of a critic ensemble (the standard deviation of the ensemble's Q-estimates (see Primer E)): the agent is trusted where its critics agree. Applied to robotic polishing with variable impedance, a task requiring precise force and velocity tracking, CHEQ learned effective polishing directly on hardware in eight hours of training with only five failures. The paper stresses that CHEQ gives no formal safety guarantee: the control prior shapes exploration, and hard limits truncate episodes. (This $\lambda$ is a blending weight, not a Lagrange multiplier.) Finally, Baumann & Schön, T-RO 2024, by a Trimpe alumnus and not co-authored by Trimpe, extend SafeOpt-style safe learning to discrete environmental contexts that cannot be measured (payloads, surfaces): contexts are identified through MMD-based tests on dedicated identification experiments when they cannot be inferred from measurements, and estimated online with frequentist multi-class classification bounds built on conditional mean embeddings, validated on a Furuta pendulum whose weights are classified from camera images.

7. Connections: Lagrangians, Reachability, Shields

The penalty is a Lagrange multiplier

Write (C) as the CMDP "maximise $\mathbb{E}_\mu G$ subject to $\mathbb{E}_\mu \rho \le 0$". Because $\rho \ge 0$, this averaged constraint demands safety only from $\mu$-almost every initial state (why: if $\Pr_\mu(\rho\gt0)\gt0$, then some event $\{\rho\ge1/n\}$ has positive probability, because these events increase to $\{\rho\gt0\}$, and then $\mathbb{E}_\mu\rho\ge\tfrac1n\Pr_\mu(\rho\ge1/n)\gt0$; so $\mathbb{E}_\mu\rho=0$ forces $\rho=0$ except on a set of initial states of probability zero, Primer C); the paper's constraint "$\rho(x,u) = 0$ for all $x \in X_V$" (its $(C_\eta)$ with $\eta = 0$) is the pointwise version, and it is the one the SVF theorem speaks about. Form the Lagrangian with multiplier $\lambda \ge 0$:

The penalised problem (P) with $p = \lambda$ is the inner maximisation of the Lagrangian for a fixed multiplier, and the dual function is the expected penalised value (the supremum can be taken inside the expectation because the penalised problem has a controller that is optimal from every initial state at once, as assumed in Section 3). Weak duality, $d(\lambda) \ge$ the constrained optimum, is Lemma 3's inequality (34). Now compare the two theories. Strong duality for CMDPs (Paternain et al., NeurIPS 2019; Module 8) says $\min_\lambda d(\lambda)$ equals the primal optimum and that some maximiser of $\mathcal{L}(\cdot, \lambda^\star)$ is feasible and optimal; it says nothing about the other maximisers, which is the primal recovery problem that motivates state augmentation (Calvo-Fullana et al., TAC 2024; Module 8). The SVF theorem is stronger and narrower: for the 0-risk constraint and every fixed $\lambda \gt p^\star$, every optimal controller of $V_\lambda$ is safe from every viable state and optimal for (C), and every maximiser of the averaged Lagrangian $\mathcal{L}(\cdot,\lambda)$ is safe from $\mu$-almost every initial state and optimal. The dual optimal set therefore contains the whole ray $(p^\star, \infty)$; it can be larger, since $p^\star$ is only sufficient. This is the mechanism behind a familiar observation: the same fixed penalty that works on one task fails on another. In the exact, deterministic setting a fixed multiplier works once it exceeds the task's minimal penalty, which Theorem 2 bounds through the time-to-failure, the reward scale and the discount; for PPO or SAC with function approximation, and with entropy regularisation (Section 5), this is a heuristic reading rather than a guarantee. Projected dual descent on the multiplier, $\lambda \leftarrow [\lambda + \eta_\lambda\,\mathbb{E}_\mu\rho]_+$ with step size $\eta_\lambda$ (RCPO, PPO-Lagrangian, Module 8), can be read as a search for a sufficient penalty: with threshold $0$ and $\rho \ge 0$ the multiplier never decreases and stops moving as soon as the current maximiser is safe from $\mu$-almost every initial state, and a PID-controlled multiplier (PID control: Primer D; Module 8) damps the overshoot of that search. For a risk budget $\eta \gt 0$ the SVF equivalence fails (Example 1) and the multiplier genuinely has to be found.

Viability is reachability is invariance

Three communities compute the same set. The unviability kernel $X_U$ is the backward-reachable tube of $X_F$ in HJ reachability (Module 10), obtained from the min-over-time value $V_{\mathrm{BRT}}$ and used as a least-restrictive safety filter that intervenes only at $\partial X_V$; SVFs put the same set into a Lagrange-type value via Proposition 2. $X_V$ is the largest controlled-invariant subset of $X \setminus X_F$; CBFs (Module 10) certify invariance of a chosen subset $\{h \ge 0\}$, and Proposition 3 claims that an SVF furnishes a CBF for the whole kernel under regularity, which in fact also needs a $C^1$ defining function vanishing exactly on $\partial X_V$ (Section 4). And a shield (Alshiekh et al., AAAI 2018; Module 10) is synthesised from the winning region $W \subseteq F^g$ of a safety game on a finite abstraction ($F^g$ the safe game states). The paper computes $W$ by standard safety-game solving, i.e. as the greatest fixed point $W = \nu X.\,F^g \cap \mathrm{CPre}(X)$ with $\mathrm{CPre}(X)$ the states from which the system player can force the next state into $X$, which reduces to Algorithm 1 when every action has one successor: $W$ is the viability kernel of the abstraction, a preemptive shield restricts the learner to $Q_V[x]$, and a post-posed shield substitutes an action from $Q_V[x]$ when the learner leaves it. Shielded RL is therefore the "unconstrained MDP on $X_V$ with action set $Q_V[x]$" of Section 3, enforced online rather than through the reward. The HJ, CBF and predictive safety filters surveyed by Wabersich et al., CSM 2023 all override the learner's action so that it stays in (an inner approximation of) $Q_V[x]$, CBF and predictive filters by a minimal projection, HJ filters typically by switching to a safe action at the boundary; Dyna-SAuR's half-space is a learned such approximation.

Reading the table. In the SVF column the safe actions are the maximisers of the one-step look-ahead $Q_p(x,u)=r(x,u)-p\mathbf{1}_{X_F}(x)+\gamma V_p(f(x,u))$ (discrete time, sink convention), not of the state function $V_p$; thresholding $V_p$ itself recovers $X_V$ only under the value separation of Proposition 2. In the HJ column the dynamics carry a disturbance, $f(x,u,d)$ with $d\in D$, and at the boundary the filter picks $\arg\max_{u}\min_{d}\nabla V(x)^\top f(x,u,d)$, the input with the best worst-case rate. In the CMDP column the occupancy support only describes pairs that are actually visited; it certifies nothing at unvisited viable states.

Walkthrough: Deriving the Penalty Threshold p*

The derivation of the continuous-time threshold (18a) in six steps, with the reason for each step. It reproduces Lemma 2(ii) and Theorem 2 of Massiani et al. Keep the shelf example in mind: there the bound turns out to be exact.

Interactive: Penalty vs Safety on a Cliff Gridworld

An $8 \times 5$ cliff world. The bottom row is the cliff (failure set, absorbing, penalty $p$ on entry, then nothing; a cliff cell is itself a state of $X_U$ with value $-p$). Six shaded "slope" cells are unviable: on a slope the agent cannot climb and slides one row down after every move, so from the four slope cells next to the cliff every action fails in one step, and from the two slope cells above them every action fails within two steps (moving down falls at once, any other move slides onto the lower slope), so $T_f = 2$. All other cells are viable; the number in the corner is the safety measure $\Lambda$, the count of viable actions among up, down, left, right, stay. The agent pays a living cost $c$ per step away from the goal G and earns $R_g$ per step at the goal (it can stay there). The explorer runs numerical value iteration for the penalised problem, draws the greedy policy, flags every viable state from which some optimal action leaves the kernel, rolls out the greedy policy from S, and compares three thresholds: an estimate $p_{\min}$ of the first safety threshold, an estimate $p_Z$ of the boundary where the strict zeroth-order condition (Zb) holds, and the formula $p^\star$ of Theorem 2.

The reward is paid at the current state. If an action at time $t$ reaches a cliff cell at time $t+1$, its action value is $r(x_t)-\gamma p$: the failure state has value $-p$ at the next time index. Otherwise it is $r(x_t)+\gamma V_p(x_{t+1})$. Thus 'on entry' means at the arrival state's time, matching the sums in Exercise 7.4.

At each decision, execute the intended action with probability $1-\varepsilon$; with probability $\varepsilon$, draw uniformly from all currently available actions, including the intended one. If there are $m$ actions, the intended action therefore has total probability $1-\varepsilon+\varepsilon/m$, and every other action has probability $\varepsilon/m$. The value calculation averages over these outcomes. The drawn line instead follows nominal successors of selected greedy actions without sampling slip; it is an illustrative intended path.

Exercises

Graded practice — Build the argument yourself

There are 12 new problems: four Easy, four Medium, and four Hard. Start with the level that lets you make progress without opening the solution. Easy problems rebuild individual operations; Medium problems combine them; Hard problems ask for proofs, boundary cases, and the limits of a guarantee. The required concepts are explained on this page or in the linked earlier material.

Easy — Warm up one skill at a time.

Medium — Combine definitions and compute a certificate.

Hard — Explain why the argument works and where it stops.

Further practice — Original problems and research connections

The original exercises below retain their numbering. Some compare later methods or ask for longer research derivations; use the graded set above first, and return to a research-connection problem after reading the relevant linked module.

Key Papers

Flashcards