Open Problems & Research Gaps

What is still unsolved in safe learning as of autumn 2026, organised by research line. Each problem comes with a plain-language explanation, its notation and a worked example, and links to the place in these notes where its background is taught

1. How to Read This Page

Reading this for the first time? Choose one problem connected to a module you have studied. Read its plain-language explanation and worked example, then follow its background links before tackling the formal question. These are research directions; use the graded exercises in the primers and modules to check your understanding first.

The fifteen modules present what is known. This page collects what is not known: the problems that the papers behind the modules, and the surveys and reviews that summarise them, name as open.

Every problem has the same parts, in the same order:

• The question (blue box): the problem, as precisely as the current literature allows.
• In plain words: the same problem without formulas, and why anyone should care.
• Symbols: every symbol the problem uses, with its meaning. The complete list is in Section 2.
• What is known: the best current results, with citations.
• Why it is hard: the mathematical obstacle.
• Worked example (click to open): a small calculation with concrete numbers that shows the difficulty. Unless a paper is cited, the numbers are invented for illustration. Every number has been recomputed by a script.
• Where in these notes and Possible directions: the sections that teach the background, and routes that look promising. The directions are suggestions, not results.

Problems are numbered by research line:

• T: the Trimpe line, safe exploration and learning-based control;
• P: the Pauli line, certified networks via robust control;
• R: constrained deep RL;
• F: safety filters;
• S: statistical guarantees;
• B: bridges between the Trimpe and Pauli lines.

2. Notation Used on This Page

The problems come from five literatures, and each has its own notation. The table lists every symbol used below, what it means, a tiny example, and where it is taught. Keep this section open in a second tab while reading.

Interactive: Problem Map

Each row is an open problem, each column a module. A dot means the problem builds on that module. Filter by research line; click a row label to jump to the problem, or a column header to open the module.

3. Safe Exploration with Checkable Assumptions (Trimpe line)

The Trimpe group's recent work (RWTH Aachen, DSME) argues that a safety guarantee should rest only on assumptions a user can check. Their "On Safety in Safe Bayesian Optimization" paper made that concrete. They tested 100 functions with 10,000 runs each. SafeOpt with the common heuristic $\beta \equiv 2$ violated the safety constraint in 3.95% of runs on average, and in 28.62% of runs on the worst function (Fiedler et al., TMLR 2024). The problems below are what remains after that analysis.

T1Checkable regularity and noise assumptions

In plain words. Safe BO decides that an untested input is safe by reasoning: "the function cannot change too much between the points I measured and this new point". To make "too much" precise, SafeOpt assumes the function is not too wiggly, in a sense set by the kernel, and that you know a number $B$ that bounds this wiggliness. Nobody can look at a robot and read off $B$. Guess it too small and the guarantee silently fails; guess it too large and the algorithm is so cautious that it barely explores.

What is known.

• LoSBO replaces the RKHS bound by a Lipschitz constant $L$ and a hard noise bound $E$ (Fiedler et al., TMLR 2024). Gaussian noise, the default noise model, violates any hard bound $E$.
• Data-driven over-estimates of the RKHS norm exist, but they need strong assumptions: Tokmak et al., AISTATS 2025; the PAC variant PACSBO (Tokmak, Schön & Baumann, SysDO 2024); and a 2026 sampling-based estimator based on superconvergence (Wenzel, Tokmak & Fiedler, arXiv 2026).
• Adapting kernel hyperparameters during a run invalidates standard bounds. Capone, Lederer & Hirche (ICML 2022) handle unknown hyperparameters under a hierarchical Bayesian model.

Why it is hard. No finite data set can bound $\|f\|_k$ from above. If the RKHS is infinite-dimensional, some nonzero function $g$ vanishes at every sampled point: take any direction orthogonal to $\operatorname{span}\{k(\cdot,x_1),\dots,k(\cdot,x_n)\}$, since $g(x_i) = \langle g, k(\cdot,x_i)\rangle = 0$. Then $f + c\,g$ fits the data exactly for every number $c$, and its norm grows without bound as $c$ grows. Any norm estimate therefore needs an extra assumption about which functions are plausible.

Where in these notes. Module 3 §7, Module 4 §3, Module 5 §1, Module 5 §7.

Possible directions. Derive Lipschitz or Hölder moduli from physics or from certified models (see B1). Use bounded-noise models calibrated from sensor specifications. Build confidence sets that degrade gracefully under misspecification rather than failing silently.

T2Exploration guarantees for Lipschitz-only safe BO

In plain words. LoSBO is safe for a simple reason: a function with Lipschitz constant $L$ cannot drop faster than slope $L$, so one good measurement certifies a whole interval around it. But being safe is not the same as making progress. The safe region only grows if the algorithm measures near its edge and finds good values there. LoSBO lets a GP choose where to measure, and nothing in the safety proof forces that choice to push the edge outward.

What is known. Safety is proved (Proposition 1 of Fiedler et al., TMLR 2024). The authors point out that SafeOpt's exploration proof relies on the GP model interacting with the safety mechanism, which is exactly what LoSBO removes. They suspect that pathological situations exist in which LoSBO fails to explore properly, although none appeared in their experiments, and they leave (conditional) exploration guarantees to future work.

Why it is hard. In SafeOpt the safe set grows exactly when the GP lower bound at an expander rises. Uncertainty sampling shrinks that bound's width, and this coupling is the engine of the proof. In LoSBO the safe set grows only from measured values minus $E$. The GP decides where to sample but no longer controls how the safe set grows, so the old argument has nothing to hold on to.

Where in these notes. Module 4 §3, Module 5 §3.

Possible directions. Prove exploration guarantees that hold only when the GP happens to be well specified, while keeping safety unconditional. Design acquisition rules that force expansion from the Lipschitz cones alone.

T3Tight online confidence bounds and the price of safety

In plain words. Regret measures the total price of learning: how much worse your choices were, summed over all rounds, than always picking the best input. A good algorithm has regret that grows more slowly than the number of rounds, so its average loss per round goes to zero. The open questions are how fast regret can shrink with GP models, and how much extra a hard safety constraint costs, since it may stop the learner from trying the inputs that would teach it the most.

What is known.

• Whether confidence intervals for adaptively collected data can be made as tight as for independent data was posed as a COLT open problem (Vakili, Scarlett & Javidi, COLT 2021).
• The standard analysis of GP-UCB gives regret $O(\gamma_T\sqrt T)$. That is a factor $\sqrt{\gamma_T}$ worse than the $\tilde O(\sqrt{\gamma_T T})$ of more complicated algorithms, and for Matérn kernels it does not even guarantee sublinear regret.
• Whitehouse, Ramdas & Wu (NeurIPS 2023) show that GP-UCB with carefully chosen regularisation has sublinear regret for every kernel with polynomial eigendecay, which partially resolves the problem. Flynn & Reeb (AISTATS 2025) tighten the bounds further.
• Under hard per-step safety, most safe-BO results bound simple regret or sample complexity relative to the reachable safe set. Cumulative regret is understood only with extra structure. For linear rewards and constraints with a known safe action, Safe-LUCB has regret $\tilde O(\sqrt T)$ when a positive safety gap is known and $\tilde O(T^{2/3})$ otherwise, and its authors leave open whether $T^{2/3}$ can be improved (Amani et al., NeurIPS 2019). With a known safe baseline arm, Khezeli & Bitar (AAAI 2020) obtain $O(\sqrt T\log T)$.

Why it is hard. Even if the measurement noise is independent, each new input is chosen using earlier noisy measurements, so the inputs depend on the noise. Confidence bounds must therefore use martingale and self-normalised arguments, which pay for this dependence. Hard safety also stops the learner from sampling where information is highest, so the usual exploration arguments break.

Where in these notes. Module 3 §5, Module 4 §6, Primer C (martingales), Primer E (regret).

T4Rigorous safety with learned, high-dimensional models

In plain words. GP-based safe exploration works on small problems, such as tuning five controller gains. Modern robots learn neural-network models of their own dynamics, in hundreds of dimensions. To keep a safety guarantee you must know how wrong the learned model can be everywhere the robot might go. Neural networks do not come with such error bounds. The usual stand-in, disagreement within an ensemble of networks, can be small exactly where all members are wrong in the same way.

What is known.

• UPSi (Frauenknecht et al., RLC 2026) builds a predictive safety filter on a probabilistic ensemble. Its guarantees assume, among other things, that inside a "certain set" the ensemble never underestimates the process noise and has negligible bias, and that its noise estimate and certain set improve monotonically from episode to episode. None of these is easy to verify.
• Dyna-SAuR (Eisele et al., arXiv 2026) reports two orders of magnitude fewer training failures than state-of-the-art baselines on CartPole and MuJoCo Walker. Its filter restricts actions to a learned, state-dependent halfspace, a form the paper motivates with control-affine dynamics. A single halfspace need not describe the set of safe actions exactly.
• GoSafeOpt (Sukhija et al., AIJ 2023) obtains its backups from the Markov property, so it needs measurements of the full state (noise-free in the paper's main analysis).
• HdSafeBO (Wei et al., CoRL 2024) scales to problems with a few hundred dimensions, under probabilistic safety constraints.

Why it is hard. A safety guarantee needs a uniform error bound on the model over every state the system might visit. Neural networks come with no such bound: ensembles can agree with each other and all be wrong.

Where in these notes. Module 6 §5, Module 7 §6, Module 10 §9.

Possible directions. Use certified model classes, for example Lipschitz-bounded networks (Module 13 §1), so that model error can be propagated rigorously. Combine them with statistical validation on held-out rollouts (Module 15 §6).

T5Safety when the system changes

In plain words. A safe set learned yesterday may be unsafe today: friction changes, a payload is added, a part wears out. A learner must either notice the change before it matters, or keep a margin that covers every change that could have happened since its last measurement. Both are possible only if you know something about how fast, or how often, the system can change.

What is known.

• ET-GP-UCB (Brunzema et al., TMLR 2025) and event-triggered SafeOpt (Holzapfel, Brunzema & Trimpe, L4DC 2024) reset data and safe sets when a trigger based on a GP error bound fires. The triggers test consistency only at queried inputs, and they trade detection delay against false alarms. In the quadcopter experiments the trigger threshold is additionally rescaled by hand (Eq. 7 of Holzapfel et al.).
• TVSafeOpt (Li et al., NeurIPS 2024) avoids explicit change detection. It models time with a spatio-temporal kernel, lets the safe set shrink, and proves high-probability safety. This needs a known bound on how fast the functions change, and Lipschitz constants in both input and time. Its near-optimality guarantee holds only once the problem becomes stationary again.

Why it is hard. A change can happen where you are not sampling, and safety requires noticing it before it matters. Guarantees that let the safe set shrink exist when the rate of change is known. Drift without a known rate, and abrupt changes away from the sampled inputs, have no satisfactory treatment yet.

Where in these notes. Module 5 §6, Primer C (hypothesis tests).

T6A theory of robust safety in RL

In plain words. One way to make an RL agent safe without writing explicit constraints is to punish failure: a large negative reward $-p$ when the system crashes. If $p$ is large enough, an optimal agent never chooses an action that leads to a crash. The question is how large is large enough. Future rewards are discounted, so a crash that happens late counts for little. When the crash comes long after the decisive mistake, the penalty therefore has to be enormous.

What is known.

• Suppose every trajectory that leaves the viability kernel fails within a uniform time $T_f$. Then a finite penalty threshold $p^\star$ exists: any larger failure penalty makes the optimal policies safe. The sufficient threshold that Massiani et al. derive grows like $e^{T_f/\tau}$ in continuous time, or $\gamma^{-T_f}$ in discrete time, and $T_f$ is usually unknown (Massiani et al., TAC 2023).
• With entropy regularisation no finite penalty is exact, and only a relaxed ("$\delta$-") notion of safety is available (Massiani et al., ECML-PKDD 2025).
• The robustness of SAC-style agents to action noise is documented empirically, not proved.

Why it is hard. A penalty paid at the moment of failure is discounted by the time it takes to fail. The known sufficient thresholds therefore grow exponentially with $T_f$ relative to the discount time scale. Such large penalties make learning numerically hard and exploration conservative.

Where in these notes. Module 7 §4, Module 7 §5, Module 8 §3.

T7Benchmarks for safe controller tuning

In plain words. Papers evaluate their algorithms on their own test problems with their own metrics, so results are hard to compare. Safe tuning adds a twist: an algorithm that finds a slightly better controller but crashes the robot twice may be worse than a cautious one. Which one is "better" depends on what you measure.

What is known. The group's 2026 tutorial and review calls the lack of standardised benchmark problems for control applications a significant gap. It starts a lightweight benchmark suite for control engineering and robotics, and proposes metrics and best practices for comparing algorithms (Stenger et al., arXiv 2026, under review).

Why it is hard. Real hardware is slow and expensive, so benchmarks must be cheap simulations that still capture what makes control hard: noise, delays, constraints and crashes. And the metric itself is a design choice, as the example shows.

Where in these notes. Module 4 §5.

4. Certified Neural Networks via Robust Control (Pauli line)

Patricia Pauli's work (PhD in Stuttgart with Frank Allgöwer, now at TU/e) treats a neural network as a feedback system. Its nonlinearities are described by quadratic constraints, and its robustness is certified with LMIs, from LipSDP-constrained training to LipKernel. The open problems here are about making these certificates tighter, larger and usable for synthesis.

P1Certificates that are both tight and scalable

In plain words. A Lipschitz constant $L$ of a network bounds how much its output can change when its input changes: $\|f(x) - f(x')\|_2 \le L\,\|x - x'\|_2$. Computing the smallest such $L$ exactly is NP-hard, so we compute upper bounds instead. Cheap bounds are very loose. Tight bounds such as LipSDP solve a semidefinite program whose size grows with the number of neurons, and that soon becomes too expensive. The goal is bounds that are both tight and cheap for real network sizes and architectures.

What is known. LipSDP (Fazlyab et al., NeurIPS 2019) with diagonal multipliers remains the reference for tight polynomial-time bounds. Structure-exploiting reformulations extend its reach:

• chordal sparsity (Xue et al., CDC 2022);
• layer-wise recursions (ECLipsE, NeurIPS 2024);
• layer-wise LMIs built from state-space realisations of convolutions (Pauli, Gramlich & Allgöwer, arXiv 2024).

Kuang et al. (arXiv 2026) prove negative results for this whole family of certificates. They give explicit ReLU examples on which such bounds are provably loose, by factors that grow with the width, and link part of the difficulty to hard reachability questions about hidden states. Module 12 states their results precisely.

Why it is hard. An SDP over all neurons at once costs at least cubic time in the number of neurons. Attention is neither slope-restricted nor globally Lipschitz, so the quadratic constraints that describe ReLU or tanh do not apply to it.

Where in these notes. Module 12 §2, Module 12 §9, Module 13 §6.

Possible directions. Certify locally, over a given input set, rather than globally. Exploit reachable-set structure. Develop quadratic constraints for attention restricted to bounded inputs.

P2Richer multipliers for Lipschitz and contraction certificates

In plain words. LipSDP describes each neuron by one simple fact: the slope of its activation lies between $\alpha$ and $\beta$. Each fact gets a nonnegative weight (a multiplier), and the SDP searches for the best weights. Richer facts, such as facts that couple several neurons, or that look at how signals evolve over time, could make the bounds tighter. But every fact must hold for every pair of inputs, and some tempting couplings are simply false, as the example shows.

What is known.

• Dynamic multipliers such as acausal Zames–Falb multipliers are valid for stability around a fixed equilibrium, and they reduce conservatism for linear systems with neural-network nonlinearities (Pauli et al., CDC 2021).
• For generic slope-restricted activations applied elementwise, Lipschitz and contraction certificates still use diagonal incremental multipliers. The coupled static multipliers of the original LipSDP-Network variant are invalid in this setting, as two counterexamples by Pauli et al. (L-CSS 2022) show.
• Structured activations are an exception: GroupSort and Householder layers admit valid coupled incremental constraints within each group (Pauli et al., ICLR 2024).

Why it is hard. An incremental constraint compares the network at two arbitrary inputs. A coupling term between two neurons then treats a difference of two chords of the activation, taken at different places on its graph, as if it were a single chord, which nothing justifies (Module 12, §5).

Where in these notes. Module 2 §5, Module 12 §5, Module 12 §8, Module 14 §3.

P3Synthesis, not only analysis

In plain words. Checking that a given neural controller keeps a system stable (analysis) is now routine for small systems: you search for a Lyapunov certificate with an SDP. Designing the controller together with its certificate (synthesis) is harder, because the controller's weights and the certificate's matrices multiply each other. Products of unknowns make the problem non-convex, so the efficient SDP machinery no longer applies directly.

What is known. Analysis is mature (Module 14 §2). For synthesis, Junnarkar, Arcak & Seiler (Automatica 2026) project RL policies onto closed-loop dissipativity LMIs. Recurrent equilibrium networks are stable by design (Revay, Wang & Manchester, TAC 2024).

Why it is hard. Controller parameters and multipliers multiply each other in the certificate, which makes it a BMI. BMIs are non-convex and hard to solve in general.

Where in these notes. Module 14 §6, Module 13 §5, Primer D (state feedback).

P4The expressivity–robustness gap

In plain words. A network that is Lipschitz by construction comes with a robustness certificate for free: if the correct class wins by a large enough margin, no small perturbation can flip the decision. The price is expressivity. Constraining every layer makes the network less accurate on clean inputs. The open problem is to close that gap, with networks fast enough for a control loop.

What is known.

• Among Lipschitz-by-design networks, the strongest results we found are those of LipNeXt (Hu, Hu & Fredrikson, ICLR 2026). For $\ell_2$ perturbations of radius $\varepsilon = 36/255$ on ImageNet, its 2-billion-parameter model certifies 41.2% at 57.0% clean accuracy, without extra generated training data. At the same $\ell_2$ radius on CIFAR-10, a 256-million-parameter model reaches 73.2% certified at 85.0% clean.
• LipKernel (Pauli et al., Automatica 2026) keeps standard convolution kernels, so inference is fast, but it has not been scaled to such sizes.

Why it is hard. A certificate at radius $\varepsilon$ needs a margin larger than $\sqrt2\,L\,\varepsilon$. Lipschitz layers keep $L$ small, but they also limit how sharply the network can separate classes, so many correctly classified inputs end up with margins too small to certify. Larger networks relax the restriction (LipNeXt uses 2 billion parameters), at the cost of inference time.

Where in these notes. Module 13 §6, Module 13 §4, Primer E (margins).

P5Extending the 2-D and N-D systems view

In plain words. A convolution slides a small filter over a signal. Control theorists noticed that this is exactly what a linear dynamical system does: it keeps a short memory of past inputs (the state) and combines it with the current input. Writing convolutional layers this way lets the whole toolbox of LMIs and dissipativity certify them. The open question is how far the trick extends: to strides, dilations, higher-dimensional data and modern sequence models, ideally with the smallest possible state.

What is known. Convolutions can be realised as 2-D systems of Roesser type (Pauli, Gramlich & Allgöwer, 2024; Gramlich et al., Automatica 2026). Minimality is proven only when the input and output channel counts agree. LipKernel's ideas were carried over to cascaded state-space models by other authors (LipSSM, arXiv 2026).

Why it is hard. Each architectural feature changes the state-space structure: a stride skips outputs, a dilation spreads the kernel out, and N-D data need memory in N directions. Minimality matters because the state size sets the size of the LMIs, and proving it needs a separate algebraic argument for each structure.

Where in these notes. Module 12 §7, Module 13 §4, Primer D (realisations).

5. Constrained & Safe Deep RL

Constrained RL treats safety as a budget: maximise reward while keeping an expected cost below a limit $d$ (Module 8). The problems below ask what that formulation guarantees and when, and how it scales to deep networks and language models.

R1Guarantees during learning at deep-RL scale

In plain words. Most constrained-RL algorithms promise that the policy they converge to satisfies the constraint. That says nothing about the thousands of episodes during training, which on a real robot is exactly when crashes happen. Guarantees during learning exist for small or structured problems. Extending them to neural world models and camera inputs is open.

What is known. SafeMDP (Turchetta et al., NeurIPS 2016), ActSafe (As et al., ICLR 2025) and safe exploration via policy priors (Wendl et al., ICLR 2026) give guarantees during learning. They rely on calibrated uncertainty, known Lipschitz constants or a safe prior policy. Certifying those ingredients for high-dimensional neural models is open.

Why it is hard. To stay safe while learning, you must know that an action is safe before you try it. That needs a trustworthy model, or a trustworthy prior, exactly where the learner has little data. With neural models, the uncertainty estimates that would provide this are not calibrated (see T4).

Where in these notes. Module 6 §7, Module 9 §5.

R2Constraint satisfaction at the last iterate

In plain words. Lagrangian methods turn "maximise reward subject to cost at most $d$" into a game. The policy maximises reward minus a price $\lambda$ times cost; the price rises when the cost is too high and falls when it is low. This game can circle around its solution forever. The policy overshoots the constraint, the price goes up, the policy becomes too careful, the price goes down, and so on. On average the constraint holds, but the policy you deploy, the last one, may violate it.

What is known. Lagrangian iterates oscillate, and most guarantees are for averaged iterates. Last-iterate results exist, but under restrictive assumptions: regularised or optimistic updates for tabular softmax policies (Ding et al., NeurIPS 2023), or gradient-domination conditions (Montenegro et al., NeurIPS 2024).

Why it is hard. Gradient descent–ascent on a non-convex–concave saddle problem rotates around the saddle point instead of converging to it. The PID view of Module 8 explains the oscillation, but it does not remove it.

Where in these notes. Module 8 §4, Module 8 §5, Primer B (saddle points).

R3Choosing and reporting the constraint

In plain words. "Safe" can be made precise in several ways. The cost can be small on average (expected cost), a crash can be rare (chance constraint), even the bad cases can be not too bad (CVaR), the cost can be small at every step (state-wise), or a crash can never happen (almost surely). These are different promises, and a policy can satisfy one while badly failing another. Results are also sensitive to the chosen limit $d$, so a single number per method can mislead.

What is known. Expected-cost constraints allow rare catastrophes. Stricter semantics trade tractability for guarantees. Lagrangian methods are sensitive to the cost limit, and Spoor et al. (arXiv 2025) recommend reporting results over task-specific sets of cost limits.

Why it is hard. Expected cost keeps the CMDP a linear program over occupancy measures (Module 8 §2). Chance constraints, CVaR constraints and almost-sure budgets on the accumulated cost do not, and need extra state or other reformulations (Module 8 §6, Module 8 §7). One useful exception: "never fail", with a nonnegative failure cost, is the linear constraint $J_c(\pi) = 0$, because a nonnegative random variable with mean zero is zero almost surely.

Where in these notes. Module 1 §1, Module 8 §6, Module 9 §7, Primer C (VaR and CVaR).

R4Cost estimation under distribution shift, and offline-to-online safety

In plain words. Offline RL learns from a fixed log of past experience, without new trials. To respect a constraint it must estimate the cost of actions, including actions the log never contains. Learned critics guess those costs by extrapolation, and the guesses are often too optimistic. The policy then drifts toward exactly the untested actions whose cost was underestimated. Being pessimistic about unseen actions fixes this but makes the policy timid. Moving safely from offline training to online fine-tuning is the open part.

What is known. Off-policy and offline critics underestimate the cost of out-of-distribution actions, which motivated CPQ (Xu et al., AAAI 2022). Pessimism can make policies over-conservative. Feasibility-based methods such as FISOR (Zheng et al., ICLR 2024) improve offline safety, but safe offline-to-online fine-tuning with guarantees is largely unsolved.

Why it is hard. The data say nothing about unseen actions, so any estimate there is an assumption. Optimistic assumptions break safety, pessimistic ones break performance. And once the policy goes online, the data distribution changes again (see S1).

Where in these notes. Module 9 §6.

R5Constrained alignment of foundation models

In plain words. Safe RLHF fine-tunes a language model to be helpful while keeping a learned "harmfulness" score below a limit. Both the helpfulness reward and the harm cost are themselves learned from human judgements, so they are approximations. If the cost model underestimates harm, the optimiser exploits exactly that error. The true harm can then exceed the limit even though the training constraint looks satisfied.

What is known. Safe RLHF (Dai et al., ICLR 2024) trains a cost model from human judgements and runs Lagrangian PPO. Theory is starting to appear:

• Du, Kong & Srikant (arXiv 2025) analyse a primal–dual DPO method with bounds that include reward- and cost-estimation error.
• Latham & Moharrami (arXiv 2026) treat offline constrained RLHF with several preference oracles and constraints.

Both rest on realisability and data-coverage assumptions. Misspecified neural reward and cost models, and PPO-style training with many interacting constraints, remain open.

Why it is hard. The optimiser actively searches for responses where the learned cost is low, so cost-model errors are not random noise: they are selected for. Bounds therefore need control of the error everywhere the policy might go, which brings back the coverage problem of R4.

Where in these notes. Module 8 §8.

R6Unifying CMDP learning with control certificates

In plain words. Constrained RL and control theory express safety in different languages: an expected-cost budget, a failure penalty, a barrier-function filter, a reachable set. Each has strengths, and combining them could give policies that learn well and are provably safe. Today such combinations are built case by case. No common theory says which combination guarantees what, and they are rarely compared on the same control benchmarks.

What is known. Individual combinations exist: HJ feasibility in FISOR, safe value functions (Massiani et al.), CBF filters around RL policies. They are ad hoc, and control-oriented benchmarks such as safe-control-gym (Yuan et al., RA-L 2022) are rarely used for comparison.

Why it is hard. The guarantees live on different objects. A CMDP constrains an average over episodes, a penalty constrains optimal policies, and a filter constrains each action in each state. Composing them requires stating all three in one framework, which is the goal of the unified safety-filter view (Module 10 §8).

Where in these notes. Module 7 §7, Module 9 §6, Module 10 §8.

6. Safety Filters & Learned Certificates

A safety filter sits between a policy and the system, and corrects unsafe actions before they are executed (Module 10). It separates performance (the policy's job) from safety (the filter's job). The problems below are about what happens when the filter itself is learned, when it has no valid action, and how it interacts with learning and with the real world.

F1Filters built on learned models

In plain words. A classical filter relies on a known model of the system. Newer filters learn the model, and sometimes even what "failure" looks like, from data such as camera images. The filter is then only as reliable as what it learned. When the learning errors have verified bounds, robust filters can absorb them into a guarantee (Module 10 §9); for filters that learn from images, such bounds are not available today.

What is known. Latent safety filters (Nakamura, Peters & Bajcsy, RSS 2025) enforce safety in a world model's latent space. The authors state that their practical implementation lacks formal assurances, because errors in the learned world model, failure classifier and HJ value function are not bounded. How such errors propagate to the safety guarantee is open. Calibrated, verifiable uncertainty for ensembles and world models is missing more generally (see also T4).

Why it is hard. Three learned components feed into each other. Errors in the world model change which states the filter thinks are reachable, errors in the failure classifier change what it avoids, and errors in the value function change where it intervenes. Bounding each error is hard; bounding how they combine is harder.

Where in these notes. Module 10 §9.

F2Verifying learned certificates at scale

In plain words. A neural barrier or Lyapunov function is trained so that its defining condition holds at sampled states. A certificate needs the condition at every state, including the infinitely many that were not sampled. Formal verifiers can check this, but their cost explodes with the dimension of the state and the size of the network.

What is known. Training losses on samples do not make a certificate. Policy neural CBFs (So et al., ICRA 2024) and DeepReach (Bansal & Tomlin, ICRA 2021) scale well but are not verified. Formally verified Lyapunov-stable neural controllers exist for modest sizes (Yang et al., ICML 2024); SMT solvers and bound-propagation verifiers still limit the size.

Why it is hard. Exact verification of ReLU networks is NP-hard, so in the worst case its cost grows exponentially. Sampling-based certificates face the same growth in a more visible form, as the example shows.

Where in these notes. Module 10 §9, Module 11 §4, Module 15 §2.

F3Feasibility, deadlock and liveness

In plain words. At each moment, a CBF filter picks the action closest to what the policy wants among the actions that keep the system safe. Sometimes no such action exists: the motors are not strong enough, or two safety constraints demand opposite things. Then the filter has no answer. And even when it always has an answer, it may keep the system safe by stopping it forever, short of its goal: safe but useless.

What is known. The CBF-QP can become infeasible under input bounds or several interacting constraints. Combined CLF–CBF quadratic programs can introduce spurious asymptotically stable equilibria (Reis, Aguiar & Tabuada, L-CSS 2021). Reach-avoid formulations address liveness for specific settings (Hsu et al., RSS 2021).

Why it is hard. Feasibility depends on the barrier, the input bounds and the dynamics together. Checking that a barrier is feasible everywhere is itself a verification problem (F2). The natural remedy, a barrier that accounts for the input limits, is essentially the viability kernel, which is hard to compute.

Where in these notes. Module 10 §3, Module 10 §4, Module 10 §6.

F4Filters that do not ruin performance

In plain words. A filter makes a policy safe by overriding it, and every override is a moment where the policy does not do what it was trained to do. If overrides are frequent, performance collapses. If the learner does not account for them, it learns from actions it never actually took. The goal is policies that rarely need correcting, because they learned with the filter in the loop.

What is known. Filters shift the action distribution, can cause chattering, and degrade RL or imitation performance. Training with the filter in the loop helps (Pizarro Bejarano, Brunke & Schoellig, RA-L 2025). Keeping filtered actions consistent with the training distribution of a generative policy is the aim of PACS (Römer et al., ICRA 2026).

Why it is hard. With a fixed filter in the loop, the policy learns in the combined system "world plus filter". That is a valid learning problem, but many proposed actions map to the same executed action, so the learning signal about them is flat. If the filter itself is learned or retuned, the combined system changes during training. Without the filter, the policy is trained for a world it will never face. Generative policies add a twist: their outputs come from a learned distribution, and pushing an action outside that distribution can produce motions the policy has never seen.

Where in these notes. Module 10 §8.

F5Real-world effects

In plain words. Safety proofs are usually written for an idealised controller that measures the exact state, acts instantly and continuously, and is alone. Real controllers sample the state every few milliseconds and act with delay. They see the world through noisy sensors, share space with other agents, and, with generative policies, output whole chunks of motion at once. Each of these gaps can break a guarantee unless a margin accounts for it.

What is known.

• Continuous-time CBF guarantees need margins when implemented at a fixed sampling rate.
• Under partial observability the filter acts on estimates. Safety of the true state can still be guaranteed if estimation errors are bounded, as with measurement-robust CBFs (Dean et al., CoRL 2020); otherwise safety has to be stated over beliefs.
• Multi-agent filters scale to many agents with learned graph barriers (GCBF+, Zhang et al., T-RO 2025).
• For generative policies, PACS (Römer et al., ICRA 2026) gives formally safe, real-time filtering of diffusion and flow-matching policies, including the SmolVLA vision–language–action model. It assumes bounded tracking and measurement errors and bounded object motion. Relaxing these assumptions, for example to learned perception without error bounds, is open.
• The unified safety-filter view organises all of these questions (Hsu, Hu & Fisac, Annual Review 2024).

Why it is hard. Each effect needs its own margin, margins add up, and too much margin makes the system useless. Bounding the errors of learned perception, which PACS and measurement-robust CBFs both require, is itself an open problem.

Where in these notes. Module 10 §8, Primer D (sampling a continuous system).

7. Statistical Guarantees & Trustworthy Certificates

Some guarantees are not proofs about every input, but statements that hold with high probability over random data or random noise. They need fewer assumptions about the system and more about the data (Module 15 §7). The last problem in this section asks how far any certificate, deterministic or statistical, can be trusted.

S1Statistical guarantees under feedback and distribution shift

In plain words. Conformal prediction wraps any predictor in an error bar that is correct with a chosen probability, say 90%, using nothing but a list of past prediction errors. The catch is one assumption: future errors must be statistically interchangeable with past ones. A robot that acts on the error bars changes the world it is predicting, and that breaks the assumption.

What is known.

• Split conformal prediction needs exchangeable data.
• Some feedback is compatible with finite-sample coverage. Under feedback covariate shift, the model chooses the next inputs, but the distribution of outcomes given an input stays fixed. Coverage then holds when the likelihood ratios are known (Fannjiang et al., PNAS 2022).
• In control, the robot's actions usually change the outcome distribution itself. Lindemann et al. (RA-L 2023) therefore assume that the controller does not affect the environment's distribution.
• Adaptive variants give weaker guarantees, for example coverage only in the long run.

Why it is hard. Feedback couples the data to the policy. When the policy changes how the environment responds, even the conditional distribution of outcomes shifts, so tomorrow's scores are not exchangeable with today's.

Where in these notes. Module 15 §5, Primer C (exchangeability).

S2Beyond randomized smoothing

In plain words. Randomized smoothing turns any classifier into a new one, the smoothed classifier, which returns the class that is most likely when Gaussian noise is added to the input. If one class is likely enough, no small perturbation can change the smoothed classifier's answer. Those probabilities are estimated by classifying many noisy copies of the input. So the certificate holds only with high probability, it needs thousands of network evaluations per input, and it works well only for the $\ell_2$ norm. For other norms, especially $\ell_\infty$ (every pixel may change a little), its guarantees fade in high dimensions.

What is known.

• Randomized smoothing certificates hold only with high probability over Monte Carlo sampling, and they need many forward passes per input: Cohen et al. (ICML 2019) used 100,000 noise samples per certified input.
• For $\ell_p$ with $p \gt 2$, the certifiable radius of i.i.d. smoothing shrinks like $O\big(1/d^{1/2-1/p}\big)$ with the input dimension $d$ (Kumar et al., ICML 2020).
• Deterministic $\ell_\infty$ certified training has its own plateau. CTBench (Mao, Balauca & Vechev) found that the claimed advantage of recent methods drops significantly once older baselines are tuned fairly. On CIFAR-10 at $\varepsilon = 8/255$, all methods it tested certify about 35%.

Why it is hard. An $\ell_\infty$ ball of radius $\varepsilon$ in $d$ dimensions has corners at $\ell_2$ distance $\varepsilon\sqrt d$ from its centre, so any method built on $\ell_2$ geometry pays a factor $\sqrt d$. Kumar et al. show this is not just a weakness of one conversion: smoothing with i.i.d. noise cannot avoid it.

Where in these notes. Module 15 §4, Module 13 §6, Primer C (Clopper–Pearson bounds).

S3Certificates you can trust

In plain words. A certificate is a proof, and proofs can be wrong. Some published certificates had subtle errors that went unnoticed until someone built a counterexample. Others are computed numerically: an SDP solver returns a matrix that it says is positive semidefinite, but only up to a rounding tolerance. A certificate that is "almost" valid is not valid.

What is known. Published proofs have needed corrections, including the original coupled-multiplier LipSDP variant (Pauli et al., L-CSS 2022) and the $\tau \gt 0$ case of chordal LipSDP (Xue et al.). Certificates computed with floating-point SDP or LP solvers are only approximately feasible. The same lesson appeared while writing these notes: independent review caught many statements that did not match their cited sources.

Why it is hard. Floating-point arithmetic cannot represent most numbers exactly, and SDP solvers stop at a tolerance. Exact checks are possible, but they cost more and need the certificate to have some slack.

Where in these notes. Module 12 §5, Module 15 §7, Primer A (positive definite matrices).

Possible directions. Re-check numerical certificates with exact or interval arithmetic after solving. Machine-check the key lemmas with a proof assistant.

8. Where the Two Lines Meet

B1Certified Lipschitz constants as checkable assumptions for safe exploration

In plain words. LoSBO needs a Lipschitz constant, but nobody knows it (T1). The Pauli line computes certified Lipschitz constants of neural networks. If the controller is such a network, could we assemble LoSBO's constant from certified pieces instead of guessing it? The obstacle is that constants multiply when maps are chained, and a closed loop chains the same map many times.

The catch. Lipschitz constants multiply under composition. A closed loop evaluated over $T$ steps can have a constant as large as $L^T$ when the per-step constant is $L$, so this worst-case bound can become too loose over long horizons unless the loop contracts or the available margins are large. This is a reason to look at contracting architectures such as RENs (Module 13 §5).

Background. T1, Module 5 §3 (LoSBO), Module 6 §4 (GoSafeOpt), Module 12 §4, Module 13 §5, Primer B (composition rule).

B2Statistical model-error bounds plus IQC analysis of neural loops

In plain words. Learning gives statements like "the model error is at most 0.1, with probability 99%". Robust control gives statements like "the loop is stable for every model error up to 0.1". Chaining the two yields "the loop is stable with probability 99%", a guarantee neither tool provides alone. Fiedler, Scherer and Trimpe did this for linear controllers; doing it for neural-network controllers analysed with Pauli-style IQCs is the proposed direction.

Background. Module 11 §7, Module 14 §2, Module 3 §6.

B3Certified approximate MPC with Lipschitz-bounded networks

In plain words. Model predictive control is powerful but slow: it solves an optimisation problem at every step. A neural network can learn to imitate it and run in microseconds, but MPC's guarantees then carry over only if the imitation is accurate everywhere. Lipschitz-bounded networks make "accurate everywhere" checkable from a finite set of tests.

Background. Module 11 §6, Module 13 §4.

B4Uncertainty-aware safety filters with certified ensembles

In plain words. UPSi trusts its ensemble only in a "certain set", where the members agree. Today the certain set is estimated, not certified. If each member is Lipschitz-bounded, their disagreement cannot jump: it can only grow at a known rate as you move away from a point where it was measured. That turns "the members agree here" into a certified statement about a whole neighbourhood.

Background. Module 7 §6, Module 10 §9, Module 13 §1.

9. Gaps in These Notes

Topics these notes touch only briefly or not at all, which would make good future modules:

Discussion Questions

These have no single answer; each points at the core difficulty of one or more problems above. A suggested angle is hidden under each.

Key Papers