4. Safe Bayesian Optimization: SafeOpt & Controller Tuning

SafeOpt, its guarantees, practical variants, and the DSME line of work on BO for controller tuning

Book contents · Apply this chapter to a real decision · Glossary

1. The Safe BO Problem

Tuning a controller on hardware is a sequence of expensive experiments: choose gains, fly the quadrotor, measure a cost. Standard Bayesian optimization (BO; Primer E) fits a surrogate, a cheap statistical model of the expensive experiment (here a GP), and runs next the experiment that maximises an acquisition function, a score computed from that model. It happily spends a few bad experiments to learn, because in its world a bad experiment only costs regret (defined below). In safety-critical tuning it may cost the platform. Sui, Gotovos, Burdick and Krause (ICML 2015) formalised the version of the problem in which every single evaluation must be safe and gave the first algorithm with guarantees, SafeOpt. This page covers the algorithm, what it proves, the variants that actually run on robots, and how the Trimpe group (DSME, RWTH Aachen; earlier MPI-IS) turned BO into a controller-tuning tool.

Regret. Let $x^*$ maximise $f$ over a comparator set $A$ (all of $D$, or only its safely reachable part defined below). The regret of query $x_t$ is $r_t = f(x^*) - f(x_t)$, the cumulative regret after $T$ rounds is $R_T = \sum_{t=1}^T r_t$, and an algorithm is no-regret if $R_T/T \to 0$. A final recommendation $\hat x_T$ is judged instead by its simple regret $f(x^*) - f(\hat x_T)$. (Each $r_t \ge 0$ when $x_t \in A$; a query outside $A$ can have negative regret.) SafeOpt bounds the simple regret with respect to the reachable safe set; it promises nothing about the cumulative regret of its exploratory queries.

Notation clash, said once. In the safe-BO modules $h$ is SafeOpt's scalar threshold, following the papers; in Module 10 the same letter is a barrier function $h(x)$ whose zero superlevel set is the safe set. Likewise $\gamma_t$ below is the maximum information gain, not the RL discount. In controller tuning $f$ is a performance measure $J(a)$ of parameters $a$ and the constraint $J(a) \ge J_{\min}$ declares a controller worse than a baseline unsafe (Berkenkamp, Schoellig and Krause, ICRA 2016); separate constraints $g_i(a) \ge 0$ arrive in Section 4.

The GP posterior after $t$ observations is the kernel-ridge estimate of Module 3,

with $k_t(x) = [k(x_1,x), \dots, k(x_t,x)]^\top$ and $K_t = [k(x_i,x_j)]_{ij}$. The prior $f \sim \mathcal{GP}(0,k)$ is a computational device only: $f$ is a fixed RKHS function, the noise need not be Gaussian, and every statement below is a frequentist high-probability statement over the noise. The GP is deliberately misspecified but its intervals are provably valid under the assumptions above.

Safe BO is not constrained BO

"With probability at least $1-\delta$ over the whole run" is a statement about one event covering every round: $\Pr\big(f(x_t) \ge h \text{ for all } t \ge 1\big) \ge 1-\delta$. It is much stronger than $\Pr\big(f(x_t) \ge h\big) \ge 1-\delta$ for each round separately, which by the union bound only guarantees $1 - T\delta$ for a run of $T$ rounds (Primer C). SafeOpt obtains it from confidence intervals that are valid jointly for all rounds and all inputs (Section 3).

The best any Lipschitz-certifying algorithm could do

If we knew $f$ exactly on the seed, $x$ could be declared safe whenever some $x' \in S_0$ satisfies $f(x') - L\,d(x',x) \ge h$; exploring those points would certify further ones, and so on. With noise we only ever know $f$ up to some accuracy $\epsilon$ inside the safe set. That gives the benchmark against which SafeOpt is measured.

Three remarks. (i) $R_\epsilon$ depends on the unknown $f$: it is a yardstick, not something the algorithm computes. (ii) $\bar R_\epsilon(S_0) \subseteq \bar R_0(S_0) \subseteq \{x : f(x) \ge h\}$, and the last inclusion can be strict: a safe point behind a shallow valley (where $f$ dips close to $h$) or behind an unsafe gap is safe but unreachable. (iii) Sui et al. argue, explicitly "under the Lipschitz-continuity assumption", that no algorithm that learns $f$ only up to $\epsilon$ can ever certify a point outside $\bar R_\epsilon(S_0)$, so no safe algorithm can be asked to find the global optimum $\max_D f$. The argument presumes that the Lipschitz bound is the only way to carry knowledge to unevaluated points: the closure is the benchmark for Lipschitz certificates, not an absolute limit. An algorithm that also exploits valid kernel structure, e.g. by certifying $x$ whenever its own GP lower bound clears $h$ (Sui et al.'s variant in Section 2, the Lipschitz-free rule of Section 4), can certify points outside it, even across an unsafe gap (Exercise 4.4).

2. SafeOpt: Safe Set, Expanders, Maximisers

SafeOpt maintains three sets: decisions certified safe, safe decisions that could enlarge that certificate, and safe decisions that could be the optimum. It samples the most uncertain point of the last two. Everything is built from GP confidence intervals.

The initialisation encodes the seed: points of $S_0$ start with $l_1(x) \ge h$ whenever the intersected interval is nonempty, so the seed certifies itself and stays certified. Intersecting is what Sui et al. call a technical choice; Section 3 shows exactly where the proof needs it.

Read the three sets as three uses of one interval. The safe set uses the lower bound at an already-safe point and pays the Lipschitz price $L\,d$ to reach a neighbour: this is the only way a point enters $S_t$, and it is deterministic once $l_t$ is valid. Expanders use the upper bound: safe points that, in the best case, would certify something new. Maximisers compare the upper bound with the best lower bound: a point whose optimistic value cannot beat what is already known to be achievable is discarded forever, since its $u_t$ only falls and $\max l_t$ only rises.

Dynamics of a run. $S_t$ is monotone, $S_0 \subseteq S_1 \subseteq \cdots$, and a run consists of stages during which $S_t$ is constant. Within a stage $G_t$ and $M_t$ only shrink; the most uncertain candidate is sampled until its interval is narrow enough to certify a new point (the stage ends with an expansion) or nothing is left to learn. The stopping rule "$S_t = S_{t-1}$ and $\max_{G_t \cup M_t} w_t \le \epsilon$" then delivers, w.h.p., an $\epsilon$-optimal $\hat x_t = \operatorname{arg\,max}_{S_t} l_t$ (Section 3; Steps 4–5 of the derivation apply to such a round, with the width bound supplied by the test instead of Step 3); the algorithm itself never uses $\epsilon$. Sui et al. state the rule with the width test alone, but their own proof sketch says that narrow expanders mean either that $S_t$ will expand after the next evaluation or that it already contains $\bar R_\epsilon(S_0)$, and only the second case licenses stopping. A narrow expander certifies its neighbours only in the next safe-set update, so the width test alone can fire one round too early, right after an expansion. Example: $D = \{0,1,2\}$, $d(x,x') = |x - x'|$, $h = 0$, $L = 1$, $S_0 = \{0\}$, $f = (1.2, 2.2, 3.2)$, the rank-one kernel $k(x,x') = f(x)f(x')/3.2^2$ (so $\|f\|_k = 3.2$), noise bounded by $10^{-5}$, the theoretical $\beta_t$ with $\delta = 0.1$, and $\epsilon = 0.1$. One observation at $0$ gives $S_2 = \{0,1\}$ and $G_2 \cup M_2 = \{1\}$ with $w_2(1) \approx 0.011 \le \epsilon$, so the width test stops with $f(\hat x_2) = 2.2$, although $f^*_\epsilon = 3.2$ and point $2$ is certified from $1$ in round $3$; with the extra condition the run stops in round $4$ at $\hat x = 2$. Sui et al. also mention a variant that additionally certifies $x'$ whenever $l_t(x') \ge h$ directly, i.e. the condition becomes $\max\{l_t(x) - L\,d(x,x'),\ l_t(x')\} \ge h$; the theorem then holds with $|\bar R_0(S_0)| + 1$ replaced by the worse constant $|D|$, and this is the seed of the Lipschitz-free practical variant of Section 4.

To reproduce the numbers: with $\phi(x) = f(x)/3.2 = (0.375, 0.6875, 1)$ the kernel is $k(x,x') = \phi(x)\phi(x')$, whose RKHS consists of the multiples $c\,\phi$ with norm $|c|$, so $\|f\|_k = \|3.2\,\phi\|_k = 3.2$. Take the noise-free observation $y_1 = 1.2$ and $\lambda = \sigma^2 = 10^{-10}$. For a rank-one kernel $\det(I + \sigma^{-2}K_2) = 1 + \sigma^{-2}\big(\phi(x_1)^2 + \phi(x_2)^2\big)$, largest for $x_1 = x_2 = 2$, so $\gamma_2 = \tfrac12\log(1 + 2\cdot 10^{10}) \approx 11.86$ and $\beta_2 = \sqrt{2 \cdot 3.2^2 + 300\,\gamma_2\log^3(2/0.1)} \approx 309$. The one-observation posterior gives $\mu_1(1) = 2.2$ and $\sigma_1^2(1) = k(1,1) - k(1,0)^2/(k(0,0) + \lambda) = \phi(1)^2\lambda/(\phi(0)^2 + \lambda)$, so $\sigma_1(1) \approx 1.83 \times 10^{-5}$ and $w_2(1) = 2\beta_2\sigma_1(1) \approx 0.011$.

3. What SafeOpt Guarantees

Everything hinges on the confidence intervals being valid simultaneously for all rounds and all points. Sui et al. import this from the RKHS analysis of GP-UCB (Srinivas, Krause, Kakade and Seeger, ICML 2010, Theorem 6). The statements are given in the paper's convention, then translated.

In words. On one high-probability event, no evaluation is ever unsafe, and after a number of rounds scaling like $\beta_t^2\gamma_t\,|\bar R_0(S_0)|/\epsilon^2$ the pessimistic best guess is within $\epsilon$ of the best value in the $\epsilon$-reachable set. Both statements share one $\delta$.

Why each assumption matters. The RKHS bound and the noise model give the Lemma; without them $l_t, u_t$ are just numbers. The Lipschitz constant is used twice: to transfer safety (the safety half needs nothing else) and to define $\bar R_\epsilon$, i.e. to say what "reachable" means. The seed is the base case of every induction. Finiteness of $D$ is used to count stages: the safe set can expand at most $|\bar R_0(S_0)|$ times, whence the factor $|\bar R_0(S_0)| + 1 \le |D| + 1$, which is of the order of the whole grid when most of $D$ is safely reachable.

The proof architecture

The ICML paper gives the idea and defers details to a longer version; the complete lemma chain, extended to several constraints, is in the proof section of Berkenkamp, Krause and Schoellig (Machine Learning 2023), which follows Sui et al.'s outline. Three ingredients: safety is by construction once the intervals are valid; the expander definition forces the safe set to grow whenever it can; and the GP-UCB bound $\sum_t \sigma^2_{t-1}(x_t) \le C_1\gamma_t/4$ turns "eventually" into a number.

What the theorem does not say. It bounds a sufficient number of rounds $t^*$, not cumulative regret: SafeOpt is a best-arm-identification style result, and nothing is promised about how good the sampled $x_t$ are along the way (expanders are typically mediocre boundary points). Nor can $t^*$ be computed in advance, since it depends on the unknown $|\bar R_0(S_0)|$; the observable stopping test of Section 2 is what tells the user that the guarantee applies. It says nothing about an underestimated $L$, a misspecified kernel or an underestimated $B$; each silently invalidates both halves (an overestimated $L$ or $B$ only slows expansion). And it is stated for finite $D$; continuous domains need a grid with a Lipschitz covering margin, or the grid-free methods of Section 6 and Module 5. The margin works as follows (Primer B): if $f$ is $L$-Lipschitz on the whole continuous domain and every point $x$ lies within distance $r$ of some grid point $z$, then $f(x) \ge f(z) - Lr$, so a grid point with $l_t(z) \ge h + Lr$ certifies the whole ball of radius $r$ around it, whereas $l_t(z) \ge h$ certifies only $z$ itself. Likewise the best certified grid point is within $Lr$ of the continuous optimum whenever the grid point nearest to that optimum is certified.

4. Practical Variants: SafeOpt-MC, StageOpt & Friends

Three things make the 2015 algorithm awkward on a robot: the Lipschitz constant, the single function that is both objective and constraint, and the coupling of expansion and optimisation. Each was addressed within three years, and each fix trades away part of the theorem.

4.1 Lipschitz-free SafeOpt: the version most code runs

Berkenkamp, Schoellig and Krause (ICRA 2016) tuned a quadrotor's position controller with SafeOpt and, to avoid specifying $L$, defined all three sets from the GP alone ($a \in A$ are parameters, $J_{\min}$ the performance threshold).

The journal version states the practical safe set for several constraints as $S_n = S_0 \cup \{a \in A : l^i_n(a) \ge 0\ \forall i \in I_g\}$ (their eq. 24) and proves in Lemma 2 that, with the theoretical $\beta_n$ of its Lemma 1 (whose printed constant carries the regularisation caveat discussed below 4.3), every evaluated parameter still satisfies all constraints with probability at least $1-\delta$: the safe set is by definition a set of points whose valid lower bounds certify the constraints, and it is never empty because $S_0 \subseteq S_n$. Computing $G_n$ needs one GP update per candidate, but since only the most uncertain candidate is ever selected, it suffices to test the points of $S_n \setminus M_n$ with $w_n$ above $\max_{M_n} w_n$ in decreasing order of $w_n$ and stop at the first expander.

4.2 SafeOpt-MC: several constraints and contexts

Berkenkamp, Krause and Schoellig (Machine Learning 2023; arXiv since 2016) separate the objective $f$ from $q$ safety constraints $g_i(a) \ge 0$, $i \in I_g = \{1,\dots,q\}$, stacked into one function $h(a,i)$ with $h(a,0) = f(a)$, $h(a,i) = g_i(a)$, modelled by a single GP over the index-augmented input $(a,i) \in A \times I$, $I = \{0\} \cup I_g$ (a kernel block-diagonal in $i$ means independent functions; a shared kernel lets constraints and objective inform each other). Each evaluation returns $|I| = q+1$ measurements. With per-function bounds $l^i_n, u^i_n$ intersected over time as in Section 2:

Read the safe set inside out: for each constraint $i$, the union collects the points certified by some old safe point; the intersection keeps a point only if every constraint certifies it. Different constraints may use different witnesses $a_i$ (the order of quantifiers is "for every $i$ there is an $a_i$", weaker than one witness for all constraints; Primer 0). (The function $h(a,i)$ is unrelated to the scalar threshold $h$ of Sections 1–3; SafeOpt-MC writes every constraint as $g_i \ge 0$.) The matching benchmark replaces $l^i_n$ by the true $g_i$ minus the accuracy $\epsilon$ (their eqs. 8–9): $$R_\epsilon(S) = S \cup \bigcap_{i \in I_g}\big\{a \in A : \exists a' \in S,\ g_i(a') - \epsilon - L\|a - a'\| \ge 0\big\},$$ iterated to its closure $\bar R_\epsilon(S_0)$ as in Section 1, and $f^*_\epsilon = \max_{\bar R_\epsilon(S_0)} f$.

A point is safe only if every constraint certifies it (intersection), it is an expander if some constraint might newly certify something, and the acquisition takes the most uncertain function at the most uncertain point. The theory mirrors Section 3 with a Chowdhury–Gopalan-type multiplier (box below; the paper keeps Sui's $\beta^{1/2}$ notation, and the warning below 4.3 applies before reusing its constant with a $\sigma^2$-regularised posterior); the information gain is indexed by $|I|n$ because each iteration yields $|I|$ observations. Contexts $z \in Z$ (battery level, weather, or, in their experiment, the speed of the reference trajectory) are set externally each round; the GP lives on $(a,i,z)$, so data transfer across contexts through the kernel. On the quadrotor this tuned a position controller across trajectory speeds without restarting from scratch.

4.3 StageOpt: expand first, then optimise

SafeOpt's acquisition compares widths of objective and constraints on one scale, which is meaningless when they are dissimilar (the paper's example is temperature versus gripping force; in the clinic, a patient's comfort with a stimulus versus its therapeutic effect). Sui, Zhuang, Burdick and Yue (ICML 2018) split the run into an expansion stage driven only by the safety GPs and an optimisation stage inside the certified region. With $n$ safety functions $g_i$, thresholds $h_i$ and Lipschitz constants $L_i$:

Here $G_t$ collects the certified points that could still certify a new point under at least one constraint, as in SafeOpt-MC. In the theory the switch happens at a fixed round, $T_0 = t^*$ (box below); a remark in the paper instead ends stage one once every expander width $\max_{G_t} w^i_t$ has fallen below $\epsilon$, and the experiments end it when the safe region has not grown for 10 iterations. Stage two can be any online optimiser (the paper also gives a dueling-bandit version for preference feedback, used in a live clinical spinal-cord-stimulation experiment).

4.4 Scaling up: lines, trust regions, learned priors

SafeLineBO (Kirschner, Mutný, Hiller, Ischebeck and Krause, ICML 2019) runs BO on a sequence of clipped lines $\mathcal{L}_i = \{\hat x_{i-1} + \alpha\,l_i : \alpha \in \mathbb{R}\} \cap X$ (their $L_i$; calligraphic here to avoid a clash with the Lipschitz constants $L_i$): the line through the current best point $\hat x_{i-1}$ in direction $l_i$, restricted to the allowed decisions $X$. The directions come from a random, coordinate (change one parameter) or descent oracle, and each line is solved with SafeOpt on a finite grid of the line that contains a known safe point, so each 1-D sub-problem inherits SafeOpt's guarantees on that line's reachable grid points. Unconstrained LineBO has global convergence for random directions and fast local rates for strongly convex objectives (box below; strong convexity and smoothness: Primer B), and the method tuned the Swiss Free Electron Laser with up to 40 parameters. The price is that each line only sees the slice of the safe set it intersects.

HdSafeBO (Wei, Yi, Li, Soedarmadji and Sui, CoRL 2024) combines an isometric embedding, a trust region around the best safe point (both explained below) and optimistic safety identification: a point counts as safe when its upper bound $\mu_{t-1}(x) + \beta_t\sigma_{t-1}(x)$ clears the threshold $0$. Under their Assumption 3.1 ($f$ and $g$ are GP samples, Gaussian noise), choosing $\beta_t$ with $\Phi(\beta_t) \le 1-\alpha$ gives $\Pr(g(x) \ge \mu_{t-1}(x) + \beta_t\sigma_{t-1}(x)) \ge \alpha$ for every $x$ and step (Prop. 4.1), so a point declared safe is safe with probability at least $\alpha$ ($\Phi$ and its quantiles: Primer C). The rule is genuinely optimistic ($\beta_t \ge 0$) only for $\alpha \le 1/2$; a larger $\alpha$ forces $\beta_t \lt 0$, i.e. a lower confidence bound, as the paper notes. Theorem 4.2 states, for $V_T = \sum_t \max(0, -g(x_t))$: with probability at least $1-\delta$, $\mathbb{E}[V_T] \le (1-\alpha)\sqrt{C_1 T\beta^c_T\gamma_T}$ with $C_1 = 8/\log(1+\sigma^{-2})$ and $\beta^c_T = 2\log(|X|T^2\pi^2/(6\delta))$, which presupposes a finite candidate set $X$. Read it with care: the statement does not say what the expectation averages over or what the outer probability refers to, and the proof step that produces the factor $1-\alpha$ (their Lemma A.2, eq. 12) treats the random $g(x_t)$ as a constant; the box below gives what does follow. Either way this is a per-step probabilistic statement about a random $g$, not SafeOpt's joint guarantee for a fixed $g$; it buys scalability to hundreds of dimensions (musculoskeletal control, neural stimulation).

Which kernel? Safety in all of the above is only as good as the prior. Rothfuss, Koenig, Rupenyan and Krause (CoRL 2022) meta-learn the GP prior from related tasks (F-PACOH) and pick hyperparameters $\omega$ by maximising sharpness subject to an empirical calibration constraint, $\min_\omega \text{avg-std}(\{D_i\},\omega)$ s.t. $\text{avg-calib}(\{D_i\},\omega) \ge 1$: the data-driven answer to the question every SafeOpt user faces first. Two other answers attack the hyperparameters and the norm bound directly. Berkenkamp, Schoellig and Krause (JMLR 2019) adapt the hyperparameters of a stationary kernel slowly over time, so that the function class expands, and keep no-regret guarantees without knowing lengthscales or $B$ in advance (a regret result for GP-UCB, not a safety result). PACSBO (Tokmak, Schön and Baumann, SysDO 2024) estimates an RKHS-norm bound from data, treated as a local rather than a global quantity, and plugs it into a safe BO algorithm with a probably-approximately-correct safety guarantee; Module 3 discusses the related norm estimate and its i.i.d.-norm assumption. When decisions are states of a system rather than static parameters, safety also requires being able to return; SafeMDP and its descendants add reachability and returnability to the same expander template (Module 6).

5. BO for Controller Tuning (Trimpe group)

Independently of the safe-BO line, Sebastian Trimpe's group (then MPI for Intelligent Systems, now DSME at RWTH Aachen) built a decade-long programme around one question: how do you tune a real controller from a handful of experiments? The answers reuse GPs and BO, but the safety mechanisms differ from SafeOpt's, and the differences are instructive.

5.1 Automatic LQR tuning with Entropy Search (Marco et al., ICRA 2016)

Marco, Hennig, Bohg, Schaal and Trimpe (ICRA 2016) tune a state-feedback controller without touching its gains directly:

1. A nominal linear model $\tilde x_{k+1} = A_n\tilde x_k + B_n u_k + w_k$ of the true, unknown, nonlinear dynamics about the equilibrium (linearisation: Primer D).
2. A performance criterion $J = \lim_{K\to\infty}\frac1K\,\mathbb{E}\big[\sum_{k=0}^{K-1} x_k^\top Q x_k + u_k^\top R u_k\big]$ with fixed performance weights $(Q,R)$ (quadratic costs and LQR: Primer D).
3. A controller family parametrised through design weights: $u_k = F(\theta)x_k$, $F(\theta) = \mathrm{lqr}(A_n, B_n, W_x(\theta), W_u(\theta))$ with, e.g., diagonal $W_x, W_u$ depending on $\theta \in \mathbb{R}^D$. Two sign conventions change here: this subsection minimises a cost $J$, whereas SafeOpt maximises $f$ (set $f = -J$; then $f \ge h$ means $J \le -h$), and the gain is written $u = Fx$, so the common LQR convention $u = -Kx$ corresponds to $F = -K$.
4. An experiment of horizon $K$ returning the noisy finite-horizon cost $\hat J(\theta)$, a few minutes of balancing per evaluation.
5. A GP prior on $J(\theta)$ and Entropy Search (ES) as optimiser: ES keeps a distribution $p_{\min}(\theta) = \Pr[\theta \in \operatorname{arg\,min} J]$ over the location of the minimum and evaluates where the expected entropy reduction of $p_{\min}$ is largest; the answer is the best guess $\theta_{BG} = \operatorname{arg\,max} p_{\min}$, not the last query. On a finite candidate set $\{\theta_1,\dots,\theta_m\}$, $p_{\min}(\theta_j)$ is the posterior probability that $\theta_j$ is the minimiser and its entropy is $H(p_{\min}) = -\sum_j p_{\min}(\theta_j)\log p_{\min}(\theta_j)$ (Primer C); on a continuous domain a single point has probability zero, so $p_{\min}$ is a density over the minimiser's location, which ES approximates on a finite set of points. Exercise 4.5 uses the finite version.

Why the detour through LQR weights? Because every $\theta$ then yields a gain that stabilises the nominal model (under the usual stabilisability and detectability conditions, which positive diagonal weights satisfy whenever $(A_n, B_n)$ is stabilisable; see below), whereas tuning $F$ directly would search a much larger space full of destabilising gains. Marco et al. add an inverse-optimality remark (their footnote 1, citing classical results, among them Kalman's 1964 paper "When is a linear control system optimal?"): any stabilising gain whose return difference exceeds one in magnitude is an LQR solution for some weights, so parametrising by general LQR weights only discards gains that destabilise the nominal plant or have poor robustness. Kalman's frequency-domain criterion is a continuous-time, single-input result; the exact discrete-time test is the matrix condition in the box below. Either way the remark asserts the existence of some weight pair $(W_x, W_u)$; it does not cover a restricted family such as the bounded diagonal $W_x(\theta)$ tuned below, which shrinks the search space further (Marco et al. motivate that restriction separately, as a way to focus on the relevant parameters) and may exclude robust stabilising gains as well. Do not read it the other way round either: "every LQR gain has a return difference of at least one" is the continuous-time result, and in discrete time, the setting here, it can fail. For $A = B = Q = R = 1$ the Riccati equation $P = 1 + P - P^2/(1+P)$ gives $P^2 = P + 1$, so $P = (1+\sqrt5)/2$ and the gain is $K = P/(1+P) \approx 0.618$; with negative feedback $u = -Kx$ the loop transfer function is $K/(z-1)$, and at $z = -1$ the return difference is $|1 + K/(z-1)| = |1 - K/2| \approx 0.69$ (transfer functions and frequency response: Primer D). This is safety by parametrisation: free at run time, but a guarantee only for the nominal plant.

The conditions behind "every $\theta$ stabilises": take $W_u \succ 0$ and $W_x \succeq 0$. $(A_n, B_n)$ is stabilisable if every mode that the input cannot influence already has eigenvalue magnitude below one, and $(A_n, W_x^{1/2})$ is detectable if every mode invisible to the state penalty is already stable (controllability and observability: Primer D). Under these two conditions the stabilising Riccati solution yields a gain $F$ with all eigenvalues of $A_n + B_nF$ inside the unit circle; $W_x \succ 0$ makes detectability automatic. All of this concerns the nominal model.

On the humanoid Apollo balancing an inverted pole with its 7-DoF arm, 20 iterations of 2-D tuning ($W_x(\theta) = \mathrm{diag}(1, 50\theta_1, 10, 50\theta_2)$, $W_u = 10$, $\theta \in [0.01, 10]^2$) improved the nominal LQR's cost by 31.9%; with a pole of double length and the same, now poor, model the initial controller was unstable and ES found a stabilising one; and 46 iterations of 4-D tuning improved on the 2-D long-pole result by a further 31.7%. Experiments that destabilised the system were stopped and assigned a fixed penalty cost $J_u$ (3.0 and 5.0 in the 2-D and 4-D runs), the crude ancestor of the crash-constraint models below.

5.2 Virtual vs. real: multi-fidelity Entropy Search (Marco et al., ICRA 2017)

Marco, Berkenkamp, Hennig, Schoellig, Krause, Schaal and Trimpe (ICRA 2017) add a simulator as a cheap, biased information source. The model is $J(\theta) = J_{\mathrm{sim}}(\theta) + J_{\mathrm{err}}(\theta)$ with independent GP priors on both terms; extend the input by a fidelity flag $\delta \in \{0,1\}$ ($0$ = simulation, $1$ = robot) and use the kernel

with separate noise variances for the two sources. Two robot evaluations covary through both terms, a simulation and a robot evaluation only through $k_{\mathrm{sim}}$, so before any robot data a simulation can explain exactly the $k_{\mathrm{sim}}$ share of the robot cost's variance and nothing about the model error (Marco et al. describe this split of the robot cost's variance "before any data is observed"). Robot data couple the two terms: an experiment constrains their sum, so afterwards $J_{\mathrm{sim}}$ and $J_{\mathrm{err}}$ at the tested parameter are negatively correlated in the posterior (derivation below) and later simulations inform $J_{\mathrm{err}}$ too; at a parameter with noise-free robot and simulation values, their difference is $J_{\mathrm{err}}$ exactly. The distribution over the minimum is defined on the real system, $p_{\min}(\theta) = \Pr[\theta \in \operatorname{arg\,min}_\theta J(\theta,\delta{=}1)]$, and the acquisition trades expected information gain against evaluation effort $t_{\mathrm{sim}} \lt t_{\mathrm{exp}}$ (their eq. 10):

Here $\Delta H_i(\theta) = H(p_{\min} \mid D) - H\big(p_{\min} \mid D \cup \{(\theta, i, Y)\}\big)$ is the drop in the entropy of $p_{\min}$ if source $i$ were evaluated at $\theta$ and returned $Y$. Since $Y$ is not known before the evaluation, the expectation averages over its current GP predictive distribution; dividing by the effort $t_i$ gives information per second. (Exercise 4.5(d) uses the information about one cost value as a simpler proxy.)

The algorithm decides by itself when the simulator has become uninformative and switches to the robot (and back). On a cart-pole with an LQR-parametrised controller, with simulations 30 times cheaper than experiments ($t_{\mathrm{sim}} = 1$ s, $t_{\mathrm{exp}} = 30$ s), it needed fewer physical experiments than ES on the robot alone. Fewer hardware trials is itself a safety lever.

5.3 Crash constraints: failures as data

On a jumping quadruped a bad parameter does not return a large cost, it returns a fall and no cost at all. Marco, Baumann, Khadiv, Hennig, Righetti and Trimpe (RA-L 2021) call this learning with crash constraints: each experiment returns a label $l_i \in \{0,1\}$ (crash / success; their notation, not the lower confidence bound $l_t$ of Section 2) and a value $y_i$ only if $l_i = 1$. Their GP classified regression (GPCR) models a latent constraint $g$ with an unknown threshold $c$ through the likelihood

so successes are regression data below $c$ and crashes are censored observations above $c$; $c$ is learned by MAP together with the hyperparameters after an expectation-propagation approximation. The acquisition is expected improvement with constraints, $\alpha_{EIC}(x) = \mathbb{E}_{f(x)}[\max\{\eta_{cons} - f(x), 0\}]\,\Gamma(x)$, where $\eta_{cons}$ is the best feasible value observed so far and, in EIC$^2$, $\Gamma(x) = \prod_j \Pr(l_j(x) = 1) = \prod_j \Phi\big((\hat c_j - \mu_j(x))/\sigma_j(x)\big)$ is the success probability under the EP-approximated GPCR posterior with the learned threshold $\hat c_j$ (each factor is their eq. 9; plain EIC uses $\Pr(g_j(x) \le 0)$ with a known zero threshold). Before any successful point it uses $\Gamma$ alone, and the reported solution is $\operatorname{arg\,min}\mu(x)$ of the objective's posterior mean subject to $\Gamma(x) \ge 1-\delta$. Crashes are explicitly tolerated: the third regime of the table in Section 1. On the quadruped the learned jumps outperformed manual tuning.

Two independence assumptions hide in these formulas. The product $\prod_j$ in $\Gamma$ equals the probability that all constraints succeed only if the constraint models are independent given the data, and multiplying expected improvement by $\Gamma$ further assumes that improvement and feasibility are independent given the data. Without them the right quantity is the joint expectation $\mathbb E\big[(\eta_{cons} - f(x))^+\,\mathbf 1\{\text{all constraints succeed at } x\} \mid D\big]$, where $v^+ = \max(v,0)$.

von Rohr, Stenger, Scheurenberg and Trimpe (at – Automatisierungstechnik 2024) bring crash constraints to local BO. VDP-GIBO estimates the objective's gradient from a GP, chooses experiment batches that minimise the gradient's total variance $\alpha_{TV}(x_*, X') = \operatorname{tr}\Sigma_{\nabla,D'}(x_*)$, takes a gradient step on $\nabla\mu_{\hat D}(x_*)$, and handles crashes with adaptive virtual data points: a crashed $\hat x_i$ receives the pessimistic value $\hat y_i = \max\{\mu_D(\hat x_i), \mu_D(x_*)\} + \beta\sqrt{k_D(\hat x_i)}$ ($\beta = 3$ empirically), which pushes the posterior minimum out of the crash region without a classifier; an infeasible iterate resets to the best known feasible point. Menn, Stenger and Trimpe (RA-L 2026) do the same for preferential BO, where a human only says which of two runs was better (probit likelihood $\Phi\big((f(x_A) - f(x_B))/(\sqrt2\sigma)\big)$): CrashPBO adds a virtual duel $x_s \succ x_c$ for every crashed $x_c$ against every successful $x_s$, hyperparameter-free, evaluated on three robot platforms (gradient steps: Primer B).

Where the probit likelihood comes from: suppose larger utility is better and the human perceives run $x$ as $f(x) + \xi$ with independent $\xi \sim \mathcal N(0, \sigma^2)$. The difference of two perceptions has mean $f(x_A) - f(x_B)$ and variance $2\sigma^2$, so $\Pr(A \text{ preferred}) = \Pr\big(\xi_B - \xi_A \lt f(x_A) - f(x_B)\big) = \Phi\big((f(x_A) - f(x_B))/(\sqrt2\sigma)\big)$. A virtual duel $x_s \succ x_c$ records the successful run as preferred to the crashed one; it is an imposed comparison, not a measurement.

5.4 Robustness, simulation aid, and the 2026 tutorial

Turchetta, Krause and Trimpe (ICRA 2020) make robustness a second objective of model-free policy search: gain and delay margins, the classical robustness indicators, are estimated from experiments with scaled or delayed actions, and multi-objective BO (expected hypervolume improvement) traces the Pareto front between performance and robustness on a Furuta pendulum, in sim-to-real and pure hardware experiments. He, von Rohr, Baumann, Xiang and Trimpe (IEEE T-RO 2025) propose HCI-GIBO, a local zero-order policy search that keeps querying a derivative-GP model until it can certify, with probability at least $\alpha$, that a gradient step improves the policy, and S-HCI-GIBO, which first exhausts gradient information from an imperfect simulator (modelled as robot objective plus a GP reality gap) before querying the robot; the guarantee is on improvement per update, not on constraint safety, validated on a manipulator balancing a pendulum.

The multi-objective vocabulary (Primer E): with two quantities to maximise, one decision dominates another if it is at least as good in both and strictly better in one. The non-dominated decisions form the Pareto front, along which one quantity can only improve at the expense of the other. Fix a reference point worse than every outcome of interest; the hypervolume is the area dominated by the current outcomes and bounded by that reference point, and expected hypervolume improvement queries where the expected increase of this area is largest.

The entry point for practice is the tutorial Stenger, Brunzema, Menn, von Rohr, Schoellig and Trimpe, "A Decade of Bayesian Optimization for Controller Tuning and Robot Learning: Tutorial, Review, and Future Prospects" (arXiv 2026, under review). It positions BO against deep RL and data-driven control, walks through a controller-tuning setup from the practitioner's side, and catalogues constrained, safe, crash-constrained, multi-objective, preferential, contextual, time-varying, local and multi-fidelity BO. Its safe-BO section is candid: "in most practical applications, $\beta$ is set to a constant heuristic value, e.g., $\beta = 2$ […] or $\beta = 3$" (citations elided), and "while such heuristics often invalidate theoretical guarantees, they may still yield useful 'cautious' behavior in practice"; it points to LoSBO as the alternative that needs only a Lipschitz bound and bounded noise, and starts a lightweight benchmark suite for control problems because the field has none.

6. Beyond SafeOpt: Constrained BO, Barriers, Information-Theoretic Safe Exploration

SafeOpt sits at the hard end of a spectrum. Moving along it one can tolerate bounded violations, replace the Lipschitz certificate by information-theoretic reasoning, replace the GP by a barrier, or restrict the function class to get regret bounds. Each choice changes the guarantee, so the comparison at the end is by guarantee, not by acquisition function.

Constrained BO: feasibility only at the end. Constrained expected improvement (Gardner, Kusner, Xu, Weinberger and Cunningham, ICML 2014) multiplies EI over the best feasible point by the GP probability of feasibility, $\mathrm{EI}_C(x) = \mathrm{PF}(x)\,\mathrm{EI}(x)$, under conditionally independent objective and constraint GPs. It is the default baseline for safe BO and it is not safe: nothing stops it from querying a point with $\mathrm{PF} = 0.3$. (A slip to watch: the paper minimises but prints $Z$ with the sign of the maximisation case; derive EI yourself with $Z = (\ell(x^+) - \tilde\mu(x))/\tilde\sigma(x)$.) Gelbart, Snoek and Adams (UAI 2014) extend the idea to decoupled constraints evaluated independently of the objective.

The EI formula in this notation: Gardner et al. minimise an objective $\ell$; let $b = \ell(x^+)$ be the best feasible value so far and $\ell(x) \mid D \sim \mathcal N(\tilde\mu(x), \tilde\sigma(x)^2)$ the posterior. The improvement is $(b - \ell(x))^+$ with $v^+ = \max(v, 0)$. Writing $\ell(x) = \tilde\mu + \tilde\sigma u$ with $u \sim \mathcal N(0,1)$, the improvement is positive exactly when $u \lt Z := (b - \tilde\mu)/\tilde\sigma$, so $$\mathrm{EI}(x) = \int_{-\infty}^{Z} (b - \tilde\mu - \tilde\sigma u)\,\phi(u)\,du = (b - \tilde\mu)\,\Phi(Z) + \tilde\sigma\,\phi(Z),$$ using $\int_{-\infty}^{Z} u\,\phi(u)\,du = -\phi(Z)$; here $\phi(u) = e^{-u^2/2}/\sqrt{2\pi}$ and $\Phi$ are the standard normal density and CDF (Primer C). If $\tilde\sigma(x) = 0$, $\mathrm{EI}(x) = (b - \tilde\mu(x))^+$.

Soft constraints with bounded cumulative violation. CONFIG (Xu, Jiang, Svetozarevic and Jones, ICML 2023) is optimism applied to everything: $x_t \in \operatorname{arg\,min}_{x \in X} l_{0,t}(x)$ s.t. $l_{i,t}(x) \le 0$ for all constraints, with lower confidence bounds in the Chowdhury–Gopalan form. Theorem 4.3: w.p. $\ge 1-\delta$, cumulative regret $R_T \le 4\beta_{0,T}\sqrt{(T+2)\gamma_{0,T}} = O(\gamma_{0,T}\sqrt T)$ and cumulative violation $V_{i,T} = \sum_t [g_i(x_t)]^+ \le 4\beta_{i,T}\sqrt{(T+2)\gamma_{i,T}}$ for every $i$; infeasibility can be declared when $\min_x \max_i l_{i,t}(x) \gt 0$. Violations are allowed, and their sum grows sublinearly whenever $\gamma_{i,T} = o(\sqrt T)$, which holds for linear and squared-exponential kernels and for Matérn kernels with $\nu \gt d/2$ (their Table 2) but not for every kernel: the right notion for a reactor's temperature excursions, the wrong one for a crash.

Barriers instead of GPs. LB-SGD (Usmanova, As, Kamgarpour and Krause, JMLR 2024) runs stochastic gradient descent on the log-barrier surrogate $B_\eta(x) = f^0(x) - \eta\sum_{i=1}^m\log(-f^i(x))$ (log barriers: Primer B) with an adaptive step size (their $\gamma_t$, unrelated to the information gain) chosen from the smoothness constants and lower confidence bounds on the current slacks $-f^i(x_t)$ (their eq. 15) so that no constraint can lose more than half of its slack in one step, $f^i(x_{t+1}) \le f^i(x_t)/2 \lt 0$: a constraint at $-0.8$ may rise to $-0.4$ but never reaches $0$. The mechanism is visible already with Lipschitz constants alone: if $\underline s_i \gt 0$ is a valid lower bound on the slack $-f^i(x_t)$, any step $v$ with $L_i\|v\| \le \underline s_i/2$ gives $f^i(x_t + v) \le f^i(x_t) + \underline s_i/2 \le f^i(x_t)/2$. All iterates stay feasible w.h.p. using zeroth- or first-order feedback. The analysis rests on their Assumptions 1–4: a bounded feasible set; objective and constraints $M_i$-smooth and $L_i$-Lipschitz with known constants; a known starting point at which all constraints are strictly negative; and an extended Mangasarian–Fromovitz constraint qualification (box below), plus accuracy conditions on the function and gradient estimators. Under these, counting noisy function measurements (finite-difference batches included), the zeroth-order rates are $O(d^2/\epsilon^7)$ to an $\epsilon$-KKT point (non-convex), $\tilde O(d^2/\epsilon^6)$ (convex, with $\epsilon$ the objective gap) and $\tilde O(d^2/\epsilon^4)$ if the barrier is strongly convex (JMLR version; the earlier arXiv version had $\epsilon^5$). No kernel, no $\gamma_T$, polynomial in $d$: this is what scales, and the paper itself applies it to safe policy search in Safety Gym.

Information-theoretic safe exploration. ISE (Bottero, Luis, Vinogradska, Berkenkamp and Peters, NeurIPS 2022) drops both the grid and $L$. The safe set is $S_n = \{x : \mu_n(x) - \beta_n\sigma_n(x) \ge 0\} \cup \{x_0\}$, and the next query maximises an approximate mutual information between a new observation at $x \in S_n$ and the safety indicator $\Psi(z) = \mathbb{1}\{f(z) \ge 0\}$ of any $z$ in the whole domain, $x_{n+1} \in \operatorname{arg\,max}_{x \in S_n}\max_{z \in X}\hat I_n(\{x,y\}; \Psi(z))$, with the closed form from the second-order approximation $\hat H_n[\Psi(z)] = \ln 2\,\exp\!\big(-\mu_n(z)^2/(\pi\ln 2\,\sigma_n(z)^2)\big)$ of the binary entropy. Safety holds w.h.p. by construction of $S_n$. The exploration guarantee (Theorem 1) is conditional: if the safe set eventually stops expanding ($S_{n+1} \subseteq S_n$ for all $n \ge \hat n$) and the posterior mean stays uniformly bounded on it ($|\mu_n(x)| \le M$ for $x \in S_n$, $n \ge \hat n$), then for every $\epsilon \gt 0$ the posterior variance on $S_n$ drops below $\epsilon$ after a finite number $N_\epsilon$ of further rounds, which exists whenever $\gamma_N$ grows sublinearly in $N$. So no informative uncertainty is left unexplored. The authors regard the mean bound as mild, since $|\mu_n| \le 2\beta_n$ w.h.p. for valid, non-decreasing $\beta_n$ (their Lemma 11); but valid $\beta_n$ grow with $\gamma_n$, so this gives no uniform $M$, and a constant heuristic $\beta$ is not valid in the first place: $M$ remains a genuine assumption. ISE-BO (Bottero, Luis, Vinogradska, Berkenkamp and Peters, JMLR 2026) adds the objective, under sub-Gaussian noise, and learns the value of the safe optimum to arbitrary precision.

Transductive active learning. ITL (Hübotter, Sukhija, Treven, As and Krause, NeurIPS 2024) keeps a pessimistic safe set $S_n = \{x : l^g_n(x) \ge 0\}$ and an optimistic one $\hat S_n = \{x : u^g_n(x) \ge 0\}$, defines the targets $A_n = \{x \in \hat S_n : u^f_n(x) \ge \max_{x' \in S_n} l^f_n(x')\}$ (possibly unsafe points that might be the optimum) and samples inside $S_n$ where the observation is most informative about the targets, $x_n = \operatorname{arg\,max}_{x \in S_n} I(f_{A_n}; y_x \mid D_{n-1})$, with $f_{A_n}$ the objective values on the targets. This is transductive learning: measurements are allowed only in $S_n$, while accurate predictions are wanted on $A_n$. Theorem 5.1: with $f, g$ in the RKHS and sub-Gaussian noise, w.p. $\ge 1-\delta$ every $x_n$ is safe; and if the reducible uncertainty converges (which they show under a submodularity-type condition, their Assumption 3.2, and sublinear $\gamma_n$), the simple regret on the largest reachable safe set $\mathcal{R}$ drops below $\epsilon$ after finitely many rounds. Here $\mathcal{R}$ is the closure of a reachability operator with $(\epsilon,\beta)$-slack built from the GP's irreducible uncertainty (their Definition C.30), not SafeOpt's Lipschitz $\bar R_\epsilon$, and no Lipschitz constant is needed. Convergence to that closure is guaranteed for finite safe sets; on continuous domains the proof covers a fixed finite number of reachability steps, with a sample complexity growing in that number (their Remark C.31). The pessimistic/optimistic safe-set pair is the one familiar from GoOSE (Turchetta, Berkenkamp and Krause, NeurIPS 2019).

Safe bandits and regret. In linear bandits with a linear safety constraint $\mu^\top Bx_t \le c$ for all $t$, Safe-LUCB (Amani, Alizadeh and Thrampoulidis, NeurIPS 2019) explores randomly inside a known safe set, then is optimistic in the loss and pessimistic in the constraint; the regret is $\tilde O(\sqrt T)$ when the safety gap $\Delta = c - \mu^\top Bx^*$ is known to be positive and $\tilde O(T^{2/3})$ in general. In the GP setting, Losalka and Scarlett (AISTATS 2024) show that sublinear cumulative regret under hard per-round safety becomes achievable when the safety function is monotone in a single coordinate $s$ of the action (their algorithms know a lower bound on the growth rate of $g$ and an upper bound on that of $f$ in $s$, a weaker requirement than a global Lipschitz constant). The contrast with SafeOpt's sample-complexity-only statement is the point: hard safety plus cumulative regret needs extra structure, because expanding the safe region is itself expensive in regret.

Walkthrough: One SafeOpt Iteration

A complete round of Algorithm 1 on a six-point domain, with every number produced by an actual GP posterior (squared-exponential kernel, lengthscale $1.5$, unit prior variance, noise standard deviation $\sigma = 0.1$) and the heuristic multiplier $\beta = 2$, so that the numbers stay legible; the theorem's $\beta_t$ would be far larger. The hidden function is $f = (0.995,\ 1.502,\ 1.753,\ 1.400,\ 0.450,\ -0.498)$ at $x = 0,\dots,5$; the algorithm never sees it.

Interactive: SafeOpt in 1-D

Run SafeOpt on a 60-point grid against a random function built from the kernel: a sum of eight squared-exponential bumps with the GP's own lengthscale $0.12$, affinely rescaled to $[-1.2, 1.8]$. The rescaling adds a constant, and a nonzero constant is not in the squared-exponential RKHS of an interval (box below), so on the interval $[0,1]$ this $f$ violates the RKHS assumption. On the 60-point grid itself, however, every function has a finite RKHS norm $\sqrt{f_D^\top K_D^{-1} f_D}$ (Sui et al. note that the RKHS assumption is automatic for a finite domain and a universal kernel): about $3.9$ for the initial $f$ in exact arithmetic, a fragile number, since floating-point errors below $10^{-15}$ in the stored values raise it to about $6 \times 10^9$ (the $60 \times 60$ Gram matrix is nearly singular; conditioning: Primer A). The demo computes neither, so $\beta$ is only a tuning knob here. The GP posterior is computed from the full formulas at every step (Cholesky solve, Primer A; repeated noisy observations at one grid point are merged into one observation of their mean with noise variance $\sigma^2/m$, which is exact). The readout counts how often the true function was below the threshold when it was evaluated (zero with a large $\beta$, possibly positive with a small one: the gap that Module 5 is about) and how often an intersected interval came out empty (counted per point and round), the visible symptom of the confidence event failing.

Exercises

Graded practice — Build the argument yourself

There are 12 new problems: four Easy, four Medium, and four Hard. Start with the level that lets you make progress without opening the solution. Easy problems rebuild individual operations; Medium problems combine them; Hard problems ask for proofs, boundary cases, and the limits of a guarantee. The required concepts are explained on this page or in the linked earlier material.

Easy — Warm up one skill at a time.

Medium — Combine definitions and compute a certificate.

Hard — Explain why the argument works and where it stops.

Further practice — Original problems and research connections

The original exercises below retain their numbering. Some compare later methods or ask for longer research derivations; use the graded set above first, and return to a research-connection problem after reading the relevant linked module.

Key Papers

Flashcards