D. Dynamical Systems, Stability & Feedback Control

State-space models, Lyapunov stability, invariant sets, frequency response, LQR, PID and MPC

Readiness checks — a short prerequisite refresher

These checks are diagnostic, with the needed calculation explained in each answer. A gap tells you where to review; it is not a reason to skip this primer.

Book contents · Apply this chapter to a real decision · Glossary

Every control-theoretic guarantee in this section is a statement about a dynamical system: a set that trajectories never leave (safety), a point they approach (stability), or a bound on how much a disturbance can be amplified (a gain). This primer builds these three ideas from a first course in linear algebra and calculus. It starts with what a state is, solves linear systems exactly, and then introduces the certificates the modules rely on: Lyapunov functions, invariant sets, comparison functions, gains and quadratic constraints, and value functions. Every concept ends with the module sections that use it.

1. State-Space Models, Trajectories & Equilibria

The state is the smallest summary of the past that, together with the future inputs, determines the future. For a pendulum it is the angle and the angular velocity; for a learning algorithm it can be the policy parameters together with a Lagrange multiplier.

Example. The pendulum $\ddot\theta=\frac gl\sin\theta-b\dot\theta+u$ (angle $\theta$ measured from the upright position, as in Module 11's explorer) is second order. With $x_1=\theta$, $x_2=\dot\theta$ it becomes two first-order equations, $\dot x_1=x_2$, $\dot x_2=\frac gl\sin x_1-bx_2+u$: here $n=2$, $m=1$. A discrete example: $x_{t+1}=0.5x_t+u_t$ with $u_t=1$ and $x_0=0$ gives $0,\ 1,\ 1.5,\ 1.75,\ 1.875,\dots\to2$.

Why it matters. For any differentiable function $h$ of the state, the chain rule gives $\frac{\mathrm d}{\mathrm dt}h(x)=\nabla h(x)^{\top}f(x)+\nabla h(x)^{\top}g(x)\,u$, which is affine in $u$. A requirement such as "$\dot h\ge$ something" is therefore a half-space of inputs, and choosing the input closest to a desired one is a quadratic program. This is the reason control barrier function filters are QPs (Section 4, Module 10).

Models, disturbances and closed loops. We design on a nominal model $\hat f$; the physical plant differs from it, for example $x_{t+1}=f(x_t,u_t)+w_t$ with a process disturbance $w_t$. A guarantee proved for the model transfers to the plant only with margins for this mismatch (Modules 4 and 11). Substituting a controller $u=\pi(x)$ gives the autonomous closed loop $x_{t+1}=f(x_t,\pi(x_t))=:F_\pi(x_t)$. If the controller has memory (an integrator, an observer, a multiplier), the closed-loop state is the pair (plant state, controller state), for example $(\theta,\lambda)$ in Module 8.

Existence, uniqueness and forward completeness

In words. Local Lipschitz continuity gives a unique trajectory for a while; it does not promise that the trajectory lasts forever. Three one-line examples: $\dot x=x^2$, $x_0=1$ has the unique solution $x(t)=1/(1-t)$, which escapes at $t=1$. $\dot x=\sqrt{|x|}$, $x_0=0$ has two solutions, $x\equiv0$ and $x(t)=t^2/4$ (check: $\tfrac{\mathrm d}{\mathrm dt}\tfrac{t^2}{4}=\tfrac t2=\sqrt{t^2/4}$). $\dot x=-x$ is globally Lipschitz and forward complete.

Equilibria and linearisation

Examples. The unforced pendulum has $x_2=0$ and $\sin x_1=0$: the upright position $\theta=0$ and the hanging position $\theta=\pi$. The affine map $x_{t+1}=ax_t+b$ with $a\ne1$ has the single fixed point $x^\ast=b/(1-a)$.

Worked example (Module 13). A Parseval-network update acts on each singular value as $g(s)=(1+\beta)s-\beta s^3$. Then $g(1)=1$ and $g'(1)=1-2\beta$, so the criterion certifies that $s^\ast=1$ attracts when $|1-2\beta|\lt1$, i.e. $0\lt\beta\lt1$ (and repels for $\beta\gt1$). With $\beta=0.1$ and $s_0=0.5$ the iterates are $0.5375$, $0.5757$, $0.6142$, $0.6525$, $0.6899,\dots$, converging to $1$ and eventually shrinking the error by about $g'(1)=0.8$ per step.

Example. For the pendulum with $g/l=10$, $b=0.1$: at the upright equilibrium $A=\begin{bmatrix}0 & 1\\ 10\cos0 & -0.1\end{bmatrix}=\begin{bmatrix}0 & 1\\ 10 & -0.1\end{bmatrix}$ and $B=\begin{bmatrix}0\\ 1\end{bmatrix}$; at the hanging one $\cos\pi=-1$ turns the $10$ into $-10$. Section 2 reads off stability from the eigenvalues (Exercise D.1). Scaled coordinates. Module 11 works in normalised coordinates $z=S^{-1}x$ with $S=\mathrm{diag}(0.5,1.5)$; then a quadratic form becomes $x^{\top}Px=z^{\top}(S^{\top}PS)z$ and the Jacobian becomes $S^{-1}AS$.

Sampling, holding, saturating and switching

Example. For $\dot x=-x$ with $\Delta t=0.1$ Euler multiplies by $0.9$ per step, the exact factor is $e^{-0.1}=0.9048$. With $\Delta t=2.5$ Euler multiplies by $1-2.5=-1.5$ and diverges, while the true system shrinks by $e^{-2.5}=0.082$ per step. Sample-and-hold control computes $u_k=\kappa(x(t_k))$ only at the sampling instants and cannot react in between. If $\|\dot x\|\le M$ along the trajectory, then $\|x(t)-x(t_k)\|\le M\Delta t$ on $[t_k,t_k+\Delta t)$, so a margin checked at the samples must include this buffer: for the safe set $\{x\le1\}$, speed $|\dot x|\le2$ and $\Delta t=0.05$, require $x_k\le0.9$. Saturation clips inputs, componentwise $\mathrm{sat}(u)=\max\{-u_{\max},\min\{u_{\max},u\}\}$ or radially $u\mapsto u\min\{1,u_{\max}/\|u\|\}$ (defined as $0$ at $u=0$, and otherwise keeping the direction); a saturated linear feedback is a nonlinear system (Section 7).

Switched systems. $\dot x=f_{\sigma(t)}(x)$ with a piecewise-constant mode $\sigma(t)\in\{1,\dots,M\}$. The dwell time is the minimum time between consecutive switches. Switching fast enough between individually stable modes can destabilise, whereas a sufficiently long minimum dwell time preserves stability when, for example, every mode is linear with a Hurwitz matrix (Section 2). Rapid back-and-forth switching near a threshold is called chattering; an ideal switching law and its sampled implementation can then behave very differently.

Practice — Turn a model into an actual trajectory

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

2. Linear Systems: Solutions, Eigenvalues & Stability

Linear systems are the one class we can solve in closed form, and every local analysis of a nonlinear system reduces to one (Section 1). Their behaviour is written in the eigenvalues of a single matrix.

Solutions. Iterating the recursion (induction on $t$) gives $$x_t=A^tx_0+\sum_{s=0}^{t-1}A^{t-1-s}Bu_s .$$ For the scalar example of Section 1 ($a=0.5$, $b=1$, $u_s=1$, $x_0=0$): $x_t=\sum_{s=0}^{t-1}0.5^{t-1-s}=2(1-0.5^t)$, i.e. $0,1,1.5,1.75,\dots$ An affine scalar recurrence $x_{t+1}=ax_t+b$ ($a\ne1$) is solved by subtracting its fixed point $x^\ast=b/(1-a)$: $x_{t+1}-x^\ast=a(x_t-x^\ast)$, so $x_t-x^\ast=a^t(x_0-x^\ast)$. For $0\lt a\lt1$ the approach is monotone (no overshoot), for $-1\lt a\lt0$ it alternates in sign, for $|a|\gt1$ it diverges. In Module 6's explorer, the Euler step of $\dot x=-x+u+w$ under the proportional controller $u=\kappa(\rho-x)$, $x_{k+1}=x_k+\Delta t\,[-(1+\kappa)x_k+\kappa\rho+w]$, has $a=1-\Delta t(1+\kappa)$ and $x^\ast=(\kappa\rho+w)/(1+\kappa)$; with $\kappa=3$, $\rho=0.8$, $w=0.6$, $\Delta t=0.0025$ this is $a=0.99$ and $x^\ast=0.75$. From $x_0=0$ the state rises monotonically, so the supremum of the state along the infinite trajectory is the limit $x^\ast$ itself (approached, never exceeded).

Examples. $N=\begin{bmatrix}0 & 1\\ 0 & 0\end{bmatrix}$ is nilpotent ($N^2=0$), so $e^{Nt}=I+Nt=\begin{bmatrix}1 & t\\ 0 & 1\end{bmatrix}$: a double integrator drifts linearly. For the skew-symmetric $J=\begin{bmatrix}0 & -\theta\\ \theta & 0\end{bmatrix}$ the series sums to the rotation $e^{J}=\begin{bmatrix}\cos\theta & -\sin\theta\\ \sin\theta & \cos\theta\end{bmatrix}$. In general $J^{\top}=-J$ gives $(e^{J})^{\top}e^{J}=e^{-J}e^{J}=I$ by (ii) and (iv): the exponential of a skew-symmetric matrix is orthogonal (Module 13's SOC layers).

Modes. If $Av=\lambda v$ and $x_0=v$, then $x_t=\lambda^tv$ and $e^{At}v=e^{\lambda t}v$. For diagonalisable $A$ with $x_0=\sum_ic_iv_i$, $x_t=\sum_ic_i\lambda_i^tv_i$ and $x(t)=\sum_ic_ie^{\lambda_it}v_i$: every solution is a superposition of modes, and the slowest one (largest $|\lambda|$ in discrete time, largest $\mathrm{Re}\,\lambda$ in continuous time) dominates in the end. Example: $A=\begin{bmatrix}0.9 & -0.3\\ 0.3 & 0.9\end{bmatrix}$ has $\lambda=0.9\pm0.3j$, $|\lambda|=\sqrt{0.9}=0.9487$ and $\theta=\arctan(1/3)=0.3218$: trajectories spiral inward, shrinking by $0.9487$ per step and turning once about every $2\pi/0.3218\approx19.5$ steps (a nominal cycle length, not an exact period). The matrix $M=\begin{bmatrix}0 & -1\\ 1 & 1\end{bmatrix}$ (Module 8's integral-only multiplier loop with $cK_I=1$) has $\lambda=\tfrac12\pm\tfrac{\sqrt3}{2}j=e^{\pm j\pi/3}$ of modulus $1$: indeed $M^3=-I$ and $M^6=I$, a sustained oscillation of period $6$.

Schur and Hurwitz stability

Proof for diagonalisable $A$. $A^t=V\Lambda^tV^{-1}$ gives $\|A^t\|\le\|V\|\,\|V^{-1}\|\,\rho(A)^t$. Conversely, an eigenvector $v$ with $|\lambda|\ge1$ gives $\|A^tv\|=|\lambda|^t\|v\|\not\to0$ (for complex $v$ one of the real vectors $\mathrm{Re}\,v$, $\mathrm{Im}\,v$ does not decay). Non-diagonalisable matrices add polynomial factors $t^k\lambda^t$ from Jordan blocks, which still vanish when $|\lambda|\lt1$; this is why $q$ must be strictly larger than $\rho(A)$. Tests for $2\times2$ matrices: Hurwitz $\iff\tau\lt0$ and $\Delta\gt0$; Schur $\iff|\Delta|\lt1$ and $|\tau|\lt1+\Delta$ (the $n=2$ case of Jury's test). Examples: $\begin{bmatrix}0 & 1\\ -2 & -3\end{bmatrix}$ has $\tau=-3$, $\Delta=2$: Hurwitz (eigenvalues $-1,-2$). The spiral matrix above has $\Delta=0.9$, $\tau=1.8\lt1.9$: Schur. Marginal cases are not asymptotically stable: $M$ above oscillates forever, and the Jordan block $\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}^t=\begin{bmatrix}1 & t\\ 0 & 1\end{bmatrix}$ grows although both eigenvalues equal $1$.

Example (Module 9). For a column-stochastic transition matrix $P$ (columns sum to one, so $\|P\|_1=1$) and a discount $0\lt\gamma\lt1$: $\|\gamma P\|_1=\gamma\lt1$, hence $(I-\gamma P)^{-1}=\sum_t\gamma^tP^t$ with $\|(I-\gamma P)^{-1}\|_1\le1/(1-\gamma)$. Applied to an initial distribution $\mu$, $P^t\mu$ is the state distribution after $t$ steps, so $(1-\gamma)(I-\gamma P)^{-1}\mu$ is the discounted average of the successive distributions. Scalar check: $1/(1-0.5)=2=1+0.5+0.25+\cdots$

Phase portraits in the plane. For $\dot x=Ax$ with $A\in\mathbb R^{2\times2}$: two real eigenvalues of the same sign give a node (stable if negative), opposite signs a saddle, a complex pair a focus (spiral; stable if $\mathrm{Re}\,\lambda\lt0$), a purely imaginary pair a center (closed orbits). At a saddle with $\lambda_s\lt0\lt\lambda_u$ and eigenvectors $v_s,v_u$, $x(t)=c_se^{\lambda_st}v_s+c_ue^{\lambda_ut}v_u$ tends to $0$ only if $c_u=0$, i.e. $x_0$ lies on the line through $v_s$, the stable manifold; for a nonlinear system it is a curve tangent to $v_s$. Example: $A=\begin{bmatrix}0 & 1\\ 1 & 0\end{bmatrix}$ has $\lambda=\pm1$, $v_s=(1,-1)$: from $x_0=(1,-1)$ the state decays like $e^{-t}$, from $(1,-0.99)$ it diverges. Converging along one special curve is not stability. Conversely, feedback can create unwanted equilibria: where a safety filter exactly cancels the nominal controller, the closed loop can have an equilibrium on the boundary of the safe set that is even asymptotically stable, so the system stops short of its goal (Module 10). The explorer at the end of this page draws all four portraits.

Practice — Separate continuous and discrete stability tests

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

3. Lyapunov Functions, Invariant Sets & Regions of Attraction

Solving a nonlinear system is usually impossible. Lyapunov's idea is to certify its behaviour without solving it: find an "energy" that can only decrease along trajectories. The same device certifies safety, because the sublevel sets of such a function are sets that trajectories cannot leave.

Neither half implies the other. The center $\dot x=\begin{bmatrix}0 & 1\\ -1 & 0\end{bmatrix}x$ moves on circles: stable, not attractive. On the circle, $\dot\theta=1-\cos\theta$ carries every angle to $\theta=0$ (mod $2\pi$), but from $\theta_0=0.01$ the state first travels all the way around: attractive, not stable. An equilibrium reached only along a special curve, like the saddle of Section 2, is neither.

Examples. A quadratic form $V=x^{\top}Px$ with $P\succ0$ satisfies $\lambda_{\min}(P)\|x\|^2\le V(x)\le\lambda_{\max}(P)\|x\|^2$ (Primer A): positive definite, radially unbounded, with ellipsoidal sublevel sets. $V=x_1^2+x_2^4$ is positive definite but not quadratic. Positive definiteness belongs to a function on a domain: $V=1-\cos x_1+\tfrac12x_2^2$ is positive definite on $(-2\pi,2\pi)\times\mathbb R$ but not on $\mathbb R^2$. $V=\frac{x_1^2}{1+x_1^2}+x_2^2$ is positive definite but not radially unbounded: $V(x_1,0)\lt1$ for every $x_1$, so $\mathcal V(1)$ contains the whole $x_1$-axis and is unbounded, and a decreasing $V$ then no longer confines the state. Barbashin and Krasovskii's example (Kuznetsov et al., arXiv 2021, Example 1) is $\dot x_1=-\frac{2x_1}{(1+x_1^2)^2}+2x_2$, $\dot x_2=-\frac{2(x_1+x_2)}{(1+x_1^2)^2}$ with this $V$: $\dot V=-\frac{4}{(1+x_1^2)^2}\big(\frac{x_1^2}{(1+x_1^2)^2}+x_2^2\big)\lt0$ for $x\ne0$, yet every solution that starts with $x_1x_2\ge2$ is unbounded. A continuous $V$ has closed sublevel sets; if they are also bounded they are compact (Primer 0).

In words. A function that is zero only at the equilibrium and never increases along trajectories traps them near it; if it strictly decreases, they converge. No trajectory is ever computed. Why the assumptions: positive definiteness makes "small $V$" mean "small $x$"; strict decrease at every $x\ne0$ prevents trajectories from stalling on a level; radial unboundedness excludes escape along unbounded sublevel sets. Example where linearisation fails: $\dot x=-x^3$ has Jacobian $0$ at the origin (inconclusive), but $V=x^2/2$ gives $\dot V=-x^4\lt0$ for $x\ne0$, and $V$ is radially unbounded: GAS. It is not exponentially stable: the solution $x(t)=x_0/\sqrt{1+2x_0^2t}$ decays only like $t^{-1/2}$.

Worked example. $\dot x=-x+x^3$ has equilibria $0,\pm1$. With $V=x^2/2$: $\dot V=-x^2+x^4=-x^2(1-x^2)\lt0$ for $0\lt|x|\lt1$. Every sublevel set $\{x^2/2\le c\}$ with $c\lt\tfrac12$ lies in $|x|\lt1$ and is therefore an invariant subset of the region of attraction; their union is $(-1,1)$, which here is exactly $\mathcal R$ (the points $\pm1$ are equilibria, and from $|x_0|\gt1$ the state escapes in finite time).

Quadratic Lyapunov functions and the Lyapunov equation

For $\dot x=Ax$ and $V=x^{\top}Px$: $\dot V=\dot x^{\top}Px+x^{\top}P\dot x=x^{\top}(A^{\top}P+PA)x$. For $x_{t+1}=Ax_t$: $\Delta V=x^{\top}(A^{\top}PA-P)x$. The decrease condition becomes a matrix equation.

Worked example. $A=\begin{bmatrix}0 & 1\\ -2 & -3\end{bmatrix}$ (eigenvalues $-1,-2$), $Q=I$, $P=\begin{bmatrix}p_1 & p_2\\ p_2 & p_3\end{bmatrix}$. Multiplying out, $$A^{\top}P+PA=\begin{bmatrix}-4p_2 & p_1-3p_2-2p_3\\ p_1-3p_2-2p_3 & 2p_2-6p_3\end{bmatrix}=-I,$$ so $p_2=\tfrac14$, then $2p_2-6p_3=-1$ gives $p_3=\tfrac14$, and $p_1=3p_2+2p_3=\tfrac54$. $P=\begin{bmatrix}1.25 & 0.25\\ 0.25 & 0.25\end{bmatrix}$ has trace $1.5\gt0$ and determinant $0.25\gt0$, so $P\succ0$ (eigenvalues $(3-\sqrt5)/4\approx0.190983$ and $(3+\sqrt5)/4\approx1.309017$), and $V=x^{\top}Px$ certifies $\dot V=-\|x\|^2$. The exponential theorem with $k_1=(3-\sqrt5)/4$, $k_2=(3+\sqrt5)/4$, $k_3=1$, $a=2$ gives $\|x(t)\|\le\frac{3+\sqrt5}{2}\,e^{-(3-\sqrt5)t/2}\|x_0\|\le2.619\,e^{-0.3819t}\|x_0\|$ for $t\ge0$: valid, but slower than the true rate $e^{-t}$ of the slowest mode. Lyapunov bounds are certificates, not exact rates.

Forward invariance, controlled invariance and Nagumo's condition

Example. $x_{t+1}=2x_t+u_t$ with $|u_t|\le0.5$ and constraint set $K=[-1,1]$. The interval $[-0.5,0.5]$ is controlled invariant: $u=-x$ is admissible there and gives $x_{t+1}=x_t$. Under the fixed policy $u\equiv0$ only $\{0\}$ is invariant. From $x_0=0.6$ nothing works: the best input $u=-0.5$ gives $0.7$, then $0.9$, then $1.3\notin K$. Section 9 computes this kernel by backward iteration. Pitfall: invariance of a flow does not survive discretisation. $[0,\infty)$ is invariant for $\dot x=-x$, yet the Euler map $x\mapsto(1-\Delta t)x$ with $\Delta t=2$ sends $1$ to $-1$; an algorithm that follows a flow in finite steps inherits invariance only approximately (Module 9).

Intuition. Nothing happens in the interior; on the boundary the velocity must point inward or be tangent. For the unit disk, $h=1-\|x\|^2$ and $\dot x=Ax$: the rotation $A=\begin{bmatrix}0 & 1\\ -1 & 0\end{bmatrix}$ gives $\dot h=-2x^{\top}Ax=0$ (tangent, invariant); $A=\begin{bmatrix}-1 & 1\\ -1 & -1\end{bmatrix}$ gives $\dot h=2\|x\|^2\gt0$ on the boundary (inward, invariant); the constant field $\dot x=(1,0)$ gives $\dot h=-2$ at $(1,0)$ and leaves. For control-affine dynamics $\dot h=\nabla h^{\top}f+\nabla h^{\top}g\,u$, so the boundary condition is a linear inequality in $u$ (Section 4). Why the assumptions: for $h=-x^2$ and $C=\{0\}$ the gradient vanishes on the boundary, $\dot h(0)=0$ holds for every $f$, yet $\dot x=1$ leaves $C$. Without uniqueness, $\dot x=\sqrt{|x|}$ with $C=\{x\le0\}$ satisfies the boundary test at $x=0$ ($\dot h=-f(0)=0$) while the solution $t^2/4$ leaves.

Three relatives: contraction, control Lyapunov functions, and checking the conditions

Stability is not contraction. $\dot x=-x/(1+x^2)$ is GAS ($V=x^2$, $\dot V=-2x^2/(1+x^2)$), but it is not contracting. Trajectories from $2$ and $3$ first move apart, $\frac{\mathrm d}{\mathrm dt}(y-x)=f(3)-f(2)=-0.3+0.4=0.1\gt0$, and the gap reaches about $1.36$ before both converge; this rules out $c=1$. It does not by itself rule out a bound with an overshoot factor $c\gt1$, but none exists: since $|f|\le\tfrac12$, the trajectory from $x(0)=R$ still has $x(R)\ge R-\tfrac R2=\tfrac R2$ while $y\equiv0$ from $y(0)=0$, so the gap at time $R$ is at least $R/2$, whereas a contraction bound allows only $c\,e^{-\lambda R}R$, which is smaller for large $R$. Lyapunov decrease towards one point says nothing about distances between trajectories.

How the conditions are checked in practice. (i) Linear systems: the Lyapunov equation or LMI (Module 2). (ii) Polynomial systems: sum-of-squares. A polynomial is SOS if it is a sum of squares of polynomials, which holds iff $p(x)=m(x)^{\top}Gm(x)$ for a vector of monomials $m(x)$ and some $G\succeq0$; SOS implies $p\ge0$ (sufficient, not necessary), and finding $G$ is a semidefinite program. Example: $x^2-2x+2=\begin{bmatrix}1\\ x\end{bmatrix}^{\top}\begin{bmatrix}2 & -1\\ -1 & 1\end{bmatrix}\begin{bmatrix}1\\ x\end{bmatrix}$ with $G\succeq0$ (trace $3$, determinant $1$); indeed it equals $(x-1)^2+1$. (iii) Gridding with a Lipschitz margin: if $\Delta V$ is $L$-Lipschitz on a region, every point of which lies within $\tau$ of a grid point $x_i$, then $\Delta V(x_i)\lt-L\tau$ at all grid points implies $\Delta V(x)\le\Delta V(x_i)+L\|x-x_i\|\lt-L\tau+L\tau=0$ everywhere.

Practice — Use a scalar certificate along a trajectory

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

4. Class-K Functions & the Comparison Lemma

Lyapunov and barrier conditions are differential inequalities such as $\dot V\le-\alpha_3(\|x\|)$ or $\dot h\ge-\alpha(h)$. Comparison functions describe rates without committing to a formula, and the comparison lemma turns a scalar differential inequality into a bound on the solution.

Why they appear. For a time-invariant system with locally Lipschitz $f$, the origin is GAS exactly when there is a $\beta\in\mathcal{KL}$ with $\|x(t)\|\le\beta(\|x_0\|,t)$ for all $x_0$ and $t\ge0$ (Khalil 2002, Lemma 4.5): the first argument encodes "start close, stay close", the second "converge". A general Lyapunov theorem reads: $\alpha_1(\|x\|)\le V(x)\le\alpha_2(\|x\|)$ and $\dot V\le-\alpha_3(\|x\|)$ with $\alpha_i\in\mathcal K_\infty$ imply GAS; the exponential theorem of Section 3 is the case $\alpha_i(r)=k_ir^a$.

The linear case, in full. If $\dot v\le-av$, then $\frac{\mathrm d}{\mathrm dt}(e^{at}v)=e^{at}(\dot v+av)\le0$, so $v(t)\le e^{-at}v(0)$, whatever the sign of $v$. A nonlinear case: $\dot v\le-v^2$ with $v(0)=1$ is compared with $\dot w=-w^2$, $w(0)=1$, whose solution is $w(t)=1/(1+t)$; hence $v(t)\le1/(1+t)$, e.g. $v(3)\le0.25$.

From $\dot h\ge-\alpha(h)$ to $h(x(t))\ge0$: a preview of Module 10

Let $\eta(t)=h(x(t))$ along a closed-loop trajectory with $\dot\eta\ge-\alpha(\eta)$ and $\eta(0)\ge0$, where $\alpha$ is extended class $\mathcal K$ (and locally Lipschitz, for simplicity).

1. Comparison system. $\dot y=-\alpha(y)$, $y(0)=\eta(0)\ge0$. Since $\alpha(0)=0$, $y\equiv0$ is a solution, and by uniqueness no other solution can cross it: $y(t)\ge0$ for all $t$.
2. Flip the signs, because the lemma is stated with "$\le$". Put $v=-\eta$, $w=-y$ and $F(s)=\alpha(-s)$. Then $\dot v=-\dot\eta\le\alpha(\eta)=\alpha(-v)=F(v)$, $\dot w=-\dot y=\alpha(y)=\alpha(-w)=F(w)$, and $v(0)=w(0)$.
3. Conclude. The comparison lemma gives $v\le w$, i.e. $h(x(t))=\eta(t)\ge y(t)\ge0$: the trajectory never leaves $C=\{h\ge0\}$.

For the linear choice $\alpha(r)=\alpha_0r$ the comparison solution is explicit: $h(x(t))\ge h(x_0)e^{-\alpha_0t}$. With $\alpha_0=2$ and $h(x_0)=0.5$, $h(x(1))\ge0.5e^{-2}\approx0.068$. The barrier may decrease, but ever more slowly as the boundary approaches. Outside $C$ ($h\lt0$) the bound $\dot h\ge-\alpha(h)\gt0$ pushes $h$ back up, which is why $\alpha$ must be defined for negative arguments: small violations are corrected. Discrete time needs no lemma: $h(x_{t+1})-h(x_t)\ge-\kappa\,h(x_t)$ with $\kappa\in(0,1]$ gives $h(x_t)\ge(1-\kappa)^th(x_0)\ge0$ by induction.

Example: a wall. $\dot x=u$, safe set $C=\{x\le1\}$, $h=1-x$: $L_fh=0$, $L_gh=-1$, so $K_{\rm cbf}(x)=\{u:\ u\le\alpha_0(1-x)\}$. With $\alpha_0=2$ at $x=0.8$ the speed towards the wall may not exceed $0.4$. The safety filter $u=\min\{u_{\rm nom},\alpha_0(1-x)\}$ leaves any command away from the wall untouched and caps the approach speed in proportion to the remaining distance; even at the cap, $1-x(t)=(1-x_0)e^{-\alpha_0t}$ never reaches $0$.

Example. $\dot x=-x+d$ has $x(t)=e^{-t}x_0+\int_0^te^{-(t-s)}d(s)\,\mathrm ds$, so $|x(t)|\le e^{-t}|x_0|+\sup|d|$ because $\int_0^te^{-(t-s)}\mathrm ds\le1$: ISS with $\beta(r,t)=re^{-t}$ and $\gamma(r)=r$. Non-example: $\dot x=-x+xd$ is GAS for $d=0$, but $d\equiv2$ gives $\dot x=x$, which diverges. Module 10's input-to-state safety is the barrier analogue: the safe set is inflated by $\gamma(\|d\|_\infty)$ rather than kept exactly.

Practice — Translate differential inequalities into bounds

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

5. Feedback Control: State Feedback, LQR & PID

Feedback means computing the input from measurements, so that the system corrects its own errors. Everything a reinforcement-learning agent calls a policy is a feedback controller in this sense.

Worked example: deadbeat gain and overshoot (Module 1's explorer). $x_{t+1}=x_t+u_t+w_t$ with $u_t=-k(x_t-x_g)$. The error $e_t=x_t-x_g$ obeys $e_{t+1}=(1-k)e_t+w_t$. Without noise $e_t=(1-k)^te_0$: for $0\lt k\lt1$ the error shrinks monotonically; $k=1$ is deadbeat ($e_1=0$ from any $e_0$); for $1\lt k\lt2$ the state overshoots and the error alternates in sign while shrinking; for $k\ge2$, $|1-k|\ge1$ and a nonzero initial error does not converge to zero. With independent noise of variance $\sigma^2$, $\mathrm{Var}(e_{t+1})=(1-k)^2\mathrm{Var}(e_t)+\sigma^2\to\frac{\sigma^2}{1-(1-k)^2}=\frac{\sigma^2}{k(2-k)}$. This stationary variance is smallest at the deadbeat gain, where it still equals $\sigma^2$: feedback cannot remove noise that enters after the input is chosen. For $k=0.5$ or $k=1.5$ the standard deviation is $\sigma/\sqrt{0.75}\approx1.155\sigma$. The limit describes a distribution, not an equilibrium (Primer C).

The linear-quadratic regulator (LQR)

Worked example. $A=B=Q=R=1$: the DARE reads $P=1+P-\frac{P^2}{1+P}$, i.e. $P^2=P+1$, so $P=\frac{1+\sqrt5}{2}\approx1.618$, $K=\frac{P}{1+P}\approx0.618$, and the closed loop is $x_{t+1}=0.382\,x_t$. The weights set the aggressiveness. Keeping $A=B=Q=1$ and varying $R=r$: $P=\frac{1+\sqrt{1+4r}}{2}$ and $K=\frac{P}{r+P}$, so $r=0.1$ gives $K=0.916$ (nearly deadbeat, cheap input), $r=1$ gives $0.618$, and $r=10$ gives $0.270$ (gentle, expensive input). In Module 4 the weights used to design $K$, $(W_x(\theta),W_u(\theta))$, are the tuning parameters, while fixed weights $(Q,R)$ evaluate the resulting closed loop. Average cost with noise: for $x_{t+1}=Ax_t+Bu_t+w_t$ with independent zero-mean $w_t$ of covariance $W$ and $u=-Kx$ stabilising, $\lim_{T\to\infty}\frac1T\mathbb E\sum_{t\lt T}(x_t^{\top}Qx_t+u_t^{\top}Ru_t)=\operatorname{tr}(WP_K)$, where $P_K=Q+K^{\top}RK+(A-BK)^{\top}P_K(A-BK)$ (a Stein equation, Section 3). In the scalar example $P_K=\frac{1+0.618^2}{1-0.382^2}=1.618=P$, so with $W=0.01$ the average cost is $0.0162$; the LQR gain also minimises this average cost.

PID control and integral action

Example. Plant $x_{t+1}=0.5x_t+u_t+d$ with an unknown constant $d=1$ and reference $r=1$. P-only control $u=K_P(r-x)$ with $K_P=1$ settles where $x=0.5x+(1-x)+1$, i.e. $x=4/3$: an offset of $-1/3$ that a larger $K_P$ shrinks but never removes. PI control with $K_P=0.5$, $K_I=0.25$ has state $(x_t,I_{t-1})$ and loop matrix $\begin{bmatrix}0.5-K_P-K_I & K_I\\ -1 & 1\end{bmatrix}$ with trace $0.75$ and determinant $0.5-K_P=0$, eigenvalues $0$ and $0.75$: it converges, and by the fact above to $x=1$ exactly (the integrator settles at $I=-2$, cancelling $d$).

The multiplier as a controller. The Lagrangian update $\lambda_{k+1}=[\lambda_k+K_I(J_C(\theta_k)-d)]_+$ of constrained RL is an integral controller: the plant is the policy optimiser, the output is the expected cost $J_C$, the reference is the budget $d$, and the error is the constraint violation. If it converges with $\lambda\gt0$, the constraint holds with equality. In the simplified multiplier loop of Module 8 (the matrix $M$ of Section 2), integral-only updates give a sustained oscillation (for $0\lt cK_I\lt4$, Exercise D.2); this is a property of that loop, not of integral control in general (for $\dot x=-x+u$, $u=I$, $\dot I=0.1(r-x)$ the error has the real poles $-0.11$ and $-0.89$). PID Lagrangians add a proportional term (react to the current violation) and a derivative term (react to increases) to improve the damping (Module 8). A small $K_I$ makes the multiplier slow compared with the policy: two timescales.

PD control and impedance. $u=-k_p\theta-k_d\dot\theta$ is P on the angle and D on its rate (the error derivative when the reference is $0$). It acts like a virtual spring of stiffness $k_p$ and a damper $k_d$; an impedance controller $u=-k(x-x_{\rm ref})-b\dot x$ prescribes exactly this spring–damper behaviour around a desired position (Module 6). A saturating PD law, $u=\mathrm{sat}(\cdot)$ or a small tanh network, keeps the input bounded (Module 14's explorer).

Example. Scalar $a=2$, $b=c=1$ (unstable plant, only $x$ measured): $K=1.5$ gives $a-bK=0.5$, $L=1.8$ gives $a-Lc=0.2$, and the closed loop has eigenvalues $\{0.5,0.2\}$. The augmented state (plant state, estimation error) is the state of the closed loop; Module 11 writes it as $\xi=(x,\hat x-x)$.

Practice — Design feedback and then check its assumptions

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

6. Transfer Functions, Frequency Response, Gains & Parseval

Sections 1–5 followed one trajectory. Robustness asks a different question: how much can a system amplify a whole input signal? For linear systems the answer is read off frequency by frequency, and this is where the complex-valued conditions of Modules 2 and 14 come from.

Examples. $w_t=0.5^t$ has $\|w\|_2^2=1+\frac14+\frac1{16}+\cdots=\frac43$. The constant signal $w_t=1$ has $\|w\|_\infty=1$ but infinite energy. The zero initial state matters: a nonzero $x_0$ produces output even for $w=0$, and a gain measures amplification of the input only.

LTI systems are convolutions. From the solution formula of Section 2 with $x_0=0$, $$z_t=\sum_{s=0}^{t}h_{t-s}\,w_s,\qquad h_0=D,\quad h_k=CA^{k-1}B\ \ (k\ge1),$$ where $(h_k)$ is the impulse response (the output for $w=(1,0,0,\dots)$). For $x_{t+1}=0.5x_t+w_t$, $z_t=x_t$: $h_0=0$, $h_k=0.5^{k-1}$. A finite impulse response (FIR) filter has $h_k=0$ for $k\gt r$, for example the moving average $z_t=\frac12(w_t+w_{t-1})$. The sum $\|h\|_1=\sum_k|h_k|$ (here $2$) bounds the peak gain, $\|z\|_\infty\le\|h\|_1\|w\|_\infty$.

Examples. $H(z)=\frac1{z-0.5}=\frac{z^{-1}}{1-0.5z^{-1}}=z^{-1}+0.5z^{-2}+0.25z^{-3}+\cdots$ reproduces $h_k=0.5^{k-1}$. If $A$ is nilpotent ($A^r=0$), the Neumann series of Section 2 terminates: $(zI-A)^{-1}=z^{-1}(I-A/z)^{-1}=z^{-1}I+z^{-2}A+\cdots+z^{-r}A^{r-1}$, so $H$ is a polynomial in $z^{-1}$, an FIR filter (Section 8). A chain of $r$ integrators ($h^{(r)}=\mu$) has $G(s)=1/s^r$. Properness: $G$ is proper if $G(\infty)$ is finite (numerator degree at most denominator degree) and strictly proper if $G(\infty)=0$ ($D=0$). Every $(A,B,C,D)$ gives a proper $G$; the pure differentiator $G(s)=s$ has no state-space realisation and amplifies high frequencies without bound. Real-rational means a ratio of polynomials with real coefficients; the stable proper real-rational transfer matrices form the set $\mathcal{RH}_\infty$. A stable proper inverse $G^{-1}$ requires $G(\infty)=D$ invertible and all zeros of $G$ in the stable region. For $G(s)=-s/(s+1)$ (Module 14): $1+G(s)=1/(s+1)$, whose inverse $s+1$ is improper, so the feedback loop through $1+G$ cannot be inverted by a bounded operator.

Poles, impulse responses and a positivity argument (Module 10). Pole placement on an integrator chain gives $\ddot h+(p_1+p_2)\dot h+p_1p_2h=e(t)$, i.e. the transfer function $\frac1{(s+p_1)(s+p_2)}$ with real poles $-p_1,-p_2$ ($0\lt p_1\lt p_2$). Its impulse response is the convolution of two decaying exponentials, $$g(t)=\frac{e^{-p_1t}-e^{-p_2t}}{p_2-p_1}\ \ge\ 0,$$ so $h(t)=h_{\rm hom}(t)+\int_0^tg(t-s)e(s)\,\mathrm ds\ge h_{\rm hom}(t)$ whenever the forcing is $e\ge0$: a nonnegative input can only raise $h$. With complex poles $g$ changes sign and this argument fails.

Example. For $H(z)=\frac1{z-0.5}$: $|H(e^{j0})|=2$ (a constant input $1$ produces $0,1,1.5,1.75,\dots\to2$), while $|H(e^{j\pi})|=|1/(-1.5)|=0.667$ (the alternating input $1,-1,1,\dots$ is attenuated).

Example. $w=(1,1,0,0,\dots)$: $\hat w=1+e^{-j\omega}$, $|\hat w|^2=(1+\cos\omega)^2+\sin^2\omega=2+2\cos\omega$, and $\frac1{2\pi}\int_{-\pi}^{\pi}(2+2\cos\omega)\,\mathrm d\omega=2=1^2+1^2$.

Example. $\|1/(z-0.5)\|_\infty=2$, attained at $\omega=0$: a long constant input yields outputs close to $2$, so the energy ratio approaches $4$. The $\mathcal H_2$ norm is an average instead of a worst case: $\|H\|_2^2=\sum_k\|h_k\|_F^2=\frac1{2\pi}\int_{-\pi}^{\pi}\|H(e^{j\omega})\|_F^2\mathrm d\omega$, the energy of the impulse response or the output power under unit white noise. Here $\|H\|_2^2=\sum_{k\ge1}0.25^{k-1}=\frac43$, so $\|H\|_2=1.155\lt\|H\|_\infty=2$. For state-space systems $\|H\|_2^2=\operatorname{tr}(CW_cC^{\top})+\|D\|_F^2$ with the controllability Gramian $W_c$ of Section 8.

Example. $N=2$: $F=\frac1{\sqrt2}\begin{bmatrix}1 & 1\\ 1 & -1\end{bmatrix}$. Circular convolution with the kernel $(a,b)$ is the circulant $\begin{bmatrix}a & b\\ b & a\end{bmatrix}$, whose eigenvalues $a+b$ and $a-b$ are $\tilde c_0$ and $\tilde c_1$, with the DFT vectors as eigenvectors.

Return difference, gain and delay margins

Break the loop $u=-Kx$ at the plant input. The loop transfer function is $L(z)=K(zI-A)^{-1}B$ (single input), and the matrix determinant lemma gives $\det(zI-A+BK)=\det(zI-A)\,\big(1+L(z)\big)$: barring cancellations, the closed-loop poles are the zeros of the return difference $1+L(z)$. The smallest distance $\min_\omega|1+L(e^{j\omega})|$ of the Nyquist curve $L(e^{j\omega})$ from $-1$ measures how close the loop is to instability. The gain margin is the range of factors $\kappa$ for which $u=-\kappa Kx$ still stabilises; the delay margin is the largest extra input delay the loop tolerates. Example (Module 4). The LQR gain $K=0.618$ for $A=B=1$ gives $L(z)=\frac{0.618}{z-1}$; at $\omega=\pi$, $|1+L(-1)|=|1-0.309|=0.691$, the minimum over $\omega$. The closed-loop pole $1-0.618\kappa$ is stable iff $0\lt\kappa\lt3.236$. With a one-step delay, $x_{t+1}=x_t-0.618x_{t-1}$ is still stable (roots of modulus $\sqrt{0.618}=0.786$); a two-step delay puts roots on the unit circle ($e^{\pm j\pi/5}$ for the unrounded gain $K=(\sqrt5-1)/2$), and a three-step delay destabilises. Continuous-time LQR guarantees $|1+L(j\omega)|\ge1$ (gain margin $[\tfrac12,\infty)$, phase margin at least $60^\circ$); discrete-time LQR does not, as $0.691\lt1$ shows.

Dissipativity: from a storage function to a gain

Storage implies gain. If $V\ge0$, $V(x_0)=0$ and $V(x_{t+1})-V(x_t)\le\gamma^2\|w_t\|^2-\|z_t\|^2$, summing over $t=0,\dots,T$ telescopes the left side to $V(x_{T+1})-V(x_0)\ge0$, hence $\sum_{t\le T}\|z_t\|^2\le\gamma^2\sum_{t\le T}\|w_t\|^2$ for every $T$.

Example. For $x_{t+1}=0.5x_t+w_t$, $z_t=x_t$, take $V=2x^2$ and $\gamma=2$: $2(0.5x+w)^2-2x^2-4w^2+x^2=-0.5x^2+2xw-2w^2=-0.5(x-2w)^2\le0$. The certificate proves the gain is at most $2=\|H\|_\infty$, so it is tight.

For LTI systems and $V=x^{\top}Px$ the dissipation inequality is a matrix inequality in $P$, $$\begin{bmatrix}A^{\top}PA-P+C^{\top}C & A^{\top}PB+C^{\top}D\\ B^{\top}PA+D^{\top}C & B^{\top}PB+D^{\top}D-\gamma^2I\end{bmatrix}\preceq0,$$ here $\begin{bmatrix}-0.5 & 1\\ 1 & -2\end{bmatrix}\preceq0$ (determinant $0$, trace $-2.5$). The KYP lemma (Module 2) turns this into an equivalence, under hypotheses: for Schur $A$ and controllable $(A,B)$ (a minimal realisation suffices) the LMI has a solution $P\succeq0$ exactly when $\|H\|_\infty\le\gamma$, i.e. when $\sigma_{\max}(H(e^{j\omega}))\le\gamma$ at every $\omega$ (the bounded-real lemma); with strict inequalities on both sides controllability is not needed. One matrix inequality replaces infinitely many frequency conditions. Feasibility always implies the gain bound (the storage argument above), but the converse can fail without the hypotheses: for $A=2$, $B=0$, $C=1$, $D=0$ the zero-state transfer function is $H=0$, yet the first diagonal entry of the LMI is $3P+1$, which cannot be nonpositive for $P\ge0$. That is why frequency-domain criteria (the circle criterion, IQCs) reappear as LMIs in Modules 2 and 14.

Practice — Connect time responses to gains

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

7. Feedback with Nonlinearities: Lur'e Systems & Sector Conditions

A neural network in a control loop, a saturating actuator and a clipped multiplier update have the same structure: a linear system in feedback with a memoryless nonlinearity about which we only know inequalities. Classical absolute-stability theory asks for stability for every nonlinearity that satisfies them.

Examples. Saturated state feedback $u=\mathrm{sat}(-Kx)$ on $x_{t+1}=Ax_t+Bu_t$: $v=-Kx$, $w=\mathrm{sat}(v)$. A neural controller $u=W_1\phi(W_0x)$ on a linear plant: the plant and all weight matrices form the linear block, and the stacked activations form a diagonal static nonlinearity, one scalar function per neuron.

Examples. ReLU, tanh and saturation are slope-restricted on $[0,1]$ and hence in the sector $[0,1]$; leaky ReLU $\max\{v,av\}$ ($0\lt a\lt1$) on $[a,1]$. The chord slope $\tanh(v)/v$ decreases in $|v|$, so on $|v|\le2$ tanh lies in the local sector $[\tanh(2)/2,\,1]\approx[0.482,1]$: knowing that pre-activations stay bounded buys a positive lower sector bound. The slope restriction on that interval is weaker, $[\operatorname{sech}^2(2),1]\approx[0.0707,1]$, since $\tanh'(v)=\operatorname{sech}^2v$ and chords near $|v|=2$ are much flatter than the chord to the origin.

Where the condition comes from. Multiplying the sector QC by $\lambda\ge0$ and adding it to a dissipation inequality with storage $x^{\top}Px$ gives one LMI (the S-procedure); by the KYP lemma (under its stability and controllability hypotheses, Section 6) that LMI is feasible exactly when the frequency condition holds.

Worked example (continuous time). $G(s)=\frac1{(s+1)(s+2)}$ with $\phi$ in $[0,k]$. Constant gains give $s^2+3s+2+k$, Hurwitz for every $k\gt-2$. Circle: $\mathrm{Re}\,G(j\omega)=\frac{2-\omega^2}{(2-\omega^2)^2+9\omega^2}$ has minimum $-\frac{\sqrt2}{12+9\sqrt2}\approx-0.0572$ at $\omega^2=2+3\sqrt2$ ($\omega\approx2.50$), so the circle criterion certifies only $k\lt9+6\sqrt2\approx17.5$. Popov with $q=\tfrac13$: $\mathrm{Re}[(1+j\omega q)G]=\mathrm{Re}\,G-q\,\omega\,\mathrm{Im}\,G=\frac{2}{(2-\omega^2)^2+9\omega^2}\gt0$, so every finite sector $[0,k]$ is certified for time-invariant $\phi$. The circle criterion, which also covers time-varying gains, is conservative here. Discrete time, first order. $G(z)=\frac1{z-0.5}$: $\mathrm{Re}\,G(e^{j\omega})=\frac{\cos\omega-0.5}{1.25-\cos\omega}$, minimal $-\tfrac23$ at $\omega=\pi$, so the circle criterion certifies $\beta\lt1.5$; the constant-gain loop $x_{t+1}=(0.5-k)x_t$ is stable exactly for $-0.5\lt k\lt1.5$. For this first-order loop the two agree.

Loop transformations. A sector can be moved by redefining the blocks. $\tilde\phi(v)=\phi(v)-\alpha v$ lies in $[0,\beta-\alpha]$ if the term $\alpha v$ is moved into the linear part ($\tilde G=G/(1+\alpha G)$ in negative feedback, which requires $1+\alpha G$ to be invertible: well-posedness again). The symmetric form writes $\phi(v)=\frac{\alpha+\beta}2v+\frac{\beta-\alpha}2\delta(v)$ with $\delta$ in the sector $[-1,1]$: a gain-one uncertainty around the midpoint gain, which is the small-gain view. Module 14 transforms slope bounds $[\mu,\nu]$ the same way; the transformed nonlinearity is then a monotone relation, a set of pairs $(a,b)$ with $(a-a')(b-b')\ge0$ for any two of its pairs, which may be multivalued (a vertical piece in its graph).

Example. $G_1=\frac1{z-0.5}$ (gain $2$) in feedback with $G_2=0.4\tanh$ (gain $0.4$): $0.8\lt1$, stable. A network controller with $\pi(0)=0$ and Lipschitz constant $L$ has gain $L$, since $\|\pi(x)\|\le L\|x\|$ (in general $L$ is an incremental gain, Module 12), so in feedback with a plant of gain $\gamma$ it is certified by $\gamma L\lt1$. Without $\pi(0)=0$ there is no zero-referenced gain: $\pi(x)=0.4x+1$ has $L=0.4$, and with the plant $\frac1{z-0.5}$ ($\gamma=2$) we get $\gamma L=0.8\lt1$, yet $x_{t+1}=0.5x_t+\pi(x_t)$ has no equilibrium at the origin (it settles at $x=10$). Shift the coordinates to a closed-loop equilibrium first, or use incremental gains for both blocks. Three cautions. (i) Small gain ignores signs and phases, so it is conservative: $\frac1{z-0.5}$ in negative feedback with a constant gain $k$ is stable for all $0\le k\lt1.5$, but small gain only certifies $|k|\lt0.5$. (ii) A finite gain for each component is not a closed-loop certificate: gains $2$ and $1$ in positive feedback give $x_{t+1}=1.5x_t$, unstable. (iii) A cascade (series connection) of stable systems is always stable with gain at most $\gamma_1\gamma_2$, without any condition; this is why Module 14's Youla parameterisation, which keeps the learned block outside every feedback loop, needs no small-gain test.

Practice — Read a nonlinearity as a constraint

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

8. Controllability, Gramians & State-Space Realizations

Can the input move the state everywhere? Can the output reveal it? The answers are rank tests, and their quantitative versions, the Gramians, reappear as storage functions in Module 13. The second half turns input–output descriptions (a transfer function, a convolution) into state-space form.

Example. Position and velocity, $A=\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}$. A force on the velocity, $B=\begin{bmatrix}0\\ 1\end{bmatrix}$: $\mathcal C=\begin{bmatrix}0 & 1\\ 1 & 1\end{bmatrix}$ has determinant $-1$, controllable. A push on the position only, $B=\begin{bmatrix}1\\ 0\end{bmatrix}$: $\mathcal C=\begin{bmatrix}1 & 1\\ 0 & 0\end{bmatrix}$ has rank $1$, and the velocity can never be changed. Measuring the position, $C=\begin{bmatrix}1 & 0\end{bmatrix}$: $\mathcal O=\begin{bmatrix}1 & 0\\ 1 & 1\end{bmatrix}$, observable (the velocity is $y_1-y_0$). Measuring the velocity, $C=\begin{bmatrix}0 & 1\end{bmatrix}$: $\mathcal O=\begin{bmatrix}0 & 1\\ 0 & 1\end{bmatrix}$, and the absolute position is invisible.

Example. $A=\mathrm{diag}(0.5,2)$, $B=\begin{bmatrix}0\\ 1\end{bmatrix}$: the mode $0.5$ cannot be influenced but decays anyway, so $(A,B)$ is stabilisable, and $K=\begin{bmatrix}0 & 1.8\end{bmatrix}$ gives $A-BK=\mathrm{diag}(0.5,0.2)$. With $B=\begin{bmatrix}1\\ 0\end{bmatrix}$ the unstable mode $2$ is uncontrollable and no feedback helps: not stabilisable.

Meaning: minimum energy. The least input energy $\sum_k\|u_k\|^2$ that steers $0$ to $x$ in $N$ steps is $x^{\top}W_N^{-1}x$ when $W_N\succ0$, i.e. when $\mathcal C_N$ has rank $n$ (the least-norm solution of $\mathcal C_N\mathbf u=x$ is $\mathbf u=\mathcal C_N^{\top}W_N^{-1}x$). If $W_N$ is singular, only $x\in\operatorname{range}W_N$ can be reached in $N$ steps, at the least energy $x^{\top}W_N^{\dagger}x$ with the pseudoinverse $W_N^{\dagger}$; for the delay register below and $N=1$, $W_1=\mathrm{diag}(1,0)$ and $x=(0,1)$ is unreachable. Directions in which the Gramian is small are expensive to reach. The inverse Gramian is thus a "required supply", which is why it can serve as a storage function (Module 13's Cayley–Gramian layers). Examples. Scalar $a=0.5$, $b=1$: $W_c=\sum_k0.25^k=\frac43$, and indeed $\frac43-0.25\cdot\frac43=1$. The delay register $A=\begin{bmatrix}0 & 0\\ 1 & 0\end{bmatrix}$, $B=\begin{bmatrix}1\\ 0\end{bmatrix}$ has $A^2=0$ and $W_c=BB^{\top}+ABB^{\top}A^{\top}=\mathrm{diag}(1,0)+\mathrm{diag}(0,1)=I$: a finite sum. The dual observability Gramian $W_o=\sum_k(A^{\top})^kC^{\top}CA^k$ measures the output energy of the free response, $\sum_t\|y_t\|^2=x_0^{\top}W_ox_0$, and $\|H\|_2^2=\operatorname{tr}(CW_cC^{\top})+\|D\|_F^2$ (Section 6; here $\frac43$ for $c=1$).

FIR filters as delay registers. $y_t=h_0w_t+h_1w_{t-1}+h_2w_{t-2}$ is realised by storing the last two inputs, $x_t=(w_{t-1},w_{t-2})$: $$x_{t+1}=\begin{bmatrix}0 & 0\\ 1 & 0\end{bmatrix}x_t+\begin{bmatrix}1\\ 0\end{bmatrix}w_t,\qquad y_t=\begin{bmatrix}h_1 & h_2\end{bmatrix}x_t+h_0w_t .$$ The state update inserts the new sample and shifts the register ($z^{-1}$ is one delay). Check: $(zI-A)^{-1}=\begin{bmatrix}z^{-1} & 0\\ z^{-2} & z^{-1}\end{bmatrix}$, so $C(zI-A)^{-1}B+D=h_0+h_1z^{-1}+h_2z^{-2}$. $A$ is nilpotent ($A^2=0$), and the zero initial state is exactly zero padding. For multi-channel convolutions every entry becomes an identity block or a kernel matrix (Module 12, Exercise 12.6). An IIR example: $H(z)=\frac{b_1z+b_0}{z^2+a_1z+a_0}$ is realised by $A=\begin{bmatrix}-a_1 & -a_0\\ 1 & 0\end{bmatrix}$, $B=\begin{bmatrix}1\\ 0\end{bmatrix}$, $C=\begin{bmatrix}b_1 & b_0\end{bmatrix}$, $D=0$, since $\det(zI-A)=z^2+a_1z+a_0$ and $(zI-A)^{-1}B=\frac{1}{\det(zI-A)}\begin{bmatrix}z\\ 1\end{bmatrix}$. Acausal FIR filters such as $h_{-1}w_{t+1}+h_0w_t+h_1w_{t-1}$ use a future sample and have no causal realisation, but delaying the output by one step makes them causal; inside an IQC multiplier they are allowed, because the multiplier only appears in an inequality summed over the whole signal (Modules 2 and 14).

2-D (Roesser) systems in one paragraph. For images, "time" runs in two directions $(i_1,i_2)$. A Roesser model carries a horizontal state $x_1$, passed to the right, and a vertical state $x_2$, passed downwards: $$\begin{bmatrix}x_1(i_1{+}1,i_2)\\ x_2(i_1,i_2{+}1)\\ y(i_1,i_2)\end{bmatrix}=\begin{bmatrix}A_{11} & A_{12} & B_1\\ A_{21} & A_{22} & B_2\\ C_1 & C_2 & D\end{bmatrix}\begin{bmatrix}x_1(i_1,i_2)\\ x_2(i_1,i_2)\\ u(i_1,i_2)\end{bmatrix}.$$ A 2-D convolution is a 2-D FIR filter with such a realisation, and a convolutional network becomes a 2-D Lur'e system. The analogy has limits: there is no single ordering of time, dissipation inequalities telescope along rows and along columns separately, and minimal realisations are not guaranteed in general (Module 12).

Practice — Ask which states inputs can reach

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

9. Optimal Control, Dynamic Programming & MPC

So far the controller was given. Optimal control computes it from a cost; dynamic programming turns the computation into a backward recursion on value functions; MPC solves a short-horizon version online; reachability runs the same recursion with a minimum over time instead of a sum.

Tiny example. States $x\in\{0,1,2,3\}$, inputs $u\in\{-1,0,1\}$ that keep $x+u$ in range, $\ell=x^2+|u|$, $N=2$, $V_2=0$. Then $V_1(x)=x^2$ (use $u=0$), and $V_0(x)=x^2+\min_u[|u|+(x+u)^2]$ gives $V_0=(0,2,6,14)$; from $x=2$ the optimal first move is $u=-1$ ($4+1+1=6$). In RL the same split reads "reward now plus discounted value of the next state" (Primer E). When the exact value function is too complicated, one propagates an upper bound instead: if $\bar V_N\ge V_f$ and $\bar V_k(x)\ge\ell(x,u)+\bar V_{k+1}(f(x,u))$ for the inputs actually used, then $\bar V_k$ bounds their cost-to-go from above, and quadratic $\bar V_k$ turn each step into a matrix inequality (Module 12 does this layer by layer).

Example. $\dot x=u$, $\ell=x^2+u^2$, $V=px^2$: $0=\min_u[x^2+u^2+2pxu]$ gives $u=-px$ and $0=(1-p^2)x^2$, so $p=1$, $V=x^2$, $u=-x$ (the continuous Riccati equation of Section 5 with $A=0$, $B=Q=R=1$). Viscosity solutions (a black box). Value functions are often not differentiable. The minimum time to reach $\{-1,1\}$ from $x\in[-1,1]$ with $\dot x=u$, $|u|\le1$, is $V(x)=1-|x|$, with a kink at $0$ where the optimal action switches. Where $V$ is differentiable, the HJB equation is $0=\min_{|u|\le1}[1+V'(x)u]=1-|V'(x)|$. For the viscosity formulation use $|V'(x)|-1=0$ on $(-1,1)$ with boundary values $V(-1)=V(1)=0$: the sign matters in the inequalities for smooth test functions touching $V$ from above or below. At the kink, a test function touching from above has slope in $[-1,1]$, so the subsolution inequality $|\varphi'|-1\le0$ holds; no smooth test function touches from below there. This boundary-value problem has the unique continuous viscosity solution $1-|x|$. More general HJB and reachability equations need their own comparison-principle and boundary assumptions for uniqueness; the modules use these results as an interface to grid-based solvers.

Reachability and safety value functions

Continuous time (black box, Module 10): on a horizon $[0,T]$, $V$ is the viscosity solution of the HJI variational inequality $$\min\Big\{l(x)-V(x,t),\ \ \partial_tV(x,t)+\max_{u\in U}\min_{d\in\mathcal D}\nabla_xV(x,t)^{\top}f(x,u,d)\Big\}=0,\qquad V(x,T)=l(x).$$ The robust Hamiltonian $H(x,p)=\max_u\min_dp^{\top}f(x,u,d)$ is the best worst-case rate of change of $V$ when $p=\nabla V$ (the gradient turns a velocity into a rate of change of $V$), and the safe action on the boundary of the safe set is $\arg\max_u\min_d\nabla V^{\top}f(x,u,d)$: the controller commits, then the disturbance answers.

Worked example: the viability kernel by backward iteration. $x_{t+1}=2x_t+u_t$, $|u_t|\le0.5$, $K=[-1,1]$. The controllable predecessor of a set $S$ is $\mathrm{Pre}(S)=\{x:\ \exists u\in U,\ f(x,u)\in S\}$ (robust version: $\exists u\ \forall d$). Iterate $X^0=K$, $X^{k+1}=X^k\cap\mathrm{Pre}(X^k)$, the states that can stay in $K$ for $k+1$ more steps. For $S=[-c,c]$, $\mathrm{Pre}(S)=\{x:\ |2x|\le c+0.5\}$, so $c\mapsto(c+0.5)/2$: $1\to0.75\to0.625\to0.5625\to\cdots\to0.5$. The kernel is $[-0.5,0.5]$, the largest controlled invariant subset of $K$ found in Section 3; it is the greatest fixed point of the iteration, and the value iteration $V_{k+1}=\min\{l,\sup_uV_k\circ f\}$ with $l(x)=1-|x|$ computes the same sets (here $U$ is compact, so the supremum is attained).

Model predictive control

$x_{k|t}$ (some modules write $x_{n|t}$) is the state predicted $k$ steps ahead at time $t$: a planned state, which differs from the realised $x_{t+k}$ under model error or disturbances. MPC is an implicit feedback policy $x\mapsto u^\ast_{0}(x)$, evaluated by optimisation; its input–output pairs can be recorded and imitated by a neural network, which then needs its own certificate (Modules 11 and 14). Stochastic MPC treats random disturbances through chance constraints; robust and tube MPC protect against every disturbance in a bounded set.

Tiny example. $x_{t+1}=x_t+u_t$, $|u_t|\le1$, $N=2$, $X_f=\{0\}$ ($\kappa_f=0$, $V_f=0$): the problem is feasible exactly for $|x_0|\le2$. From $x_0=2$ the plan is $(-1,-1)$; at $t=1$ ($x=1$) the shifted candidate $(-1,0)$ is feasible. Without a terminal set, a short-sighted plan can steer into states from which no admissible input prevents a later violation: feasible now does not imply feasible next step.

Safety filters and shields. A safety filter takes a proposed input $u_{\rm prop}$ (from RL, a human or a nominal controller), applies it if it is certified safe and otherwise replaces it, ideally by $\arg\min_{u\in U_{\rm safe}(x)}\|u-u_{\rm prop}\|$. Instances: the CBF-QP ($U_{\rm safe}=K_{\rm cbf}$, Section 4); the predictive safety filter, whose $U_{\rm safe}(x)$ are the first inputs of $N$-step plans that end in a terminal safe set; a backup policy plus a trigger that switches to it (Module 6); and shields in discrete settings, which block actions whose successors leave the winning region of a safety game. A filter that must match an unknown future desired input $u_{\rm des}(\cdot)$ over an infinite horizon, $\arg\min_{v(\cdot)}\int\|v-u_{\rm des}\|^2$, cannot be solved online, which is why filters act one step (or $N$ steps) at a time.

Tube MPC. True system $x_{t+1}=Ax_t+Bu_t+w_t$, $w_t\in W$ (compact, containing $0$); nominal plan $z_{t+1}=Az_t+Bv_t$; ancillary feedback $u_t=v_t+K(x_t-z_t)$. The deviation $e_t=x_t-z_t$ obeys $e_{t+1}=(A+BK)e_t+w_t$, independently of the plan, so with $e_0=0$ it stays in $E=W\oplus(A+BK)W\oplus(A+BK)^2W\oplus\cdots$ (a robust positively invariant set when $A+BK$ is Schur; $0\in W$ is needed here: for $A+BK=\tfrac12$ and $W=\{1\}$ the set is $E=\{2\}$, but $e_1=1\notin E$). Planning the nominal trajectory under tightened constraints $z_t\in X\ominus E$, $v_t\in U\ominus KE$ (and a terminal set inside them) keeps the true $x_t=z_t+e_t\in X$ and $u_t\in U$ for every disturbance sequence. Numbers: $x_{t+1}=x_t+u_t+w_t$, $|w_t|\le0.1$, $K=-0.5$, so $A+BK=0.5$ and $E=[-0.2,0.2]$ ($0.1(1+0.5+0.25+\cdots)$). With $X=U=[-1,1]$: $z_t\in[-0.8,0.8]$ and, since $KE=[-0.1,0.1]$, $v_t\in[-0.9,0.9]$. Probabilistic regions (conformal prediction, Module 15) enter the planner in exactly the same way, as tightened constraints. When the disturbance bound depends on the state, $W$ becomes a set-valued map $x\mapsto W(x)$ (a map that returns a set); its continuity is measured with a distance between sets such as the Hausdorff distance $d_H(S,S')=\max\{\sup_{s\in S}\mathrm{dist}(s,S'),\ \sup_{s'\in S'}\mathrm{dist}(s',S)\}$ (Module 10).

Practice — Optimise a cost-to-go and preserve feasibility

Easy checks the basic operation; Medium connects steps; Hard asks you to justify a conclusion or identify a limitation. Each solution explains why the steps work.

Interactive: Phase Portraits & Lyapunov Ellipses

The plant is the linearised inverted pendulum $\dot x_1=x_2$, $\dot x_2=x_1+u$ (open-loop eigenvalues $\pm1$, a saddle). The sliders choose the closed-loop eigenvalues; the explorer computes the pole-placement gain of $u=-Kx$ (Section 5), the closed-loop matrix $A_{\rm cl}=A-BK$, solves $A_{\rm cl}^{\top}P+PA_{\rm cl}=-I$ in closed form (Section 3) and integrates trajectories with a fourth-order Runge–Kutta method. In the saturated mode the same gain acts through an actuator limit, $u=\mathrm{sat}(-Kx)$ with $|u|\le u_{\max}$ (Sections 1 and 7); the readouts about $V$, $\|x\|$ and sampling then describe the linear design and hold only where the input is not clipped.

What to look for.

1. The default (stable node, $\lambda=-1,-2$) gives $K=[3,3]$ and exactly the matrix of Section 3; the readout reproduces $P=\begin{bmatrix}1.25 & 0.25\\ 0.25 & 0.25\end{bmatrix}$. Trajectories cross every ellipse inwards, yet $\lambda_{\max}(A_{\rm cl}+A_{\rm cl}^{\top})=0.16\gt0$: the Euclidean norm can grow for a moment.
2. "Stable focus" spirals in; the ellipses tilt but still certify.
3. "Saddle": no positive definite $P$ exists, and only starts on the stable eigenvector line converge. "Center": closed orbits; the Lyapunov equation is singular because $\lambda_1+\lambda_2=0$. The readout also shows the eigenvalues $e^{\lambda h}$ of the sampled system: inside the unit circle exactly when $\mathrm{Re}\,\lambda\lt0$.
4. Saturated mode: inside the green ellipse $\{x^{\top}Px\le c^\ast\}$ with $c^\ast=u_{\max}^2/(KP^{-1}K^{\top})$ the input is never clipped (Cauchy–Schwarz), so the loop is linear, $V$ decreases, and the ellipse is invariant: every dot inside it converges. A dot is teal when its trajectory enters the ellipse (convergence proved) and red when it leaves radius $8$ (divergence proved, see the box below). Passing close to the origin proves nothing: with the "Saddle" preset (no certificate) some starts pass within radius $0.02$ of the origin and leave again, so no dot is teal. The clipped loop has two extra equilibria $(\pm u_{\max},0)$, saddles whose stable manifolds bound the true region of attraction. By default only $3$ of the $169$ grid starts lie in the certified ellipse while $97$ converge: the certificate is safe but conservative (Module 14's local sector constraints enlarge it).
5. Faster poles ($\sigma=-3$, $\delta=-2$) raise $K$ to $[6,6]$ and shrink the unclipped region, so $c^\ast$ drops from $0.111$ to $0.011$ while the set of converging grid starts does not change: an aggressive linear design does not buy a larger certified region. Lowering $u_{\max}$ to $1$ shrinks both ($59$ converging starts).

Exercises

These cumulative problems connect several ideas. If a step feels too large, return to the section practice route, where each topic has an Easy, Medium and Hard problem with a separate hint, worked solution and prerequisite review link.

Further Reading

Flashcards