Safety from rollouts, performance from gradients

1Setting

We consider a control-affine, continuous-time system with bounded inputs,

where $\mathcal U$ is a compact set encoding actuator limits and $f,g$ are locally Lipschitz. For a state-feedback policy $\pi:\mathcal X\to\mathcal U$, the closed-loop vector field is $f_\pi(x):=f(x)+g(x)\pi(x)$, and $\Phi_\pi(x_0,t)$ denotes its flow: the state at time $t$ starting from $x_0$.

Safety is specified by a safe set $\mathcal S:=\{x\in\mathcal X: h_{\mathcal S}(x)\ge 0\}$ with $h_{\mathcal S}$ continuously differentiable. The complement $\mathcal F:=\mathcal X\setminus\mathcal S$ is the failure set.

Reinforcement learning

The task is a Markov decision process $(\mathcal X,\mathcal U,F,r,\gamma)$ with deterministic transitions $x_{k+1}=F(x_k,u_k)$, reward $r$, and discount $\gamma\in(0,1)$. A neural policy $\pi_\phi$ maximizes

with $x_0\sim\rho_0$ and $x_{k+1}=F(x_k,\pi_\phi(x_k))$.

2Control invariance and control barrier functions

Not every state in $\mathcal S$ is safe in the dynamical sense: under bounded inputs, some states inside $\mathcal S$ inevitably leave it. Safe control therefore looks for an invariant subset of $\mathcal S$.

This makes the admissible set $\mathbb K_{\mathcal C}(x):=\{u\in\mathcal U:\nabla h_{\mathcal C}(x)^\top(f(x)+g(x)u)+\alpha(h_{\mathcal C}(x))\ge0\}$ nonempty. A CBF safety filter minimally modifies a nominal input,

a convex QP whenever $\mathcal U$ is convex, because the constraint is affine in $u$. If a locally Lipschitz policy satisfies $\pi(x)\in\mathbb K_{\mathcal C}(x)$ on $\mathcal C$, then $\mathcal C$ is forward invariant (Proposition C.1). The hard part is not using a CBF but finding one whose superlevel set is control-invariant under bounded inputs. Hamilton–Jacobi reachability computes maximal invariant sets but scales exponentially with $n$; sum-of-squares programs scale poorly with dimension and degree and are often conservative.

3Backup control barrier functions

Backup CBFs (Chen et al., 2021; Gurriet et al., 2020) avoid synthesizing a global invariant set. They start from a small, known control-invariant base set

with a certified base controller $\pi_{\mathcal B}$, and a deterministic backup policy $\pi_b$ that agrees with $\pi_{\mathcal B}$ on $\mathcal B$.

The set is the zero-superlevel set of an implicit CBF, evaluated by simulation rather than stored:

From the implicit barrier to affine constraints

The $\min$ in (3) is nonsmooth, so BCBF enforces sufficient CBF inequalities pointwise along the backup rollout and at the horizon. Let the flow sensitivity be

For a candidate current input $u$, the rollout and terminal derivative terms are

The subtraction in $\mathcal D^{\mathcal S}_{t'}$ appears because the rollout constraints are indexed by future time: as real time advances, the same future point on the backup rollout moves closer at rate $f_{\pi_b}$. The terminal condition uses the fixed horizon $T$, so it has no such term. Given extended class-$\mathcal K_\infty$ functions $\alpha_{\mathcal S},\alpha_{\mathcal B}$, the BCBF-admissible input set is

For fixed $x$ and $\pi_b$, the rollout and its sensitivity are constants with respect to $u$, so every constraint is affine in $u$, and $\mathbb K_{\mathrm{BCBF}}(x;\pi_b)$ is convex whenever $\mathcal U$ is.

BCBFs also sidestep high relative degree. The composed functions $h_{\mathcal S}\circ\Phi_{\pi_b}(\cdot,t')$ depend on the current input through $\Psi_{\pi_b}(x,t')g(x)$, and under mild controllability conditions they have relative degree one for positive rollout times. Safety can therefore be written directly in physical coordinates, such as altitude or lane position, without high-order CBF constructions.

The remaining difficulty. BCBF moves the burden from synthesizing an invariant set to designing a backup policy, and the size of $\mathcal C_T(\pi_b)$ depends critically on that policy. Hand-designed backups such as LQR are easy to certify but recover only a small part of the safe region. PS2-RL learns the backup instead.

4Problem formulation and decomposition

Let $\Pi_{\text{safe}}$ be the feasible policy class of Problem 1: the constraints hold pointwise in $k$, for every trajectory and every initial state in $\mathcal X_0$. A policy is provably safe if it provably belongs to $\Pi_{\text{safe}}$. Constrained policy optimization methods (CMDPs, state-wise safe RL) may use similar formulations, but they do not certify membership in $\Pi_{\text{safe}}$.

Subproblem 1: optimize under a fixed backup policy

Subproblem 2: enlarge the certified set

The conservatism of Subproblem 1 is governed by $\mathcal C_T(\pi_b)$. PS2-RL anchors the backup policy to a certified base pair and learns only the part outside it:

Here $\pi_{\mathcal B}:\mathcal B\to\mathcal U$ renders $\mathcal B$ forward invariant, and the safe-arrival policy $\pi_{\mathrm{SA}}:\mathcal X\setminus\mathcal B\to\mathcal U$ steers states into $\mathcal B$ within $T$ while staying in $\mathcal S$. Only $(\mathcal B,\pi_{\mathcal B})$ must be certified; $\pi_{\mathrm{SA}}$ need not be.

So any stabilizing linear feedback (pole placement, LQR, …) followed by a sublevel-set check for input feasibility and $\mathcal S$-containment produces a valid base pair, with no global search. With $(\mathcal B,\pi_{\mathcal B})$ fixed, $\mathcal C_T$ grows with $T$, but so does the number of constraints. A better safe-arrival policy enlarges it at a fixed horizon instead:

The decomposition is certified

For a fixed backup policy define the BCBF-feasible class $\Pi_{\mathrm{BCBF}}(\pi_b):=\{\pi:\mathcal X\to\mathcal U:\ \pi(x)\in\mathbb K_{\mathrm{BCBF}}(x;\pi_b)\ \forall x\in\mathcal C_T(\pi_b)\}$.

5The safe-arrival value function

Subproblem 2 needs an RL objective whose optimum enlarges $\mathcal C_T$. PS2-RL uses an indicator-style value that represents the reach-avoid event directly, rather than a dense geometric surrogate. Consider the deterministic system $x_{k+1}=F(x_k,u_k)$ with base set $\mathcal B$, failure set $\mathcal F=\mathcal X\setminus\mathcal S$, and continuation set $\mathcal H:=\mathcal X\setminus(\mathcal B\cup\mathcal F)$, assuming $\mathcal B\cap\mathcal F=\varnothing$. The indicators are

For a pair $(x,u)$ and a policy $\pi$, the rollout is $x_0=x$, $u_0=u$, $x_{k+1}=F(x_k,u_k)$ and $u_k=\pi(x_k)$ for $k\ge1$. The hitting times and the safe-arrival event are

Probability and expectation notation is kept even though the deterministic case gives laws concentrated on a single trajectory: the construction extends unchanged to stochastic dynamics and policies.

Undiscounted value: feasibility

The optimal values satisfy $Q^\star_{\mathrm{SA}}(x,u)=\mathbf b(x)+\mathbf c(x)\max_{u^+}Q^\star_{\mathrm{SA}}(F(x,u),u^+)$, and they define the region where safe arrival is possible at all.

Discounted value: what PS2-RL trains

What optimal safe-arrival actions do

Partition the state space as $\mathcal X=\mathcal B\,\dot\cup\,\mathcal F\,\dot\cup\,(\mathcal R_{\mathrm{SA}}\cap\mathcal H)\,\dot\cup\,(\mathcal H\setminus\mathcal R_{\mathrm{SA}})$ (Proposition D.8):

• In $\mathcal B$ both values equal 1, and in $\mathcal F$ both equal 0; every action is equivalent.
• In $\mathcal R_{\mathrm{SA}}\cap\mathcal H$, an action is optimal for the undiscounted value iff $F(x,u)\in\mathcal R_{\mathrm{SA}}$: it preserves feasibility but does not rank arrival times. It is optimal for the discounted value iff additionally $d(F(x,u))=d(x)-1$, so a discounted-optimal policy reaches $\mathcal B$ in exactly $d(x)$ steps.
• In $\mathcal H\setminus\mathcal R_{\mathrm{SA}}$ no policy can safely arrive, and both values are 0.

The undiscounted value supplies the feasibility semantics ($\mathcal R_{\mathrm{SA}}$). The discounted value preserves that region and adds a contraction-based, time-sensitive training signal (Remark D.1).

6Phase I: training the safe-arrival policy

With a design distribution $\rho_{\mathrm{arr}}$ supported on $\Omega\setminus\mathcal B$, Phase I solves

The only ingredients specific to safe arrival are the indicators, first-hit termination, and objective (12); everything else comes from the RL backbone (actor–critic, Q-learning, fitted value iteration, …). In the experiments, a TD3-style off-policy backbone with twin critics trained by the recursion (11) under a Huber loss is used, with $\beta=0.92$ (unicycle) and $\beta=0.99$ (quadrotor). A mild curriculum that expands $\rho_{\mathrm{arr}}$ outward from $\mathcal B$ is an optimization heuristic, not a requirement.

7Phase II: the control-invariant layer

PS2-RL builds on HardNet-CVX (Min & Azizan), which projects a network's output onto a state-dependent convex set through a differentiable optimization layer.

With $\pi^\star_{\mathrm{SA}}$ from Phase I, freeze $\pi_b^\star(x)=\pi_{\mathcal B}(x)$ on $\mathcal B$ and $\pi_b^\star(x)=\pi^\star_{\mathrm{SA}}(x)$ otherwise. Then $\mathcal C_T(\pi_b^\star)$ and $\mathbb K_{\mathrm{BCBF}}(x;\pi_b^\star)$ are fixed, and the latter is convex for every $x$ when $\mathcal U$ is.

This reparameterization turns Subproblem 1 into unconstrained optimization over $\phi$, $\max_\phi J(\pi^\phi_{\mathrm{PS2}}(\cdot;\pi_b^\star))$, with gradients flowing through (13).

The QP, explicitly

Allow multiple constraints, $\mathcal S=\{x:h_{\mathcal S,j}(x)\ge0,\ j=1..n_{\mathcal S}\}$ and $\mathcal B=\{x:h_{\mathcal B,\ell}(x)\ge0,\ \ell=1..n_{\mathcal B}\}$, and a mesh $0=t'_0<\dots<t'_N=T$. For the current $x$ and the frozen $\pi_b^\star$, define

Each safety component and node gives an affine row $a^{\mathcal S\top}_{i,j}u\le b^{\mathcal S}_{i,j}$, and each terminal base-set component gives $a^{\mathcal B\top}_{\ell}u\le b^{\mathcal B}_{\ell}$, with

Stacking gives $A_{\mathrm{BCBF}}(x)u\le b_{\mathrm{BCBF}}(x)$ with $n_c=(N+1)n_{\mathcal S}+n_{\mathcal B}$ rows. With box limits $u_{\min}\le u\le u_{\max}$ written as $A_{\mathcal U}u\le b_{\mathcal U}$, the layer is the QP

Numerical safeguard: a heavily penalized slack

The implementation solves a slack-regularized version with a single scalar slack shared by the BCBF rows only. The input limits are never relaxed:

with $\lambda_\delta=10^5$ (unicycle) and $10^6$ (quadrotor); the layer returns $u^\star$. The slack absorbs small artificial infeasibilities from the finite mesh, numerical integration and sampled-data implementation near $\partial\mathcal C_T$. The formal guarantees correspond to $\delta^\star=0$. In evaluation, mean and median slack are on the order of $10^{-5}$ or smaller, the 99th percentile is at most $5.1\times10^{-5}$, and $\delta>10^{-4}$ never occurs on the unicycle and occurs on at most 0.088% of quadrotor steps (Table 7).

Training through the layer

All quantities that build $A_{\mathrm{BCBF}},b_{\mathrm{BCBF}}$ (backup rollout, sensitivity propagation, row construction) and the QP solve itself (qpax, a differentiable primal–dual interior-point solver) are differentiable JAX operations, so the actor gradient contains the QP solution:

For a stochastic actor, the nominal action is sampled, $u^{\text{nom}}_k\sim\pi_\phi(\cdot\mid x_k)$, and then projected deterministically, $u_k=\mathcal P_{\mathrm{CIL}}(u^{\text{nom}}_k)(x_k)$. The environment and replay buffer only ever see projected controls, the actor loss evaluates the critic at the projected action, and target-policy actions in Bellman backups are projected too. This is the difference from a post-hoc shield: the projection is present during both data collection and gradient-based improvement.

Together, Theorems 4.3 and 4.4 say the layer enforces safety without sacrificing expressivity: every projected policy is safe, and projection preserves the approximation power of the chosen architecture over the BCBF-feasible class.

8Scalability and extensions

PS2-RL reduces provably safe RL to rollouts and a control-space projection. The only object that must be certified is the local base pair $(\mathcal B,\pi_{\mathcal B})$, which Theorem 4.1 makes routine. The learned safe-arrival policy is certified implicitly: once frozen, rollouts extend the base set to $\mathcal C_T(\pi_b^\star)$ by construction.

• Small QPs in high dimension. The layer has $m$ decision variables and $O(Nn_{\mathcal S}+n_{\mathcal B}+2m)$ affine rows, which grows linearly with the backup mesh and the number of constraints. Discretizing the Hamilton–Jacobi PDE instead costs exponentially in $n$.
• Architecture-agnostic. The layer is appended after any policy network, and Theorem 4.4 preserves universal approximation. Unlike post-hoc network verification, no specific network family is required.
• Differentiable simulators. The control-affine form is not essential. With a differentiable simulator $F_{\Delta t}$ and the candidate-input rollout $z_0(u)=F_{\Delta t}(x,u)$, $z_{i+1}(u)=F_{\Delta t}(z_i(u),\pi_b^\star(z_i(u)))$, each rollout condition $\ell_i(z_i(u))\ge0$ yields a local affine row $\ell_i(z_i(u_b))+a_i(x)^\top(u-u_b)\ge0$ with $a_i(x)=\nabla_u\ell_i(z_i(u))\big|_{u=u_b}$ and $u_b=\pi_b^\star(x)$, computable by automatic differentiation for only the safety-relevant outputs. These rows are first-order approximations, so conservative margins or a trust region around $u_b$ are advisable. This variant is future work.

9Experimental instantiation

Unicycle lane keeping

$\Delta t=0.05$ s (20 Hz). The safe set $\mathcal S=\{|y|\le1.8,\ |\psi|\le\pi/3\}$ gives four rows: $h_{\mathcal S,1}=y_{\max}+y$, $h_{\mathcal S,2}=y_{\max}-y$, $h_{\mathcal S,3}=\psi_{\max}+\psi$, $h_{\mathcal S,4}=\psi_{\max}-\psi$. The reference $y_{\text{ref}}(t)=2.5\sin(2\pi t/10)$, $v_{\text{ref}}=5$ is intentionally unsafe. The base pair is a discrete-time LQR about $x^\star=[0,5,0]^\top$ with $Q_d=\operatorname{diag}(1,1,1)$, $R_d=\operatorname{diag}(0.01,0.5)$ and $\mathcal B=\{e^\top Pe\le c_{\mathcal B}\}$, where $c_{\mathcal B}=0.3$ lies below the actuator-admissible level $\bar c\approx1.05$. With $T=1$ s and $N=20$, the layer has $4(N+1)+1=85$ BCBF rows, or 90 inequalities with the input box and slack.

Quadrotor powerloop

with thrust $a_{\text{cmd}}\in[0,4g]$, body rates $\boldsymbol\omega_{\text{cmd}}\in[-18,18]^3$ rad/s, and $\Delta t=0.02$ s (50 Hz). The safe set is a hard ceiling, $h_{\mathcal S}(x)=z_{\text{ceil}}-p_z\ge0$ with $z_{\text{ceil}}=3$ m. It has relative degree 2 for thrust and higher for body rates. The base pair is a hover LQR on a reduced 7-D error state $x_e=[p_z-z_{\text{des}},v_x,v_y,v_z,2q_{\text{err},x},2q_{\text{err},y},2q_{\text{err},z}]^\top$ with $u^\star=[g,0,0,0]^\top$ and $c_{\mathcal B}=8.0$ (admissible level $\bar c\approx12.38$), placing the ellipsoid strictly below the ceiling. With $T=2$ s and $N=100$, the layer has $1\cdot(N+1)+1=102$ BCBF rows, or 111 inequalities.

The quadrotor warm start follows the HardNet-CVX recipe of first training the nominal network without the projection. Because the layer projects every action during training and deployment, the safety guarantee holds regardless of how $\pi_\phi$ is initialized. Full hyperparameters are in Tables 8–9 of the paper.

References: Y. Chen, M. Jankovic, M. Santillo, A. D. Ames, "Backup control barrier functions: Formulation and comparative study," CDC 2021. T. Gurriet et al., "A scalable safety critical control framework for nonlinear systems," IEEE Access 2020. Y. Min, N. Azizan, "HardNet: Hard-constrained neural networks with universal approximation guarantees," 2025. K. Tracy, Z. Manchester, "On the differentiability of the primal-dual interior-point method," 2024.