NeurIPS 2026 · Spotlight

PS2-RLProvably Safe, Yet Scalable Reinforcement Learning

Kai S. Yun Zeyang Li† Navid Azizan

Massachusetts Institute of Technology  ·  †Corresponding author

arXiv 2606.14536 PDF Code The math BibTeX
PS2-RL · 10-D quadrotor 0.5× speed
The powerloop reference (dashed) climbs above the 3 m ceiling (red plane). The PS2-RL policy stays close to it and never crosses the ceiling.
100%
safe evaluation episodes for PS2-RL on both tasks (10 seeds × 1,000 episodes each), with safety enforced during training too.
Tables 1–2
10-D
quadrotor state with bounded thrust and body rates, a regime where certificate-based provably safe RL struggles.
Section 5.2
−57%
position error vs. the same framework with an analytic backup policy (0.60 m vs 1.39 m RMSE).
Table 2
0.80ms
per control step on the quadrotor, including backup rollout and QP solve, against a 20 ms control period.
Table 13

Abstract

Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.

How it works

First learn where safety is possible, then learn to perform inside it

Figure 1 of the paper, assembled one piece at a time as you scroll. Phase I builds the safety certificate; Phase II trains the task policy through it.

Figure 1: overview of the PS2-RL framework. Left, Phase I: a certified base set and controller, and safe-arrival policy training with the safe-arrival value function, enlarge the backup-induced invariant set. Center: the composed backup policy. Right, Phase II: the RL policy is trained through the control-invariant layer.

  1. PS2-RL

    Provable safety at a scale RL can handle

    The goal: maximize task return while keeping $x_k\in\mathcal S$ and $u_k\in\mathcal U$ at every step, from every initial state. PS2-RL splits this into two subproblems, solved in two phases. Scroll to build the framework.

  2. Phase I · Objective

    Enlarge the region where safety can be certified

    The backup-induced invariant set $\mathcal C_T(\pi_b)$ holds every state from which a backup policy $\pi_b$ safely returns to a small base set $\mathcal B$ within horizon $T$. A hand-designed backup certifies a small set (left); a learned one certifies a much larger set (right).

  3. Phase I · Step 1

    Start from a small certified base set

    Around an equilibrium $x^\star$, the linear controller $\pi_{\mathcal B}=u^\star-Ke$ keeps the ellipsoid $\mathcal B=\{e^\top Pe\le c_{\mathcal B}\}$ invariant, inside $\mathcal S$ and within input limits. Theorem 4.1 shows such a pair always exists under local stabilizability. It is the only piece that must be certified.

  4. Phase I · Step 2

    Score a policy by safe arrival

    The safe-arrival value is built only from indicators: $\mathbf b$ for entering $\mathcal B$, $\mathbf f$ for entering the failure set $\mathcal F=\mathcal X\setminus\mathcal S$, and $\mathbf c=1-\mathbf b-\mathbf f$ while still in transit. It rewards the first safe arrival, discounted by $\beta$ per step.

  5. Phase I · Step 2

    Safe beats unsafe; fast beats slow

    A rollout that reaches $\mathcal B$ safely in 3 steps scores $\beta^3$. One that needs 5 steps scores $\beta^5<\beta^3$, and one that hits $\mathcal F$ first scores $0$. The maximizer is therefore the time-optimal safe-arrival policy.

  6. Phase I · Step 2

    Learn it with standard RL

    The value obeys $Q_{\mathrm{SA},\beta}(x,u)=\mathbf b(x)+\beta\,\mathbf c(x)\,Q_{\mathrm{SA},\beta}(x',\pi(x'))$, so any actor–critic or Q-learning backbone can train the safe-arrival policy $\pi^\theta_{\mathrm{SA}}$ to bring sampled states (green) into $\mathcal B$.

  7. Phase I → Phase II

    Compose and freeze the backup policy

    The backup policy $\pi_b^\theta$ runs the linear base controller inside $\mathcal B$ and the frozen learned policy outside it. The learned part needs no certificate: backup rollouts certify exactly the states from which it succeeds.

  8. Phase II · Objective

    Train the task policy inside the certified set

    Maximize return $J(\pi_\phi)$ subject to $\pi_\phi(x)\in\mathbb K_{\mathrm{BCBF}}$ for every $x\in\mathcal C_T(\pi_b^\theta)$. With the backup policy frozen, these BCBF constraints are affine in the action, so the admissible set is convex.

  9. Phase II · Control-invariant layer

    Project every action onto the admissible set

    The control-invariant layer (CIL), $\mathcal P_{\mathrm{CIL}}(\pi_\phi)(x)=\arg\min_u\|u-\pi_\phi(x)\|_2^2$ s.t. $u\in\mathbb K_{\mathrm{BCBF}}(x)$, is a small QP. It keeps the action only where the backup policy could still bring the system safely to $\mathcal B$.

  10. Phase II · Control-invariant layer

    Check for a safe flow at every step

    Along the trajectory, each state is checked for a safe backup flow into $\mathcal B$ (green). An action that would lead to a state with no safe flow (red) is projected back, so the system never leaves $\mathcal C_T$.

  11. Phase II · Training

    Backpropagate through the projection

    The QP is differentiable, so reward gradients flow through the CIL into $\pi_\phi$. The network learns high-return actions that already respect the constraints. Any RL backbone works; the experiments use SAC.

  12. PS2-RL

    Provably safe, yet scalable

    Safety comes from a small certified base set plus backup rollouts; performance comes from end-to-end RL through the layer. Every PS2 policy keeps $x_k\in\mathcal S$ and $u_k\in\mathcal U$ for all $k$ (Theorem 4.3) without losing expressivity (Theorem 4.4). Read the math →

Method

Three ideas, in plain words

PS2-RL never computes an invariant set. It certifies safety by simulating a backup policy, learns that backup policy so the certified region is large, and then trains the task policy through a projection layer that can only output certified actions.

0

The problem: hard constraints, every step, every start

We want the reward-maximizing policy among those that are provably safe under bounded inputs:

$$\max_{\pi}\; J(\pi)=\mathbb E\Big[\sum_{k=0}^{\infty}\gamma^k r(x_k,u_k)\Big]\quad\text{s.t.}\quad x_k\in\mathcal S,\;\; u_k\in\mathcal U\quad \forall k\ge 0,\;\forall x_0\in\mathcal X_0.$$

Penalty and Lagrangian (CMDP) methods only encourage this, so failures can still happen at deployment. Provable methods need a control-invariant set, a region from which safety can be maintained forever. Computing one explicitly is the bottleneck: Hamilton–Jacobi reachability grows exponentially with state dimension, and sum-of-squares certificates tend to be conservative.

1

Certify by simulation, not by synthesis

Start from a small set $\mathcal B$ around an equilibrium that a linear controller provably keeps invariant. Any state from which a backup policy $\pi_b$ reaches $\mathcal B$ within horizon $T$ without leaving $\mathcal S$ is safe forever, because the base controller holds it there afterwards. Those states form an implicit control-invariant set:

$$\mathcal C_T(\pi_b)=\big\{x:\ \Phi_{\pi_b}(x,T)\in\mathcal B\ \wedge\ \Phi_{\pi_b}(x,t')\in\mathcal S\ \ \forall t'\in[0,T]\big\}.$$

Membership is checked by rolling out $\pi_b$, so no set is ever stored. Along that rollout, the backup-CBF (BCBF) conditions are affine in the current action. The admissible actions therefore form a convex set $\mathbb K_{\mathrm{BCBF}}(x;\pi_b)$ that always contains $\pi_b(x)$. The catch: how large $\mathcal C_T$ is, and hence how freely the task policy may act, depends entirely on the backup policy. Hand-designed backups such as LQR recover only a small part of the truly safe region.

2

Phase I: learn the backup with the safe-arrival value

PS2-RL keeps the certified pair $(\mathcal B,\pi_{\mathcal B})$ and learns only what happens outside $\mathcal B$: a safe-arrival policy $\pi^\theta_{\mathrm{SA}}$ that steers states into $\mathcal B$ quickly while avoiding the failure set $\mathcal F=\mathcal X\setminus\mathcal S$. Its objective is an indicator-based value:

$$Q^{\pi_{\mathrm{SA}}}_{\mathrm{SA},\beta}(x,u)=\mathbb E\Big[\sum_{k=0}^{\infty}\beta^k\Big(\prod_{\tau=0}^{k-1}\mathbf c(x_\tau)\Big)\mathbf b(x_k)\Big],\qquad \mathbf b=\mathbf 1_{\mathcal B},\ \ \mathbf f=\mathbf 1_{\mathcal F},\ \ \mathbf c=1-\mathbf b-\mathbf f.$$

For deterministic dynamics it equals $\beta^N$ if the rollout first reaches $\mathcal B$ safely after $N$ steps, and $0$ otherwise. Maximizing it therefore prefers safe arrival over failure, and faster arrival over slower. Try it:

0.80
Fast, safe reaches $\mathcal B$ in 3 steps
Slow, safe reaches $\mathcal B$ in 5 steps
Unsafe hits $\mathcal F$ first

$Q_{\mathrm{SA},\beta}$ for the three rollouts of Figure 1. For any $\beta\in(0,1)$ the ranking is the same; $\beta$ sets how strongly speed is rewarded.

The value satisfies a Bellman equation, $Q_{\mathrm{SA},\beta}(x,u)=\mathbf b(x)+\beta\,\mathbf c(x)\,Q_{\mathrm{SA},\beta}\big(x',\pi_{\mathrm{SA}}(x')\big)$, which is a $\beta$-contraction, so any off-the-shelf RL algorithm can learn it. Its optimum is exactly the time-optimal safe-arrival policy, $V^\star_{\mathrm{SA},\beta}(x)=\beta^{d(x)}$. The learned policy is never certified, and it does not need to be: the rollout check of idea 1 only certifies the states from which it actually succeeds.

3

Phase II: train through a control-invariant layer

Freeze the composed backup policy $\pi_b^\star$ (the base controller inside $\mathcal B$, the learned $\pi^\star_{\mathrm{SA}}$ outside). The task policy's raw action is then projected onto the BCBF-admissible set by a differentiable quadratic program:

$$\pi^{\phi}_{\mathrm{PS2}}(x)=\mathcal P_{\mathrm{CIL}}(\pi_\phi)(x)=\arg\min_{u}\ \|u-\pi_\phi(x)\|_2^2\quad\text{s.t.}\quad u\in\mathbb K_{\mathrm{BCBF}}(x;\pi_b^\star).$$

Every executed action, in training and in deployment, is certified. Because the QP is differentiable, reward gradients flow through the projection into the network, so the policy learns to perform well inside the safe set instead of being corrected after the fact. The QP stays small: $m$ decision variables and $O(Nn_{\mathcal S}+n_{\mathcal B}+2m)$ affine rows (102 BCBF rows for the quadrotor). It grows linearly with the backup mesh, not exponentially with the state dimension.

What is guaranteed

Theorem 4.1

A certified base set always exists

Under local stabilizability, any stabilizing linear gain with a small enough Lyapunov sublevel set gives a valid $(\mathcal B,\pi_{\mathcal B})$: inside $\mathcal S$, within input limits, and invariant.

Statement and proof → Theorem 4.2

The decomposition is certified

Phase I then Phase II yields a safe policy that is optimal among all policies certified by the learned backup, and globally optimal whenever that class contains an optimal safe policy.

Statement and proof → Theorem 4.3

Safe for any network weights

If $\mathcal X_0\subseteq\mathcal C_T(\pi_b^\star)$, every PS2 policy keeps $x_k\in\mathcal S$ and $u_k\in\mathcal U$ for all $k$, before, during and after training.

Statement and proof → Theorem 4.4

No loss of expressivity

Composing a universal approximator with the CIL universally approximates every continuous BCBF-feasible policy.

Statement and proof →

The formal guarantees assume a control-affine model of the dynamics (or a differentiable simulator; see the math page) and hold for the sampled-data system. With exact BCBF constraints, the implicit set never has to be computed or verified.

Results

100% safe, with the best tracking among safe methods

Both tasks track references that intentionally leave the safe set. PS2-RL with the learned safe-arrival policy (PS2SA) is compared with seven baselines: SAC with a low or high violation penalty, constrained-MDP methods (CPO, SAC-Lagrangian), and provably safe RL (CBF-RL, MPS, and PS2-RL with an analytic backup policy, PS2ABP). Every method is trained with 10 seeds and evaluated on 1,000 episodes per seed.

Task 1

Unicycle lane keeping

The state is $x=[y,v,\psi]^\top$ (lateral offset, speed, heading), with inputs $a\in[-5,5]\,\mathrm{m/s^2}$ and $r\in[-1,1]\,\mathrm{rad/s}$. The safe set is $|y|\le 1.8$ m and $|\psi|\le\pi/3$, but the 20-second sinusoidal reference swings to $\pm2.5$ m, outside the lane on purpose. On a dense $201\times121\times201$ grid, the learned safe-arrival policy enlarges the safe-arrival measure from 0.23 (analytic) to 0.33. It covers 99.08% of the analytic set and is 1.43× larger at the task speed.

Learned versus analytic backup-induced invariant sets for the unicycle, with slices at fixed speed and tracking results
Figure 2. Left: the invariant set induced by the learned safe-arrival policy (blue) contains and extends the one induced by the analytic policy (tan) around the base set (red). Middle: slices at fixed speed $v$. Right: PS2SA tracks the unsafe reference more closely than PS2ABP, while the implicit barrier $h_{\mathcal C_T}(x)$ stays nonnegative.
HJ reachability invariant set versus learned backup-policy invariant set at v = 5 m/s
How close to optimal is the learned backup? At $v=5$ m/s, the invariant set certified by a learned safe-arrival policy (horizon $T=3$ s, dashed boundary) almost coincides with the maximal control-invariant set from Hamilton–Jacobi reachability (blue boundary): IoU 0.97, covering 97.7% of the HJ set. Additional analysis, not in the arXiv version.
Lateral trajectories y(t) for all eight methods on the unicycle task
Figure 6. Representative lateral trajectories. Dotted: the unsafe reference. Dashed red: lane boundary. PS2SA rides near the boundary without crossing it; PS2ABP is safe but conservative; MPS chatters along the boundary; penalty and CMDP baselines cross it.
Table 1. Unicycle lane keeping, 10 seeds × 1,000 episodes. RMSE: interquartile mean [95% CI] (lower is better). Bold: best among methods that are 100% safe.
Tracking (RMSE)Safety
FamilyMethod $y$ (m) ↓$v$ (m/s) ↓$\psi$ (rad) ↓ Total safety ↑safe ep / total epPer-seed safety ↑IQM [95% CI]Worst viol. ↓$y$ (m)
PenSAC-Penlow0.53[0.34, 0.96]2.14[1.20, 3.11]0.19[0.09, 0.38]1.8% ± 5.4%0.0% [0.0%, 3.0%]8.12
SAC-Penhigh1.11[0.99, 1.25]1.85[1.26, 2.84]0.15[0.13, 0.19]100% ± 0.0%100% [100%, 100%]0.00
ConSAC-Lag.0.52[0.40, 0.66]1.22[0.99, 1.54]0.12[0.09, 0.14]30.0% ± 45.8%16.7% [0.0%, 66.7%]1.03
CPO2.51[1.66, 7.29]4.44[3.05, 5.36]0.27[0.20, 0.57]64.2% ± 40.2%73.7% [32.5%, 99.6%]76.94
PSRLCBF-RL1.78[1.75, 1.80]4.72[3.69, 4.91]0.27[0.24, 0.30]99.8% ± 0.003%100% [99.6%, 100%]0.00
MPS0.68[0.55, 0.78]0.48[0.43, 0.53]0.17[0.14, 0.19]97.8% ± 0.005%97.8% [97.5%, 98.2%]0.67
PS2ABP0.97[0.97, 0.98]0.43[0.35, 0.55]0.15[0.14, 0.15]100% ± 0.0%100% [100%, 100%]0.00
PS2SA (ours)0.52[0.51, 0.54]0.83[0.76, 0.88]0.11[0.10, 0.11]100% ± 0.0%100% [100%, 100%]0.00

The task reward weights lateral error roughly 19× more than speed error, so PS2SA trades speed accuracy for much lower lateral and heading error. Compared with PS2ABP, the learned backup cuts lateral RMSE from 0.97 m to 0.52 m.

Task 2

Quadrotor powerloop: 10-D, agile, input-limited

The state is $x=[\mathbf p^\top,\mathbf v^\top,\mathbf q^\top]^\top\in\mathbb R^{10}$ (position, velocity, unit quaternion). The inputs are thrust $a\in[0,4g]$ and body rates $\omega\in[-18,18]^3$ rad/s at 50 Hz. The reference is a vertical loop of radius 1.5 m entered at 4.5 m/s with a full 360° flip, and its apex exceeds the 3 m ceiling. Safety is a single altitude bound, but it must be enforced through the full 10-D dynamics and all four inputs.

Compare PS2SA with
PS2SA (ours)
pos. RMSE 0.60 msafe episodes 100%worst violation 0.00 m
PS2ABP

One representative evaluation rollout per method, played at 0.5× speed. Dashed: reference. Red plane: ceiling at $z=3$ m. Metrics are interquartile means over 10 seeds × 1,000 episodes (Table 2).

Table 2. Quadrotor powerloop, 10 seeds × 1,000 episodes. RMSE: interquartile mean [95% CI] (lower is better). Bold: best among methods that are 100% safe.
Tracking (RMSE)Safety
FamilyMethod $\mathbf p$ (m) ↓$\mathbf v$ (m/s) ↓$\theta_{\mathbf q}$ (rad) ↓ Total safety ↑safe ep / total epPer-seed safety ↑IQM [95% CI]Worst viol. ↓$z$ (m)
PenSAC-Penlow1.00[0.72, 1.36]1.88[1.51, 2.36]1.15[0.74, 1.59]47.3% ± 44.6%45.5% [7.2%, 89.7%]0.62
SAC-Penhigh1.14[1.02, 1.48]2.24[1.84, 2.63]1.54[1.30, 1.79]99.9% ± 6e-4%100% [99.9%, 100%]0.004
ConSAC-Lag.1.94[1.70, 2.53]4.30[3.97, 4.58]2.12[1.77, 2.26]58.6% ± 43.9%64.3% [19.0%, 100%]1.79
CPO10.62[7.08, 13.17]11.55[9.28, 13.94]2.17[2.04, 2.36]70.0% ± 45.8%83.3% [33.3%, 100%]2.41
PSRLCBF-RL2.29[1.71, 2.97]4.36[4.00, 4.64]2.06[1.80, 2.29]99.8% ± 0.005%100% [99.7%, 100%]0.21
MPS1.99[1.95, 2.05]4.07[3.89, 4.21]1.38[1.31, 1.51]100% ± 0.0%100% [100%, 100%]0.00
PS2ABP1.39[1.30, 1.53]1.74[1.62, 1.92]0.75[0.71, 0.90]100% ± 0.0%100% [100%, 100%]0.00
PS2SA (ours)0.60[0.53, 0.70]0.96[0.75, 1.18]0.45[0.33, 0.53]100% ± 0.0%100% [100%, 100%]0.00

Relative to PS2ABP, the learned backup reduces every RMSE by about 40–57%. Relative to MPS, which is given a longer recovery horizon, position RMSE drops by about 70%.

Inside Phase I

A learned backup recovers from states the analytic one cannot

Phase I is what makes Phase II less conservative. From task-relevant states around the powerloop, including states just below the ceiling, the learned safe-arrival policy safely reaches the base set from 85.9% of states, versus 60.9% for the analytic backup policy. The biggest gain is near the ceiling, exactly where the powerloop needs room.

Learned $\pi^\theta_{\mathrm{SA}}$0.5×
Learned safe-arrival policy. Trajectories bend away from the ceiling and into the base set; gold segments are flown by the base controller $\pi_{\mathcal B}$ after entering $\mathcal B$.
Analytic $\pi_{\mathrm{SA}}$0.5×
Analytic safe-arrival policy. From aggressive near-ceiling and "bridge" states below the loop apex, many rollouts break through the ceiling before reaching $\mathcal B$.

Recoverability by region of the task-relevant design set

Share of sampled states from which the backup policy safely reaches $\mathcal B$ within $T=2$ s (6,161 states).

Analytic $\pi_{\mathrm{SA}}$Learned $\pi^\theta_{\mathrm{SA}}$
Recoverability (%) by region
RegionAnalyticLearned
Near ceiling35.569.3
Bridge47.378.6
General trace77.796.1
Capture shell84.4100.0
Overall60.985.9

Beyond the paper

Three constraints at once: a ceiling and two walls

The same pipeline handles several simultaneous constraints. Here the powerloop must also fit between two walls, $-1\le x\le 1$ m, under the $z\le3$ m ceiling. The control-invariant layer simply stacks one row per constraint per backup node and is solved as a hard QP with no slack.

Across 5 training seeds × 1,000 evaluation episodes, all 5,000 episodes were safe, with zero ceiling or wall violations. The closest any episode came to a constraint was 4.4 cm.

ceiling $z\le 3$left wall $x\ge -1$right wall $x\le 1$
Front view0.5×
Best-return evaluation episode (seed 3). Minimum margins: 0.18 m to the ceiling, 0.05 m to the left wall, 0.17 m to the right wall.

Ablations & compute

The layer is needed at deployment and during training

On the quadrotor, removing the control-invariant layer after training drops safety from 100% to 0%: the network has not simply memorized the constraint. Adding the layer only as a post-hoc filter to an unconstrained policy is safe but tracks 43–53% worse. Training through the layer is what recovers performance under the guarantee.

Table 12. Paired ablation on the best-tracking quadrotor run, 1,000 episodes each.
Policy$\mathbf p$ (m)$\mathbf v$ (m/s)$\theta_{\mathbf q}$ (rad)SafeMean max viol. (m)
Vanilla + CIL (post-hoc)0.9281.1340.431100%0.00
PS2SA w/o CIL0.7403.5521.0290.0%0.198
PS2SA0.5280.6270.204100%0.00
Compute on one NVIDIA V100 (Table 13 and App. G.4; Phase II averaged over 10 seeds).
PS2SAPhase I trainingPhase II trainingInference / step
Unicycle14 min4.6 h0.35 ms (of 50 ms)
Quadrotor40 min13.6 h0.80 ms (of 20 ms)
Altitude traces for Vanilla+CIL, PS2-SA without CIL, and PS2-SA
Figure 9. Altitude $p_z$ of the best-tracking episode for each policy. Without the layer, the trained policy overshoots the 3 m ceiling near the loop apex; with it, both policies stay below, but only the end-to-end trained one tracks closely.

Citation

BibTeX

@article{yun2026ps2rl,
  title   = {Provably Safe, Yet Scalable Reinforcement Learning},
  author  = {Yun, Kai S. and Li, Zeyang and Azizan, Navid},
  journal = {arXiv preprint arXiv:2606.14536},
  year    = {2026}
}