Skip to content

Bicycle5D — drive to a goal, avoid obstacles

A planar car with bicycle-model dynamics, circular obstacles, and a goal. Small, NumPy-only, and CPU-trainable to convergence in minutes — the library's reference validation env.

Use this environment to quickly check your installation, get started with the codebase, and observe the difference between two modes of safety training:

  • avoid (SafetySAC1P / SafetyPPO1P): learn to avoid collisions without a goal-reaching incentive (always safe).

  • reach-avoid (ReachAvoidSAC1P / ReachAvoidPPO1P): drive to the goal from anywhere on the map without hitting an obstacle (always remain safe and eventually reach the goal).

import safety_sb3.testing.bicycle5d — BicycleGoal (single-env gym) and BicycleGoalVec (batched, ~50k steps/s on CPU).

Results

Trained from full-map spawns (cars start all over the map) and evaluated on 8 spawns × 4 unseen layouts. Reach-avoid reaches from every region; avoid sits still.

Reach-avoid — cars reach from anywhere

reach-avoid rollout

avoid — cars stay put / away from the obstacles

avoid rollout

The learned value function V(x, y)

V ≥ 0 is the reach-avoid certificate (can reach the goal safely from here); the black contour is its boundary, obstacles are negative wells.

value compare PPO vs SAC

The off-policy SAC value is the clean, calibrated one — negative only around the obstacles, correct magnitude. The on-policy PPO value is muddy off its trajectory tube (it fits V only where the policy goes). This is why HJ-reachability work uses the SAC family for certificates: V(s) = minᵢ Qᵢ(s, π(s)) is calibrated over the action space, not just the on-policy distribution.

The contract

Observation — 4 + 2 + 3·n_obstacles (12 for the two default obstacles)

All in the car's body frame (translation-invariant, so the policy generalizes across the map):

slot content
v speed
sin ψ, cos ψ heading
δ steering angle
goal_x, goal_y goal position relative to the car
each of n obstacles relative position (x, y) and radius

Action — 2 (or 7 with adversary=True)

range
accel [−2, 2]
omega (steering rate) [−2, 2]
(adversary) d[0:5] additive disturbance on all 5 state derivatives

State [x, y, v, ψ, δ], v ∈ [0, 2] (can stop, cannot reverse — v_min = 0 is load-bearing: the avoid car must be able to sit still), δ ∈ [±0.35], minimum turning radius 0.70 m. RK4 integration.

Margins (g on reward, l on info["l_x"])

  • g = signed distance from the car's rectangular footprint to the nearest obstacle circle, normalized, clamped ±3. g ≥ 0 ⟺ not in collision.
  • l = target margin, piecewise: +GOAL_VALUE (0.3) at the goal center → 0 at the boundary → gentle negative outside. l ≥ 0 ⟺ in the goal. The piecewise shape gives the value real positive range while keeping a reach gradient far out (a single linear scale can't do both).

Run it

from safety_sb3 import ReachAvoidSAC1P        # or ReachAvoidPPO1P
from safety_sb3.testing.bicycle5d_vec import BicycleGoalVec

env = BicycleGoalVec(16, spawn="wide")        # batched, full-map spawns
model = ReachAvoidSAC1P("MlpPolicy", env, buffer_size=500_000, learning_starts=5000,
                        batch_size=512, train_freq=(16, "step"), gradient_steps=16)
model.learn(2_000_000)                        # ~100% coverage

The avoid setup is the same with SafetySAC1P (no l is used). The full demo — multi-car GIFs, value maps, and the PPO/SAC comparison — is examples/bicycle5d_demo.py and examples/bicycle5d_value_compare.py:

python examples/bicycle5d_demo.py --family sac --steps 2000000   # avoid + reach-avoid
python examples/bicycle5d_value_compare.py                       # PPO vs SAC value maps