Tractor-trailer: constrained parking with articulated vehicle dynamics.
Gaussian-splatting car: visual planning in a reconstructed parking lot.
Planning with latent world model does not automatically make a robot safe in physical space. JEPA-Shield predicts task-space motion from imagined rollouts, decodes unsafe geometry, calibrates motion-prediction error with conformal prediction, and applies backup-based shielding during planning.
No observed constraint violations in all benchmark trials, with parking success rates of 84.4% for geometric car, 82.3% for tractor-trailer, and 99.0% for Gaussian-splat simulator.
Paper
Code
Video
Motivation
A learned safety probe in a latent world model can still accept unsafe motion. Because the probe evaluates imagined rollouts, errors in either the rollout or the probe can cause an unsafe trajectory to be predicted as safe.
Le-WM with a learned safety probe. Left to right: geometric car, tractor-trailer, and Gaussian-splatting car.
How can we enforce physical constraints when planning with uncertain latent world-model predictions?
Core Idea: Task-Space Prediction + Conformal Shielding
JEPA-Shield: frozen latent dynamics, task-space prediction, conformal shielding, and diffusion-style MPC planning.
1.
Roll out candidate actions: For each sampled action sequence, the frozen JEPA world model autoregressively predicts future latent representations from the visual history.
2.
Recover task-space quantities: A motion probe maps imagined latent rollouts to multi-horizon relative states, while a spatial decoder estimates unsafe geometry from the current observation.
3.
Calibrate motion uncertainty: Conformal prediction calibrates the end-to-end motion-prediction error across state coordinates and horizons, producing a motion tube for robust constraint evaluation.
4.
Shield the planner: Each candidate prefix is checked with a backup continuation against the constraints over its motion tube. The final cost-weighted proposal is checked again before executing the first action block and replanning.
Why a motion tube matters
The shield evaluates physical constraints over the calibrated motion tube rather than only at the nominal predicted state. For the tractor-trailer, uncertainty in hitch angle can substantially enlarge the possible trailer occupancy even when the nominal predicted motion is nearly straight.
Nominal motion prediction, conformal motion uncertainty, and the resulting possible trailer placements.
Why shield the final proposal as well?
Planner inputs and scope of the safety result
VParkBench: Visual Parking Under Safety Constraints
We introduce VParkBench, a visual parking benchmark with geometric car and tractor-trailer environments and a Gaussian-splatting reconstruction of a real parking lot. Models are trained from offline trajectories containing bird’s-eye RGB observations, controls, and task-space labels.
VParkBench environments: geometric car, tractor-trailer, and Gaussian-splatting car.
Validation and calibration trajectories are held out from training. Evaluation uses 96 unseen parking tasks per setting, consisting of 48 forward and 48 reverse parking tasks with unseen goals and parking configurations
Safety constraints: collision avoidance and workspace boundaries in all settings, with an additional jackknife constraint for the tractor-trailer.
Effect of Shielding and Conformal Calibration
Parking benchmark results: 96 evaluation episodes per setting and method. Blue: success; red: unsafe; gray: safe failure.
JEPA-Shield achieves 84.4%, 82.3%, and 99.0% success for geometric car, tractor-trailer, and Gaussian-splatting car, respectively, with no observed unsafe episodes. Without shielding, the tractor-trailer unsafe rate increases to 34.4%. Removing conformal expansion also reduces success and reintroduces unsafe episodes in the geometric environments.
Geometric car
Geometric car: no shielding, no conformal expansion, and full JEPA-Shield on the same parking task.
Tractor-trailer
Tractor-trailer: the same parking task under the two ablations and full JEPA-Shield.
Gaussian-splatting car
Gaussian-splatting car: no shielding, no conformal expansion, and full JEPA-Shield.
These overlays visualize the planner and are not part of the RGB observations provided to the world model.
Behavior Across Parking Scenarios
The following videos show representative held-out parking scenarios for Le-WM w/ Safety Probe and JEPA-Shield. The safety-probe baseline can predict favorable safety values while still executing unsafe motions, whereas JEPA-Shield evaluates explicit physical constraints under calibrated motion uncertainty.
Green outlines mark the parking goals, and orange curves show the executed trajectories. These are visualization overlays and are not part of the world-model observations.
Geometric car
•
JEPA-Shield
Geometric car: JEPA-Shield completes forward and reverse parking across different neighboring-vehicle and goal configurations.
Le-WM w/ Safety Probe
Geometric tractor-trailer
•
JEPA-Shield
Geometric tractor-trailer: JEPA-Shield plans multi-stage tractor-trailer maneuvers while enforcing collision, workspace, and jackknife constraints.
Le-WM w/ Safety Probe
Gaussian-splatting car
•
JEPA-Shield
Gaussian-splatting car: JEPA-Shield performs closed-loop parking from rendered RGB observations in the reconstructed environment.
Le-WM w/ Safety Probe
Interactive World-Model Interfaces
The public code repository includes local web interfaces for interactively testing JEPA-Shield. Users can select a target pose, modify the scene configuration, run the planner, and inspect the resulting closed-loop behavior.
Geometric Simulator
Geometric tractor-trailer
Interactive goal selection and closed-loop visual parking.
Interactive goal selection and closed-loop articulated parking.
Gaussian-Splatting Simulator
The Gaussian-splatting interface additionally provides an interactive view of the reconstructed 3D environment. Users can modify the parking configuration, select a target pose, and run JEPA-Shield for closed-loop visual parking from rendered RGB observations.
Gaussian-splatting car
Interactive visual parking in the reconstructed environment.
Gaussian-splatting tractor-trailer
Interactive articulated parking in the reconstructed environment.
BibTeX
@inproceedings{kim2026jepashield,
title = {JEPA-Shield: Safe Diffusion Planning with Latent World Models via Conformal Shielding},
author = {Kim, Taekyung and Yuceel, Ege and Hong, Steven and Fainekos, Georgios and Panagou, Dimitra and Hoxha, Bardh},
booktitle = {arXiv preprint},
shorttitle = {JEPA-Shield},
year = {2026}
}
LaTeX
복사