Tractor-trailer: planning a maneuver through a constrained parking scene.
Gaussian-splatting car: planning from images in a reconstructed parking lot.
Planning in latent space does not automatically make a robot safe in physical space. JEPA-Shield connects a frozen JEPA world model to task-space safety checks: predict motion, decode unsafe geometry, calibrate uncertainty with conformal prediction, and shield actions before execution.
No observed constraint violations in all benchmark trials, with parking success rates of 84.4% for geometric car, 82.3% for tractor-trailer, and 99.0% for Gaussian-splat simulator.
Paper
Code
Video
Motivation
A learned safety probe in a latent world model can still accept unsafe motion. Because the probe evaluates imagined rollouts, errors in either the rollout or the probe can cause an unsafe trajectory to be predicted as safe.
Le-WM with a learned safety probe. Left to right: geometric car, tractor-trailer, and Gaussian-splatting car.
How can we evaluate physical safety from latent predictions while accounting for motion-prediction error?
Core Idea: Task-Space Prediction + Conformal Shielding
JEPA-Shield: frozen latent dynamics, task-space prediction, conformal shielding, and diffusion-inspired planning.
1.
Imagine candidate motions. For each sampled action sequence, the frozen world model predicts future latent representations autoregressively from RGB observation history.
2.
Recover task-space geometry. A motion predictor maps the latent rollouts to relative position and heading. A spatial decoder estimates obstacles and unsafe regions from the latest image.
3.
Account for prediction error. Split conformal prediction calibrates the largest normalized motion error across coordinates and horizons on held-out trajectories, producing a motion tube for safety checks.
4.
Shield and execute. Keep an action prefix only when following it with a predefined backup policy satisfies the constraints throughout the tube. Check the final cost-weighted proposal again, execute the first action block, and replan.
Why a motion tube matters
The shield checks possible vehicle footprints inside the calibrated tube, rather than only the nominal prediction. During unstable reverse driving, hitch-angle uncertainty can substantially enlarge the trailer’s possible occupancy.
Predicted motion, possible realized motion, and the corresponding trailer occupancy.
Why shield the final proposal as well?
Planner inputs and scope of the safety result
VParkBench: Visual Parking Under Safety Constraints
We introduce VParkBench, a visual parking benchmark and offline dataset for training and evaluating planning from bird’s-eye RGB observations. It includes geometric car and tractor-trailer environments and an editable Gaussian-splatting reconstruction of a real parking lot.
Geometric car
Bird’s-eye environment overview.
Geometric tractor-trailer
Articulated vehicle and parking bays.
Gaussian-splatting car
Editable photorealistic simulation.
Models are trained on collected trajectories. Validation and calibration trajectories are held out, and evaluation uses 96 unseen scenarios per setting: 48 forward and 48 reverse parking tasks with goals and configurations excluded from training.
Safety constraints: collision avoidance and workspace boundaries in all settings; jackknife avoidance for tractor-trailers.
Comparison: What Do Shielding and Calibration Add?
Full parking benchmark comparison; 96 evaluation episodes per setting and method. Blue: success. Red: unsafe. Gray: safe failure.
Full JEPA-Shield has the highest success rate in all three settings, with no observed unsafe episodes. Without shielding, the tractor-trailer unsafe rate is 34.4%. Removing conformal expansion lowers success and reintroduces unsafe episodes in the geometric environments.
In each comparison video, left to right: no shielding → no conformal tube → JEPA-Shield.
Tractor-trailer
The same tractor-trailer task with shielding and uncertainty calibration removed in turn.
Legend: blue lines = accepted candidates; red dashes = rejected candidates; yellow/black trajectory = optimized proposal; green box = parking goal; translucent yellow region = conformal motion tube, showing possible vehicle occupancy.
Geometric car comparison
Gaussian-splatting car comparison
The planning overlays explain the results; they are not included in the images supplied to the latent world model.
Parking Across Diverse Scenarios
Green outlines mark parking goals; orange curves show executed trajectories. These overlays are added for interpretation and are not world-model inputs.
Geometric car
Forward and reverse parking with varied neighboring occupancy.
Geometric tractor-trailer
Coordinated tractor and trailer maneuvers under collision, workspace, and articulation constraints.
Gaussian-splatting car
Planning from rendered RGB observations in the reconstructed parking lot.
Interactive World-Model Interfaces
The code repository includes a web interface for mouse-based interaction with the latent world model: choose a target pose, run the planner, and inspect its predictions and resulting motion. The clips below are recordings of local simulator interfaces.
Geometric tractor-trailer
Interactive target selection and articulated motion planning.
Gaussian-splatting car
Interactive planning in the reconstructed scene.
Additional Demo: Gaussian-Splatting Tractor-Trailer
We also demonstrate a tractor-trailer in the Gaussian-splatting interface, with the 3D scene and rendered bird’s-eye views shown together. This additional demonstration is separate from the 288-trial benchmark, whose quantitative Gaussian-splatting evaluation covers car parking.
Gaussian-splatting tractor-trailer: target selection, planning, and execution.
Geometric car interface
BibTeX
@inproceedings{kim2026jepashield,
title = {JEPA-Shield: Safe Diffusion Planning with Latent World Models via Conformal Shielding},
author = {Kim, Taekyung and Yuceel, Ege and Hong, Steven and Fainekos, Georgios and Panagou, Dimitra and Hoxha, Bardh},
booktitle = {arXiv preprint},
shorttitle = {JEPA-Shield},
year = {2026}
}
LaTeX
복사