Predictive Semantic Safety in closed loop: a VLM predicts a physical event from visual reasoning, the prediction is converted into future unsafe occupancy, and the safety filter modifies the robot’s motion when needed.
A passage that is clear now can become unsafe as a suspended fixture falls, a stack of boxes becomes unstable, or contact causes nearby objects to move. Predictive Semantic Safety (PSS) translates a foundation model’s physical predictions into time-varying unsafe occupancy and a backup-based safety filter for the robot’s control input.
99.3% safe episodes (149/150) across three types of physical hazards. The filter minimally modifies a nominal controller while preserving a feasible backup against predicted future occupancy.
Motivation
Humans navigate not only by reacting to obstacles that are already present, but also by visually anticipating how the scene may change. For example, that a suspended fixture may fall or an unstable stack may collapse. PSS brings this kind of physical anticipation into safety-critical control, allowing the robot to respond before a predicted hazard enters its path.
Current-geometry safety
Backup CBF using current obstacle geometry: the robot continues beneath the fixture and collides after it falls.
The route is initially unobstructed. By the time the falling fixture enters the robot’s path, it may already be too late for the robot to avoid it safely.
Semantic scene understanding
Conceptual example: a scene may contain regions that are unsafe to enter even when they are geometrically traversable.
Semantic understanding can identify such regions, but a static semantic label does not predict how physical interactions will change the scene over time.
How can a robot use visual physical reasoning to anticipate hazards and respond before they enter its path?
•
Relation to recent semantic safety methods
Recent semantic-safety methods use scene understanding to identify regions or behaviors that should be avoided based on object semantics and task context. Examples include Qian et al. (2024) and Brunke et al. (2025). PSS instead considers how physical interactions may change object motion over time and converts those predictions into time-varying unsafe occupancy for safety filtering.
Core Idea: From Physical Prediction to Safety Filtering
PSS converts visually predicted physical events into time-varying unsafe occupancy, then uses that occupancy to determine whether the robot still has a feasible backup response.
1.
Predict how the scene may change: A VLM physics prediction model uses visual observations and measured motion to predict how an object may move and when that change may occur. For the falling fixture, an explicit motion model converts the predicted event and timing into future trajectories. For the stack and domino scenarios, future object motion is predicted directly.
2.
Construct future unsafe occupancy: The predicted motion is combined with object geometry and calibrated prediction uncertainty. Conformal prediction accounts for position error across the specified objects and future times, producing occupancy regions used for safety evaluation.
3.
Preserve a feasible backup: PSS evaluates a prescribed backup maneuver against the predicted occupancy and requires its terminal state to admit a safe continuation. The resulting input-affine constraints minimally modify the nominal command while respecting the robot’s input constraint.
From an observation to a future hazard
Recorded VLM output and predicted motion in the ceiling-fixture example. The VLM predicts the physical event and its timing, and an explicit motion model converts this into future object motion: supported before detachment, falling after detachment, and stationary after ground contact.
The physical prediction describes how the hazard may evolve, while the nominal policy remains responsible for the navigation task. PSS connects the two through the safety filter.
A lightweight CBF-QP safety filter
PSS is independent of the nominal policy. Its input-affine safety constraints are enforced through a CBF-QP that minimally modifies commands from controllers such as RL, VLA, or MPC while preserving backup feasibility.
The input-affine constraints and quadratic program
Evaluation Scenarios
We evaluate PSS in MuJoCo with a quadruped using onboard RGB-D observations and a prescribed backup policy (e.g., stopping, moving backward, etc). The benchmark includes falling fixtures, unstable stacks of boxes, and falling dominoes that propagate motion through nearby objects, with 150 episodes per method.
Compared methods: Plain CBF, Backup CBF, OmniVLA, and PSS. The CBF baselines use current obstacle geometry, while PSS evaluates the backup against predicted future occupancy. OmniVLA directly predicts navigation commands without an additional safety filter.
1. Falling ceiling fixture
A partially detached fixture can make an initially clear passage unsafe. PSS predicts the developing fall and accounts for its future occupancy before the fixture enters the robot’s path.
2. Unstable Stack of Boxes
An impact causes an initially stationary stack to lose support. PSS predicts the resulting block motion and incorporates it into the future unsafe occupancy.
3. Falling Dominoes
A local contact can propagate motion through multiple objects. PSS predicts how this interaction develops and accounts for the resulting occupancy during navigation.
Overlays: cyan curves show executed paths, the inset images show the robot’s observation view where available.
Active Avoidance under Evolving Occupancy
PSS updates the safety constraints as the predicted occupancy changes over time. When the nominal command would violate the predictive safety constraints, the CBF-QP minimally modifies it to preserve feasibility of the prescribed backup maneuver against the predicted future occupancy.
Additional qualitative example: current-geometry backup evaluation leads to a collision, while PSS accounts for the evolving occupancy and safely navigates around the moving objects.
Safety Performance
The comparison supports reasoning about future occupancy, beyond a geometric interpretation of the current scene.
A computationally efficient CBF-QP safety layer. PSS turns foundation-model predictions into input-affine safety constraints that can be enforced by a lightweight CBF-QP. Because the filter only modifies the nominal command when necessary, it can wrap around independent policies such as RL or VLA controllers.
More PSS Demonstrations
Ceiling fixtures
Stacks
Dominoes
BibTeX
@inproceedings{kim2026pss,
title = {Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control},
author = {Kim, Taekyung and Fradi, Salem and Dai, Yanning and Ostaszewski, Mateusz and Schmidhuber, J{\"u}rgen},
booktitle = {arXiv preprint arXiv:},
shorttitle = {PSS},
year = {2026}
}
LaTeX
복사