SAM 3 Pedestrian, Distilled for the Edge

The production gap in modern vision models

The last two years of computer vision have been defined by one uncomfortable truth: the best models are almost never the deployable ones. Meta's Segment Anything family redefined what general purpose segmentation looks like, but the flagship ViT-H backbones run at roughly 1.5 seconds per image on a consumer GPU. That is three orders of magnitude away from the budget an automotive ECU, an on-board robot, or an AR headset can spend per frame.

Closing that gap is not an afterthought, it is its own research problem. Production perception runs under hard real time budgets, on fixed silicon, inside a thermal envelope, and with a safety case that has to be defended in court. You cannot ship a 600M parameter teacher into a vehicle and hope for the best. You have to compress it without losing the part of the intelligence that matters for your task.

Distillation as the bridge

Knowledge distillation is the most honest answer we have. Instead of training a small model from scratch and hoping it discovers the same representations as the giant one, we let the foundation model act as a teacher and we train a much smaller student to imitate its behaviour on the data distribution we actually care about. The student inherits the teacher's perceptual priors without inheriting its cost.

SAM-Pedestrian-RT-Distilled is a public reference implementation of that idea, end to end, on a single laptop GPU (RTX 4070, 8 GB). Pedestrians are treated as expert #1 of a future Mixture of Experts perception stack: each subsequent class (vehicle, cyclist, sign, lane, drivable area, long tail) clones the same recipe, so adding experts does not scale runtime cost. The closest published prior art for this scenario-aware MoE-on-edge pattern is EMC2 (ICCV 2025).

The repo ships two phases:

• Phase 1. Distill SAM 3 into a YOLOv8n-Seg edge student for real-time 2D camera-only pedestrian detection and segmentation.
• Phase 2. Use SAM 3 as a 2D oracle to produce lidar-native 3D pedestrian pseudo-labels on nuScenes, with zero human 3D labels.

Phase 1, real-time 2D student on a held-out video

Evaluation is on the Intel person-bicycle-car-detection.mp4 clip (647 frames), RTX 4070 FP16, confidence threshold 0.25. Data hygiene matters: training is on COCO val2017 person images relabeled by SAM 3, the demo footage is never seen during training.

The distilled 1 h run finds 21% more pedestrians than the COCO baseline on a video the student has never seen, while still running real time at 22 FPS on a single laptop GPU. Collapsing back to a single class head post distillation, or raising the confidence threshold, recovers the 50+ FPS regime trivially.

Phase 2, SAM 3 as a 2D oracle for 3D pedestrian supervision

Phase 2 attacks the harder problem. 3D pedestrian labels are expensive, ambiguous, and require human annotators clicking through point clouds. The recipe here flips the lift: run SAM 3 on each of the six nuScenes cameras at every keyframe, project those 2D masks onto the lidar sweep, and let multi-view consensus decide which points are pedestrians. Zero human 3D labels are used during pseudo-labeling.

Evaluation is point-level agreement against nuScenes 3D pedestrian box ground truth, on the full nuScenes mini: 404 keyframes, 14.0 M lidar points, 47,965 GT pedestrian points.

An overall F1 of 0.67 against human-curated 3D boxes, with no humans in the labeling loop, is a strong result. The structure of the failure modes matters even more than the headline number.

The named failure mode

Breaking the error down by how many of the six cameras actually saw each lidar point exposes the real story:

30.9% of GT pedestrian points sit in the union of the under-vehicle blind spot and points outside every camera frustum at the keyframe instant. The 2D-to-3D lift cannot fire on those by construction. That single sensor configuration story accounts for the entire gap between 0.67 overall recall and 0.94 recall on camera-visible points. The fix is not a better model, it is temporal aggregation across neighbouring keyframes, which is named as the next follow-up in the repo.

Architectural commitments worth keeping

1. Backbone outputs a feature pyramid at named taps (P3 / P4 / P5). The Expert protocol makes these the integration surface for future class experts. Class-specific computation never fuses into the backbone.
2. Single class today, copy paste tomorrow. Adding cyclists, vehicles, signs is a new head plus a new SAM 3 prompt set plus a new YAML. The training pipeline does not change.
3. Negative prompts matter. The pedestrian expert is explicitly negative-prompted against cyclists and scooter riders so its training set does not pollute the future cyclist expert. A post-hoc filter belt-and-suspenders this.
4. Foundation models are teachers, not oracles. SAM 3 runs at roughly 1.5 s per image on this hardware. The point of the whole recipe is to push that cost offline so the on-vehicle student does not pay it at inference time.

Why Bespoke AI is shipping this

Bespoke AI exists to be the bridge between the human and the digital world. Every sketch, every render, every 3D asset our platform produces is eventually consumed by a system that has to reason about people. Open-sourcing a real, reproducible distillation recipe (with named benchmarks, named failure modes, and named follow-ups) is our way of contributing back to the perception stack our automotive partners depend on. Great open source is not just a checkpoint, it is the recipe other teams can run on their own hardware tomorrow morning.

Code, weights, training scripts, ONNX export and the full Phase 2 orchestrator are on GitHub. Pull requests, new benchmark splits and reports from the field are very welcome.