Agents need a body to be useful in AV

LLMs are extraordinary at reasoning about driving in the abstract. They can explain right of way at a four-way stop, critique a lane change, sketch a test plan for an unprotected left turn. What they cannot do, on their own, is reach into a simulator, place a cyclist behind a parked van, dial the fog up to 90 percent, run the scenario, and tell you what the ego vehicle did. They have a mind but no body. They can talk about the world without touching it.

For autonomous driving research, that is the bottleneck. CARLA is the de facto open simulator for AV work, but every meaningful experiment still starts with a human writing Python against a low level API: spawn actors, attach sensors, tick the world, parse ground truth, glue it all together. That friction is why most teams test a tiny fraction of the scenarios they care about.

Hands and eyes through the Model Context Protocol

The Model Context Protocol is the cleanest standard we have for giving an LLM a structured, auditable way to act on the world. Tools are typed, calls are explicit, side effects are observable. carla-mcp v2.2 ships 52 such tools, including simulator-process orchestration so Claude can launch CARLA itself and short-circuit if it is already running.

With it, an agent gains two things at once. It gains hands: spawn vehicles and pedestrians, attach sensors, change weather and time of day, drive in autopilot, step the world forward, record and replay runs. And it gains eyes: structured access to RGB, depth, semantic, instance, optical-flow, DVS cameras, lidar and semantic lidar, returned in a form the model can actually reason about. The same agent that previously could only describe a scenario can now build it, run it, watch it, and iterate.

The 3D analysis suite, the actual headline

Most MCP connectors for simulators stop at camera screenshots. carla-mcp is strongly biased toward lidar, point clouds, semantic segmentation, and perception evaluation. The 3D suite is what makes it useful for AD ML engineers rather than just demos.

Tool	What it does
render_lidar_3d	Headless 3D scatter render (iso / bev / rear / side) of any sweep.
point_cloud_clusters	DBSCAN proposals to BEV PNG plus centroids and extents.
ground_plane_segment	RANSAC plane fit, ground vs obstacle BEV PNG.
voxelize / semantic_voxelize	Sparse 3D occupancy grid, geometric or class-tagged. Direct ground truth for OccNet / BEVFusion.
lidar_to_camera_overlay	Pinhole-project lidar onto RGB, color by depth.
iou_3d	AABB 3D IoU between two actors.
export_point_cloud	Write to disk as PLY, PCD, NPY, or KITTI BIN.

Real perception evaluation, no model needed

The v2.1 tools exploit semantic lidar's per-point ObjIdx to do real ML evaluation against ground truth, with no detector in the loop:

evaluate_clustering matches DBSCAN proposals against ground-truth ObjIdx and reports precision, recall, mean IoU, per-actor matches, and a BEV PNG (matched green, false positives red, missed orange).
check_sensor_consistency captures the semantic camera and projects the semantic lidar into it, reporting per-class agreement plus an overlay PNG with disagreements highlighted.
actor_visibility returns per-actor lidar hit count and a visibility class (high, medium, low, occluded), a clean filter for autolabel quality.
auto_label and export_dataset emit KITTI or nuScenes formatted labels and full dataset folders (calib, image_2, velodyne, label_2) from a single tool call.

This is the part that makes carla-mcp useful as serious infrastructure. The same primitives that power the Phase 2 work in the SAM Pedestrian repo (multi-view consensus, lidar projection, cluster-vs-GT evaluation) are exposed here as agent-callable tools.

A full AD ML loop, fully agentic

One paragraph in a chat window now drives a real perception evaluation:

Start the simulator. Load Town10HD, set weather to MidRainSunset, spawn 25 traffic vehicles and 8 pedestrians, then spawn a Tesla Model 3 with autopilot. Capture front RGB, run compare_seg_with_truth, render lidar in iso view, compute lidar stats, extract 3D bboxes in KITTI format, then spawn an adversarial cut-in 15 m ahead and capture the swerve frame. Summarize what an AD perception engineer would learn.

Claude chains the tools, returns roughly six inline images, a KITTI label dump, and an engineer-eye-view summary, in about 60 seconds of execution. The same surface supports adversarial prefabs (cut-in, sudden brake, jaywalker), failure snapshots on collision or lane invasion, scenario sweeps, OpenSCENARIO 1.x execution, and remote use through claude.ai Custom Connectors over a Cloudflare tunnel with bearer-token auth.

Why Bespoke AI is shipping this

The same agents that design vehicles inside Bespoke AI should be able to validate them in simulation. carla-mcp is a first step toward that closed loop: design, render, simulate, evaluate, iterate, all reachable from one conversation. We released it as open source because the AV community will build far more on top of it than any single team could alone, and because giving language models hands and eyes in serious simulators is one of the more interesting bridges between the human and the digital world.