The dream of a general-purpose AI that can do anything your phone’s chatbot does, including pilot a two-ton vehicle, just got a reality check. Researchers handed frontier models complete control of a Toyota Corolla and watched most of them fail spectacularly before the first turn.
Researchers Aditya Ramabadran, Simon Mahns, and Tobias Gessler built DrivingBench, a test designed to see how well off-the-shelf AI agents like Claude, GPT, and Grok handle actual driving. The hardware was borrowed from the autonomy world: Comma vision gear running OpenPilot. The course was deliberately simple: a short point-to-point route through a parking lot with a few bends, marked out by small cones.
Each agent got the same command, the same guidelines, and full control over throttle, brakes, and steering of a midsize sedan. Everything ran within one continuous chat, with commands beamed to data farms over a wireless hotspot, which is why the cars kept stopping mid-run to wait for instructions. Each agent had three attempts.
The verdict, per DrivingBench, was blunt: frontier models are “frankly, not good at this.” Across 11 runs spanning four agents, eight of them never cleared 11 percent of the route, washing out at the very first bend.
Grok read a gap in the mini-cones marking the outer boundary as a gate to drive through. It drove straight off, with its opening run over after just two commands. One GPT model invented a rule declaring the cones on one side all a single color, a mistake the human-written command had explicitly warned against. Many agents simply couldn’t calculate how much steering angle a bend required, and failed to turn enough.
Only GPT-6 Astra completed the full course, on its second attempt. The line was ugly, hugging the long right-hander and then nearly spilling off the edge just shy of the finish box, but it finished.
The cost of that win: $7.74, roughly four times the charge of an earlier attempt that got only halfway. More money, more tokens, more chances to send a Corolla into the cones. The future is here, and it bills by the token.
[Images: Driving Bench]
Become a TTAC insider. Get the latest news, features, TTAC takes, and everything else that gets to the truth about cars first by subscribing to our newsletter.
