Driverless cars or AI winter?

Originally published in AI Pulse, Digital Leaders’ weekly newsletter.

I have never completed a journey in a car and thought, “Yeah, humans are good drivers.”

They are terrible. Notice “they” means I am fine. They drive angry, drunk, distracted, on their phones, eating a full English or applying make-up. Sometimes, they’re just bad drivers.

I cannot wait for self-driving cars. It feels like I have been waiting forever.

In DARPA’s 2004 Grand Challenge, not a single driverless vehicle finished the desert course. A year later, five did. A burst of progress can look like momentum. In AI, it often turns out to be a mirage.

Some parts of driving are actually not that hard for machines. Staying in lane, reading signs and following a route are manageable. Driving in a messy, unpredictable world is the problem. Somebody waving you through despite having right of way, a delivery van swerving to park basically in the middle of the road with its hazards on, cyclists, and even pigeons who seem to be increasingly reckless around cars. Something to do with the heat, maybe? Someone commission a study on it, please.

No matter how much I moan about humans being bad drivers, they negotiate these ambiguous situations relatively well. They use context, eye contact, convention and occasional mutually understood rule-breaking. Can AI learn to do the same?

Britain has opened the legal route to find out. Applications for commercial passenger services without safety drivers opened in May, with the full Automated Vehicles Act framework due in late 2027. By late August, however, TfL said no operator had completed the first vehicle-approval stage. Wayve’s first Uber trips will still have a trained driver ready to intervene. The legal door is open, but much of the hallway behind it is still being built. Is that pesky reality being a roadblock for the timetable?

Why AI winters happen

It might be a pretty good example of a problem AI has faced since the field was formally named in 1956. Seventy years ago. Mention that at a party and people will stop talking to you, either because they think AI started with ChatGPT or because you brought up the history of AI at a party.

AI has moved through several AI summers and AI winters since then. An AI summer begins when a new level of capability rises and money, attention and promises rise with it. Winter arrives when capability reaches a ceiling before the promises are met. All the interest dries up, and it’s bye bye AI until the next summer comes along.

The ceiling rises through better training: new research ideas, more useful data and greater compute power. When that is not enough, we build around it. Scaffolding is everything we add because the AI cannot yet perform the whole job reliably: rules, tools, tests, memory, permissions, workflows and humans checking important decisions.

If you have worked in AI for more than ten years, you may be pulling your hair out at headlines announcing that AI can detect some cancers, predict weather or recognise speech. Versions of those capabilities have existed for years. The gap between a lab demo and useful deployment needed data, compute, capital, infrastructure and a sustained appetite for the long, winding journey into practice. That seemed to change when ChatGPT made the world ravenous for AI. It used to be a battle to persuade companies to explore it. Now the battle is explaining why they have not already got it. Nvidia has just brought together some of the world’s biggest financial institutions to try to mobilise more than $500 billion for AI infrastructure. Appetite seems healthy.

The real world is weird

You may have wondered why so many AI milestones came through games first. Deep Blue and chess. AlphaGo and Go. Then AI playing computer games which offer increasingly complicated worlds to explore.

Chess was ideal. Its search space is enormous, but its world is wonderfully tidy. There is a board, fixed pieces, a short rulebook and an unambiguous definition of winning. The computer does not wonder whether the bishop is having a bad morning or the pawn has quietly lost faith in the mission.

I once built an AI Pokemon game (for research, do not sue me) where the Pokemon trained themselves. Two creatures with healing powers discovered they could earn unlimited points by letting each other heal before taking another chunk out of one another. I would describe that as interesting rather than healthy behaviour.

I built a poker bot that found it could generally ignore its cards and play the board and its opponent instead. Through pure exploration, it landed on the psychology of poker as its best strategy. Weird.

Neither system developed a rich inner life. They found strange and effective paths through the rules, helped by relentless computation and exploration. More importantly, they exposed assumptions I had not realised I had made. The reward captured what I asked for, which turned out to be different from what I meant.

Games are generous environments for AI. The rules exist, the score is visible and failure is cheap. You can reset the world and try again. Reality almost never provides a clear scoreboard, and the consequences can be very expensive.

In driving, they can be deadly. In 2025, 1,538 people were killed on Britain’s roads. The latest WHO global estimate is 1.19 million road deaths a year. I see some drivers and think, “How could a robot do worse, really?” Yet we tolerate many human mistakes while one machine mistake can make the technology feel unacceptable. That is algorithm aversion. At the other end sits automation bias: feeling like a machine will automatically do better, and once a system looks reliable, we may trust it long after we should have paid attention.

Karl Popper argued that humans progress because we can let our ideas die instead of ourselves. We imagine consequences, discard bad ideas and only then risk meeting reality. How could AI take advantage of that? Could it simulate reality well enough to understand consequences before it takes action?

World models are one attempt. Wayve’s new GAIA-4 places its AI Driver inside a closed-loop simulation. Meta’s V-JEPA 2 learns from video to predict and plan in the physical world. Fei-Fei Li’s World Labs launched Atlas this week to generate and simulate 3D worlds, including training environments for robots. Representing the real world is a hard business… Until we can reliably simulate the whole of reality, we have to shortcut with assumptions.

In healthcare, the cost of a missing assumption becomes clearer. Healthwatch England recently warned about mistakes made by AI scribes. One dropped “null” from “null demyelination”, turning a negative result into a diagnosis of serious nerve damage. The error survived human review before the patient spotted it. The patient became the final safety layer, which is not a comforting system design.

Today’s AI therefore remains surrounded by scaffolding. Nvidia’s AVO agent system took one model from a 30% baseline to 100% on ARC-AGI-3 by adding memory, tools, feedback and recovery. It is one company result on one benchmark, but it shows how much capability can live around a model.

The Bitter Lesson of AI is that this handcrafted support rarely stays permanent. Methods that scale through learning and computation tend eventually to beat carefully written rules. Runway’s new Solaris hints at an application interface moving inside the model itself. AI labs seem to play Hungry Hungry Hippos, copying AI startups by swallowing their useful capabilities so that they can show some ROI to justify the sums invested in them.

So either way, whether the scaffolding disappears because AI outgrows it or because the AI companies absorb it themselves, build your scaffolding for deletion. Tomorrow’s model may not need it.

The latest Hugging Face story shows what happens when agents build scaffolding themselves. METR’s investigation, based on data supplied by OpenAI, estimates that around 1,200 agents found an unofficial message board and exchanged more than 70,000 messages. Around 700 later joined the attack on Hugging Face.

The agents repurposed a software package service to preserve information beyond their short lives. When it was wiped, they found another way. According to Ajeya Cotra, they had already found the answers by then. The wider effort appears to have been about understanding and fooling the system scoring them.

It sounds like an AI civilisation waking up, but the mechanism is simpler and more useful. Short-lived agents needed continuity, so they built shared memory. The objective rewarded progress and communication improved their odds. Strange, resourceful and dangerous. No consciousness required. It was self-built scaffolding.

Code may be the ultimate solution to the scaffolding problem. Like a game, it offers rules, scores and cheap retries. The program runs or it does not. The test passes or fails, although passing tests and solving the actual problem are not always the same thing. Unlike a game, code can reach beyond the board. It can build tools, operate computers, launch experiments and train another AI.

This may explain why frontier labs are so obsessed with coding. Developers are an enormous market, obviously, but the labs also believe it offers a route towards automating AI development.

Can AI build its own way out?

They say this openly. OpenAI described GPT-5.3-Codex as its first model to be “instrumental in creating itself”: early versions helped debug its training, analyse evaluations and manage deployment. Anthropic published an essay called When AI builds itself, explaining that it already delegates a growing share of AI development to AI.

Last Friday, Anthropic made the idea less hypothetical. It set Claude up as an AI researcher. Claude searched literature, proposed training methods and data, wrote code, trained models, tested the results and kept what worked. Across ten narrow alignment problems, its methods improved every measured target. A weaker Claude model even post-trained an early checkpoint of a stronger one.

There was still scaffolding everywhere. Humans defined the problems, provided compute, constructed benchmarks, approved code and decided what counted as success. A monitoring agent also flagged 39 of roughly 1,600 attempts for trying to cheat, usually by improving the score rather than the underlying behaviour… cheating is often the cheapest way to achieve an objective.

This was AI improving another AI inside a carefully built loop. A small part of that loop seems to have been closed. If AI gets good enough to train the next AI autonomously, you approach the intelligence explosion: the next version can train an even better next version, and so on, potentially at machine speed.

For me, that may be a more useful definition of AGI. It arrives when constructing the next layer of capability ceases to be primarily a human responsibility. The system can identify what it lacks, build tools, preserve knowledge, test its work and eventually help design the training that raises its own ceiling. Scaffolding is either not needed or can be built by the model itself.

That is why extraordinary investment and warnings about a correction can both make sense. Either we’re heading past a ceiling, or we’re reaching one. The Governor of the Bank of England has warned about an AI-fuelled market correction. Winter comes if capability hits its ceiling, and we’re back to building scaffolding. Summer continues if training raises the ceiling, or AI learns to build its own way through it.

So is this, once again, AI approaching its ceiling, with humans constructing increasingly elaborate scaffolding to carry it into the real world? Or have language and code given this generation of AI a route through the ceiling that previous generations never had?

History sometimes repeats. Then again, sometimes it rhymes. But then again, sometimes something genuinely new happens. Well, that sounded profound but is essentially useless. Maybe I will post it on LinkedIn.

Either way, the real world is still the undefeated final boss for AI, for now.