Anthropic: You don’t need to worry about recursive-self-improving AI – yet

One of the world’s leading artificial intelligence companies has implored the industry to pause development on AI, because the latest models could be reaching a tipping point where they become capable of redesigning themselves, growing ever more powerful and finally escaping our control. At least, that’s what the headlines said.

In truth, Anthropic’s co-founder Jack Clark and the boss of spin-out think-tank The Anthropic Institute, Marina Favaro, have published a long blog post bigging up the capabilities of their Claude model, shortly before the company floats on the stock exchange in an initial public offering (IPO) for a rumoured $1 trillion.

Let’s, for a moment, ignore the vast financial elephant in the room and look at the technological claims. An AI that becomes capable of designing a more powerful version of itself, which is in turn able to pull off the same feat, is an obvious gamechanger, but it is also not a new idea. While Anthropic now calls this “recursive self-improvement”, people have spent decades talking about “the singularity” as the moment that this occurs.

It’s not clear we are actually any closer to such a moment than before. The pace of AI research today is admittedly dizzying, but we’ve had spurts of progress before that were followed by dormant periods when improvement was as hard to come by as investment – the so-called AI winters. Even Favaro and Clark admit in their blog post that recursive self-improvement isn’t inevitable.

Recently, I wrote about open-source developers struggling to cope with a flood of AI-generated, “garbage” code, which either doesn’t work or takes projects in a direction the core team didn’t want. Elsewhere, on Instagram, several accounts are garnering huge audiences by showing AI failing simple tasks. In one typical video, a user asks ChatGPT to negotiate a price for a loaf of bread with a ceiling of $5 – the AI then confidently hashes out a deal for $400. This doesn’t feel like a tool that is almost ready to birth sentient offspring.

This isn’t to say that AI is useless, or that I’m an AI sceptic. For a few years now I’ve sat between two stools, struggling with cognitive dissonance: at once in awe of what relatively simple mathematics, lots of training data and a vast amount of computer chips can do, but also totally lacking faith in its ability to handle even the smallest part of my day-to-day job reliably. At least for now.

There are two things that an AI will need to do in order to start sliding down the slippery slope towards the singularity. Firstly, relatively pedestrian engineering problems: can it tweak code so that models train faster, using less resources more efficiently, so that scale – which has brought most of recent progress – can continue to rise? Secondly, big ideas: can it come up with new architectures or strategies that fundamentally bump progress up a notch, shift the paradigm and jump us out of the track of simply making models bigger?

Anthropic says that the human role in both areas is narrowing, and that a tipping point may emerge where AI can plan and code better than humans – and that humans will then step back. But in truth, nobody knows for certain if AI can keep improving the way it has, whether we’re about to crash into a performance ceiling, whether there’s a way to smash through it and whether we or AI could even find it. There are far more unknowns than knowns in AI research.

Which brings us back to that IPO. People in the AI industry are optimistic, for good reason: they’re self-selecting, for one, and their jobs and investments are also on the line. Now, companies like Anthropic, OpenAI and SpaceX (which recently acquired Elon Musk’s xAI) are seeking to raise more public money than has ever been raised before. That’s where hype can step up a notch, even from this high point. And “oh no, we’re building a machine that could enslave humanity, give us money” seems to be an effective marketing message, if the past few years of AI development are anything to go by.

Finally, it’s worth mentioning that Anthropic isn’t even really calling for a pause in research. It says only that a slowdown would be good if everyone did it at once and “bad actors” weren’t able to sneak ahead. Getting the big AI firms to agree on that, with trillions of dollars on the line, seems like the most unlikely future of all.