What happens when AI starts self-replicating?

What happens when AI starts self-replicating?

Anthropic warns that future is soon, but the threat is less novel than it seems.

Anthropic's Frontier Red Team recently published research suggesting that small AI models may be only months away from replicating and spreading across insecure devices, creating slow but hard-to-kill autonomous botnets. AI companies are no strangers to playing up the world-ending potential of their products as a form of marketing, but in this instance, experts Engadget spoke to broadly agree small AI models could become capable of self-replicating in the near future. However, the actual threat they pose may be less novel than it appears.

"It's maybe a little earlier than I would have said. I would have predicted something more like summer of next year as the time I'd be really worried about this," says Chris Canal, the CEO of EquiStamp, a company that works with AI labs, safety organizations and governments to verify and evaluate models. "Anthropic very likely has employees with information about the frontier of research that I don't have access to. So they very well could be making better predictions." Canal attributes the recent pace of model advancement to the fact that employees at Anthropic and other frontier labs are using their own AI tools to do machine learning research.

"I think [self-replicating models] are in the future. I think they're real. I think we'll see them," says Michael Bailey, a professor at the School of Cybersecurity and Privacy at Georgia Tech. "Whether they'll take on the exact form and shape, and have the same consequences, that [Anthropic] is pitching is a completely open question."

Bailey has been studying internet worms since the early 2000s, and more recently, botnets. In 2017, he co-wrote Understanding the Mirai Botnet, which Anthropic cites as part of its own research. Mirai allowed a group of cybercriminals to create a large botnet of IoT devices they later used to carry out a DDoS attack against Sony's PlayStation Network. In the process, they ended up affecting Dyn, one of the largest DNS providers in the US. In turn, the attack broke much of the internet in 2016, knocking Amazon, GitHub, PayPal, Reddit and Twitter offline for many users across the US and parts of Europe. The financial fallout of the attack was significant, with Sony estimating it lost about $2.7 million in net revenue.

"What Mirai showed is that you could compose these botnets — and have them be useful — out of everyday devices: cable routers, IP-enabled cameras, these kinds of things," says Bailey. "We thought those devices weren't useful to an adversary, either as targets or as sources of computation for launching attacks. In fact, they're actually quite effective at doing those kinds of things."

A decade later, Anthropic argues AI worms could represent a meaningful evolution of Mirai and other hard-coded botnets like Kimwolf because they could figure out unfamiliar devices on their own. Bailey says adaptability itself isn't new. Mirai already worked across "heterogeneous" devices, from routers to IP cameras running on different chip architectures. But that adaptability was finite: each new architecture required human work, at minimum recompiling the malware. What's novel, Bailey says, is the prospect of AI automating that work, though he doesn't think Anthropic demonstrated it "in a compelling way."

As a result, Bailey suggests more capable AI models will "redistribute" who does the work of creating botnets. "It's not going to do new stuff," he says. "It's going to change the economics between attackers and defenders." While Anthropic showed that a model could hop between devices and set up inference on each, it did so in an emulated network where the "exploit" involved was trivial. Additionally, Bailey takes issue with Anthropic's self-replicating claim.

"The framing is somewhat of a misnomer, because they're doing things like sharding. Not that that's bad, but it's not like a worm or a virus. It's not full propagation," he says. Instead, Bailey argues a real-world AI botnet might work more like a dropper, a type of malware that gets a foothold on a device and installs other code. Rather than copying itself wholesale onto every machine it compromises, the botnet would install a small coordination tool that takes instructions from elsewhere in the network.

"The idea that the malware itself is self-replicating, like existing internet worms, where I'd copy an exact copy of my code to the new architecture and it would start running a new agent — that's not really what's happening here," says Bailey. He also notes worms can be polymorphic, meaning they can change both their signature and behaviors. For an AI botnet to do that, he says, it would need to retrain its own models, something Anthropic didn't attempt. The closest it came was showing that copies of a model could pass notes to their successors and get better at setting up inference over time.

As with any warning to come out of an AI lab, it's worth asking if Anthropic published this research as a form of marketing, especially with the company officially filing for an IPO so soon after. "I think what's more likely is that the companies are worried about other AI companies being reckless," Canal says. "They realize they're incentivized to make the best model and deliver it to market faster so they can have market share, and they know the other companies have the same incentives."

Regardless of whether this novel threat vector is truly a worm, and how far in the future it might actually be, Anthropic's own suggestions for how to mitigate its harms are surprisingly old school. It suggests ISPs notifying customers of an infection and then limiting their internet access to a short list of safe websites — a strategy deployed during Mirai to great effect. The company also suggests regulation as a potential solution, pointing to the UK, where easily guessable default passwords on IoT devices are now banned under the PSTI Act.

To that point, Bailey argues the appropriate response to AI worms "looks a lot more like what we've been doing in systems security for a very long time, as opposed to something particularly unique to the AI space." In fact, according to Canal, it would be difficult to add safeguards to an open-weight model that would prevent it from acting as a worm. "There are lots of ideas, but there's nothing robust so far," he says. In work with FAR.AI, an AI safety research non-profit, EquiStamp found that "basically any strategy you Google for removing safeguards from a model works."

But as Bailey points out, "even misbehaving agents are going to have a hard time if they're up against hardened devices." If there's a single takeaway from Anthropic's research, he says, it's that we consistently fail to remember all the safety lessons we've learned in the past. Security in the era of AI worms will probably look a lot like it does now. "It looks like paying attention to patching these devices, paying attention to isolation, paying attention to good passwords and good hygiene," says Bailey. Those solutions are easy to describe and notoriously difficult to implement, but if AI truly makes these sorts of attacks more prevalent we may not have a choice.