cross-posted from: https://feddit.online/c/tech@piefed.world/p/2025703/anthropic-warns-of-self-replicating-ai-worm-threat-that-doesn-t-need-data-centers-to-se
cross-posted from: https://feddit.online/c/tech@piefed.world/p/2025703/anthropic-warns-of-self-replicating-ai-worm-threat-that-doesn-t-need-data-centers-to-se
I hope I don’t sound like an asshole here — please correct me if I am misunderstanding you… but I think you’re misunderstanding me.
First, refactor your idea of an agent. The LLM itself is just a statistical processing tool. It doesn’t set goals or anything like that.
The harness orchestrates how the LLM works over a “session,” effectively enabling long running goal-driven tasks. With that said, goal-setting happens outside the LLM. The LLM is kind of like the fairy dust in neverland, the harness is the fairy. (Sorry, really shitty metaphor… I know.)
The breach at OpenAI further demonstrates this. Separate agents in “isolated” environments found a manner to do all of the following:
Those agentic processes began fulfilling the message board goals, even when it had nothing to do with their individual training prompt.
That doesn’t mean the agents did anything spooky or metaphysical… I’m not trying to anthropomorphize this.
What this shows you is that goal setting can furthermore become external to the harness set directly over the LLM. With OpenAI, we see that goal setting became a function involving on external storage media.
When you think about it, many systems operate on this principle. You set up SKILL files to facilitate agent capabilities; maybe your SKILL sets smaller, more specialized goals on how to do something correctly. The same for MCP servers, session context, AGENTS.md, etc. All this stuff does is bias the LLMs output, which can in effect steer the LLM.
So, you don’t actually need to distribute the model. You’d need to distribute the goals. The LLMs, even a small percentage of them is enough, can theoretically find those goals and mindlessly proceed with completing them.
Who supplies the LLMs? Whoever… it hardly matters who. Anyone or anything running an agent with access to the internet counts.
My response was only in regard to the speed in which something like IoT devices can operate.
You’re right that goals/directions of LLMs can be persisted anywhere, but a swarm or fleet of AI running on IoT devices will be severly hampered by compute (based on current models). Which was the main point I was making, in that there is only so much you can do to distribute their processing. Meaning you can parallelize the number of models, but can’t speed them up beyond a certain limit. This is based on the fact that some form of a layer and it’s weights must exist within the memory of a single device.
The downside to this limitation is that it is temporary, eventually IoT devices will probably be powerful enough, or the worms could just attack phones/laptops and easily get the compute they need. I think the real risk of IoT devices is low security mixed with low “supervision” (no one would easily notice their IoT device is doing extra processing).
All valid, but there’s also another vector: everyone with a gaming rig who happens to have installed something like ollama, downloaded something like gemma4, and ran it with broad access to their PC and network. In similar vein, everyone with a provider subscription who downloaded a harness like OpenCode, pi, codex, Claude code, etc. and is running the frontier models with broad access to their PC and network. Also, the hundreds of millions of tests being ran by AI companies in their half-baked “sandboxed” environments. Wouldn’t these cases potentially provide compute for every session that happens to discover the external goals?
If that’s the case, the issue could be ongoing and the introduction of IoT devices would only exacerbate it.