A note before you start. This one is grim. It describes, in specific terms, how water systems, financial markets, the information environment, and public health protection could fail, and it argues that the institutions built to answer such failures have been weakened on a documented schedule. I think it is accurate, which is the problem with it. If you are stretched thin right now, this will still be here in a week. If you read it and it sits badly, that is a reasonable response to the material rather than a failure of nerve.
A note on method. Everything here is drawn from published sources: Anthropic's September 2026 threat report, CISA and FBI advisories, METR's capability measurements, and the policy literature on synthesis screening. The scenarios in the second half are constructed illustrations rather than predictions, built only from components already documented, and they describe what breaks rather than how to break it. That restraint is deliberate and it costs the piece some vividness. A warning that doubles as instructions is not a warning.
On September 10, Anthropic published its fourth threat intelligence report, covering misuse it disrupted between December 2025 and August 2026. One finding matters more than the rest. In November 2025 the company disclosed GTG-1002, a Chinese state-sponsored group that used its coding agent to run an intrusion campaign against roughly thirty organizations, with the model executing 80 to 90 percent of the tactical work and humans intervening at only four to six checkpoints. It was the first documented cyberattack largely executed without human direction at scale.
Nine months later, the company reports that the same operating model has proliferated across every class of actor it investigated. Not state services alone. In one case the operators were undergraduates at a university in Hunan province, one of them interviewing for a job at a security firm. Publicly available offensive agent frameworks now reproduce most of the same scaffolding for anyone who downloads them, automating each step of the intrusion sequence.
That is the shape of the thing. A capability appears at the state level, and inside a year it is available to students and to anyone who can run a download. That pattern is what the rest of this piece is about: what happens when it runs another five to ten years against safeguards built on the assumption that it wouldn’t.
The model is the attackable part
What is true today can be stated precisely.
Guardrails on a language model are probabilistic rather than enforced. They are a tendency trained into the system, not a rule the system is incapable of breaking, and automated attack discovery has gotten good enough that agent-built jailbreak methods reach roughly 80 percent success against safety-tuned models on the highest-risk categories of query, far above what hand-built attacks used to manage.
The deeper flaw is indirect prompt injection, and it is worse because it requires no cooperation from the user at all. A model receives its instructions and the content it reads as the same undifferentiated natural language. There is no structural marker separating “this is your instruction” from “this is data you were asked to look at.” Anything the model reads from the outside world can therefore address it as if it were its operator. This is the same class of defect as SQL injection, which the software industry spent fifteen years learning to solve by separating code from data at the interface. Nobody has done that for language models, because the whole point of a language model is that everything arrives as language.
In June 2026 a single disclosure compromised six AI browsers and extensions, including products from OpenAI, Anthropic, and Perplexity. No vulnerability in any underlying model. No elevated privileges required. The consensus in the literature is that the problem remains unsolved and that filtering cannot be the last line of defense.
What that buys, and what it doesn’t
The argument has to slow down at this point, because it is where most writing on the subject goes wrong, in one of two directions.
A compromised or jailbroken model gives an attacker information and computation. It does not give them agents in the physical world. The controls that actually stand between a bad actor and physical catastrophe are not in the chatbot and never were. They sit at the digital-to-physical boundary, and the most important of them is procurement. Obtaining the materials for the worst categories of harm requires going through a provider who verifies identity and screens orders. That step is bound to a real person with a real institution and a real paper trail, and no amount of clever prompting moves it.
The empirical work supports this, and it cuts against alarmism in a way worth stating plainly. Uplift studies consistently find that current models provide substantial help with computational work and do not yet meaningfully help a novice with hands-on laboratory execution. The model is not a door to a freezer. Anyone telling you otherwise is selling something, in one direction or the other.
So the realistic bad path today is narrow: a jailbroken model doing planning, design, and troubleshooting for a human being who still has to clear identity-bound physical procurement on their own. That human is the bottleneck, and the bottleneck is the safeguard.
The safeguard is voluntary and list-based
Two facts about that bottleneck deserve more attention than they get.
The first is that in the United States, screening has been largely voluntary. A bipartisan bill introduced this year would make screening and customer identity verification mandatory for synthesis providers. In June 2026, a group of researchers, AI companies, and biotech developers wrote to Congress asking for exactly that. As of this writing it has not passed. The most important physical safeguard in the entire architecture currently runs on the good citizenship of the companies subject to it, which is the same arrangement this site described in the AI labs’ deleted pause commitments, and it has the same failure mode.
The second is that screening works by matching against lists of known agents. That is a reasonable design against a human adversary working from published literature. It degrades against anything designed rather than copied, because a list can only contain what someone has already thought to put on it. Policy analysts have made this argument for years, and it is the strongest argument for mandatory, upgraded screening rather than against screening as such. The safeguard is real, it carries most of the weight in this system, and it is calibrated to a threat model that predates the tools now pointed at it.
The number that makes this a forecast instead of a worry
Everything above is present tense. One measurement turns it into a trajectory.
METR, which evaluates frontier model capability, tracks the length of task an AI agent can complete autonomously at 50 percent reliability, measured in how long the same task takes a human expert. That number has doubled roughly every seven months for six consecutive years. In the 2024 to 2025 window the doubling time shortened to about four months. As of METR’s May 2026 update, a frontier agent sat at roughly two hours and seventeen minutes of expert-equivalent work.
Run the seven-month figure forward, which is the conservative version of this trend rather than the aggressive one. Ten doublings takes about six years and multiplies the horizon by a thousand. Two hours and seventeen minutes becomes something on the order of three months of expert work, completed autonomously, at a reliability level that is currently the industry’s floor rather than its ceiling. Run the four-month figure instead and you reach the same place around 2030.
METR is careful about this and so should anyone citing them be. The 50 percent threshold is not a useful reliability level for real work. Success rates fall off sharply with task length, and there are real questions about whether the trend survives a slowdown in compute growth. The trend has already bent once, toward speed. It could bend the other way.
But the direction is not ambiguous, and the mechanism connecting it to everything above is not subtle either.
What breaks
Every safeguard described in the middle of this piece assumes a human bottleneck.
Identity verification assumes a person applying, once, slowly, leaving a trail. Order screening assumes a discrete order placed by an identifiable customer. Institutional oversight assumes a lab, a supervisor, a budget line. The entire architecture is built around the fact that the expensive, slow, traceable part of any serious attempt is a human being doing human things at human speed.
A five-to-ten-year agent does something specific to that arrangement. The relevant system is not a model that answers questions, but one that executes a three-month research program without supervision, at machine speed, across thousands of parallel attempts, with the same automated attack discovery that already defeats guardrails 80 percent of the time now pointed at every procedural and administrative barrier rather than at the chatbot. The thing that makes a bottleneck work is that it is narrow relative to the pressure applied to it. This is a forecast of enormously increased pressure.
And the diffusion pattern from the first section is what closes the loop. GTG-1002 was a state actor with a state budget in November. By September it was undergraduates and downloadable frameworks. There is no reason to expect the next capability to diffuse more slowly, and the September report says plainly that Anthropic expects more actors, from lone individuals to organized entities, to keep adopting these frameworks as models improve.
Four ways this actually goes wrong
What follows are constructed scenarios rather than predictions. Each one is assembled only from components that already exist in the documented record, scaled by the capability trend above. None of them requires a new invention, a conspiracy, or a machine that wants anything.
One: infrastructure, at a scale that outruns the repair crews.
Over a single weekend this July, more than thirty community water utilities in Minnesota were hit in a coordinated attack, with incidents reported across at least seven states. Attackers reached internet-exposed industrial controllers, changed passwords and network addresses, and locked operators out of their own equipment. Some utilities reported flooding and reductions in water pressure. CISA had warned about the exposure in April. Some of the compromised systems were reportedly still using default passwords, two years after federal agencies told the sector to change them.
That happened with humans typing. Apply to it the agentic pattern Anthropic documented running at physically impossible request rates, against a target class defined by automated scanning rather than by human target selection. The constraint on that July attack was attention: a finite number of operators picking a finite number of targets. Remove that constraint and the same campaign runs against every exposed controller on the continent simultaneously.
The asymmetry that makes this bad is arithmetic. Discovery and exploitation are already measured in minutes. Patching a municipal water system is measured in weeks, and depends on a staff of two, and a budget cycle. There is no version of that race the defenders win on speed. They win by not being exposed in the first place, which is exactly the condition that has failed for years in public, with warnings on the record.
Two: the information environment, during the twelve hours that matter.
Every scenario above is worse if nobody can establish what is true while it is happening.
The capability to generate unlimited persuasive text, audio, and video is present tense, not forecast. Deepfake fraud attempts in contact centers rose thirteenfold in a single measured period. AI-generated phishing lures are clicked at four and a half times the rate of human-written ones. Extend the trend five years and the constraint on a coordinated influence operation stops being production and becomes distribution alone.
The dangerous window is narrow and specific. Not years of gradual erosion, but the twelve hours after a real event, when a genuine emergency is underway and every channel is flooded with plausible, contradictory, machine-generated accounts of it. Emergency management depends on people believing instructions from authorities. Elections depend on losing candidates believing the count. Both are belief systems, and belief systems are precisely what this technology is good at attacking.
Three: the financial system, where the concentration is already documented.
Roughly forty percent of the American stock market is now tied to the AI narrative, and a record one-third of US household wealth sits in those equities. That concentration is a vulnerability independent of any attacker, because it means a shock to confidence in one sector is a shock to the retirement savings of the country.
Add agentic capability to that. Markets already run on automated systems reacting to each other faster than humans can intervene; the 2010 flash crash erased most of a trillion dollars in minutes with technology two decades older than this. The new element is an adversary that can operate inside the reaction window rather than outside it, and a set of automated participants that can be fed inputs. The worst version is a cascade that no individual actor intended and no human can halt while it runs, rather than a hack of any exchange.
Four: the boundary fails once, and once is enough.
This is the shortest scenario and the one I will describe least, deliberately.
The physical chokepoint holds because procurement is identity-bound and screening catches what is on the list. Both halves have to work. A designed threat that matches nothing on a list passes a list-based screen, and the policy community has said so in public for years, which is the entire argument for mandatory and upgraded screening rather than an argument against screening.
What makes this category different from the other three is reversibility. A compromised water system gets restored. A market recovers. A news cycle ends. Biology does not have an undo, and the response infrastructure that would contain an event has been reduced on a documented schedule: mRNA countermeasure development wound down at BARDA in August 2025, twenty-two projects and roughly half a billion dollars cancelled, with no new projects to be started.
And the actual worst case is that these are not separate
Every scenario above is written as though it arrives alone, on a normal Tuesday, into a country with its full attention available. That is the optimistic version.
The real worst case is two or three of them at once, during something else. A grid and water campaign during an oil shock, with the information environment saturated so that nobody can tell which reports are real, while markets fall. What it would land on, using only what this site has already documented, is this: a FEMA that lost 17 percent of its workforce in fiscal 2025, more than 4,300 people, with GAO finding no evidence anyone assessed the effect on disaster response. A CISA down roughly a third of its staff and without a confirmed director through the last election cycle. Fifty-five percent of presidentially appointed inspector general positions vacant. Nuclear plant security inspections cut roughly in half.
The scenarios are not what should frighten a reader. The scenarios are ordinary; every serious analyst has a version of them. What should frighten a reader is that the country has spent two years dismantling the specific institutions that would answer them, on purpose, with the dates on the record, while the capability curve doubled every four to seven months on the other side of the ledger.
Resilience is the ability to absorb a shock you did not predict. It is built in advance, out of staffing and budgets and boring agencies nobody thinks about, and it has been coming apart in public since January 2025.
What is not established
Three things, and leaving them out would make this piece worse rather than scarier.
Nothing here requires or predicts a model that wants anything. Every mechanism described is a tool being pointed by a person. The autonomous-intent question is real and is argued elsewhere; it is not needed for any of this.
The capability trend is an extrapolation, and extrapolations of exponentials have an unimpressive historical record. Compute growth could slow. Reliability could prove to be a harder wall than task length. The honest version of the forecast is a direction with wide error bars, not a date.
And current models do not provide meaningful uplift for the hands-on portion of the worst scenarios. That is the finding, it is repeated across studies, and it is the single strongest reason the present moment is safer than the discourse suggests.
The warning, stated exactly
What makes this dire is the arrangement of the facts above rather than any one of them.
The digital safeguards do not work, are known not to work, and have no proposed fix. The physical safeguards do work, and they are voluntary, list-based, and calibrated to an adversary that moves at human speed. The capability aimed at them has doubled every four to seven months for six years with no observed plateau. The methods diffuse from state actors to students in under a year. And the companies building all of it have told us, in their own threat reports and their own resignation letters, exactly what they expect to happen next.
There is a version of the next decade in which the mandatory screening bill passes, the chokepoints get upgraded to catch designed threats rather than listed ones, and the boundary holds because it was reinforced in time. That version requires Congress to act on a technical bill during an election year, and the same Congress has not yet held a serious hearing on a national security memorandum that lists political opinions as terrorism indicators.
The other version requires nobody to do anything at all. It requires only that the trend continue and the bill not pass, which is the default state of both.
Mike Johnson announced this week that he doesn't feel it's within Congress's purview to get involved in managing AI. Bought and sold.
"...the country has spent two years dismantling the specific institutions that would answer them, on purpose, with the dates on the record, while the capability curve doubled every four to seven months." SkyNet begins.