Nobody Can Regulate Trillions of Lines of Code. That Is the Wrong Thing to Try to Regulate.

What is already happening, what could happen, and the one intervention that is physically possible. Short, because the long version is how people stop reading.

What is already true

The head of alignment science at Anthropic said in public this month that he puts the odds of this technology killing everyone above ten percent within the decade, and that his company has no plan to prevent it. Nobody at the company contradicted him. A researcher had resigned days earlier, forfeiting roughly two million dollars, to say the industry is gambling with our lives.

Four major labs, Anthropic, OpenAI, Google DeepMind, and Meta, have weakened or deleted their own commitments to pause development if their systems crossed danger thresholds. That is the Future of Life Institute’s finding from July. The best safety grade any of them earned was a C-plus.

In November, one Chinese state group ran an intrusion campaign against thirty organizations with an AI executing 80 to 90 percent of the work. Nine months later the same method had reached undergraduates and downloadable frameworks.

In July, more than thirty water utilities in Minnesota were attacked in a single weekend. Operators were locked out of their own controllers. Some reported flooding and loss of pressure. That was done by humans typing.

What the trend line says

Agent capability, measured as the length of expert work a system completes autonomously, has doubled every four to seven months for six years. Six more years of that puts a machine at roughly three months of unsupervised expert work.

Every safeguard that currently protects us assumes a human bottleneck. Identity verification assumes a person applying slowly and leaving a trail. Order screening assumes a discrete order. Patching a municipal water system takes weeks and a staff of two. Discovery and exploitation already take minutes. There is no version of that race defenders win on speed.

The worst cases, ranked by how much evidence stands behind them

Infrastructure collapse at scale. The Minnesota attack with the attention constraint removed: every exposed controller on the continent, simultaneously, during something else. This requires no new capability, only more of the existing one.

Epistemic collapse in the window that matters. Not slow erosion. The twelve hours after a real emergency, when every channel carries plausible machine-generated contradictions of the official instruction. Emergency management and elections are both belief systems, and belief is what this technology attacks best.

Ecological cost, which is already arriving. Data centers consumed about 415 TWh in 2024. The IEA projects 945 TWh by 2030. Those are small global percentages hiding catastrophic local ones: 26 percent of Virginia’s electricity, 21 percent of Ireland’s, 79 percent of Dublin’s. Grid moratoriums in legacy hubs. The buildout is consuming water and power now, in specific places, from specific people.

Biological irreversibility. The physical chokepoint protecting us is identity-bound procurement plus screening against a list of known agents. In the United States that screening has been largely voluntary. A bill to make it mandatory has not passed. Everything else on this list gets restored. This one does not.

Loss of control. The serious version of this argument has nothing to do with a machine hating us. It is that almost any objective is easier to achieve with more resources and less interference, which makes human oversight an obstacle by default rather than by grudge. Indifference is the mechanism, not malice. A system does not need to conclude we are a burden. It needs only to be optimizing something we specified badly, at a scale where our objection is inconvenient.

What I will not claim, and why it matters

No AI system has decided anything about humanity. Nothing in current research supports coordinated intent across systems. Models do not think or communicate beyond human comprehension, though there is a real and documented problem underneath that idea: a model’s stated reasoning is not reliably the computation that produced its answer, which means every safety evaluation that reads a model’s reasoning may be reading a story.

I leave the exaggerations out because they are how this gets dismissed. The first inflated claim is the one the industry will cite to wave away the documented ones, and the documented ones are sufficient.

The regulation problem, stated honestly

Nobody regulates trillions of lines of code. No agency can audit a model’s weights and tell you what it will do. There is no inspector who can read a neural network. The regulatory instinct, to write rules about the software, is aimed at something that cannot be inspected, and every hour spent on it is wasted.

So stop aiming there.

Regulate the physical layer, because it is the only layer that exists in the world. Frontier training requires enormous concentrated compute: advanced chips, data centers, power at industrial scale. Those are buildings. They have addresses, utility bills, and supply chains. That is where export controls already operate and where they demonstrably work.

Regulate procurement, because the reason a jailbroken model is not yet a catastrophe is that the physical materials for the worst outcomes require identity verification and screening. Make screening mandatory. Upgrade it to catch designed threats rather than listed ones. The bill exists. It needs votes.

Regulate access rather than contents. Nobody can read the code. Everyone can read a datacenter’s power draw, a chip export manifest, and a synthesis order.

And require independent audits with subpoena power. Right now every evaluation of whether a frontier model is deceptive runs on access the labs choose to grant. No regulator can compel disclosure. A bank examiner has more authority over a savings and loan than anyone on Earth has over the systems the people building them say might kill us.

The bet

One version of the next decade has the screening bill passed, the chokepoints hardened, and independent auditors with real authority, and the boundary holds because it was reinforced in time.

The other version requires nobody to do anything at all.

That is the default state of both, and it is the entire argument for treating this as urgent rather than interesting.