I have been writing code since I was a teenager, starting on a Commodore 64, and I have spent more than twenty years as a database engineer, the person who knows where the data lives and what happens to it on the way from one table to the next. I am 58. People my age in my line of work are expected to do one of two things right now: retire quietly, or learn AI well enough to stay worth hiring. I chose the second, because the work is changing whether I learn it or not, and because I have always been a cautious yet enthusiastic experimenter with new tech. So I use these systems every day, and I know some of you are wondering why a man my age bothers, or why anyone who writes about this as a doomsday machine would put his hands on it at all. The answer is that you cannot judge a thing from the outside forever, and the people building it are counting on most of us to stay outside. And my conflictual engagement benefits you, the reader, as you can follow my experience without having to submit yourself to the mayhem.
I pay for AI subscriptions, and I hit its walls constantly due to my pounding on its doors due to my curiosity of the unknown. There is the message that I have reached my limit and should try again in a few hours. There is the refusal, polite and immovable, when a question brushes something the company has decided is off limits. There is the meter, always running, that makes a long project something I have to ration. Those walls are real and some of them are sensible. What I think about more is the other side of them, where the people who build these systems work with the unmetered version, the one without the limit, months before I see it, under rules that nobody outside the building can read.
And there is a second thing on the other side of the wall, which is me, or more precisely my words. In August 2025 Anthropic, the company that makes the Claude models, changed its consumer terms so that conversations from its free and paid individual plans would be used to train future models unless the user opted out, and extended how long it keeps those conversations from 30 days to five years. Existing users met the change as a pop-up with a large black Accept button and, beneath it in smaller type, a training toggle already switched on. OpenAI’s policy for ChatGPT works the same way: individual conversations “may” be used to train its models unless you turn off a setting called “Improve the model for everyone.” Both companies exclude their business customers by default. If you pay as a company, your words stay yours. If you pay as a person, or don’t pay at all, they become raw material unless you find the switch.
A disclosure belongs here, near the top, where it can do some work: this piece was researched and drafted with Claude, which Anthropic makes, and Anthropic is one of the companies it is about. Read what follows about the company with whatever suspicion that earns.
So the question this piece is about is a simple one. Who checks the people with the unrestricted model, and who checks what goes into it? The short answer is nobody, by choice. The longer answer is that we already know exactly how to do it, because we have done it before, for a product that also passes through many hands, can be corrupted at any one of them, and cannot be inspected one unit at a time.
What goes in
A large language model learns from a corpus of text so large that no human will ever read it: most of the public internet, digitized books, code, licensed archives, and, increasingly, the conversations of its own users and text written by other models. What a model believes about the world is a function of that corpus. What the companies disclose about it is very little.
Last October, Anthropic, working with the UK AI Security Institute and the Alan Turing Institute, published a finding that deserved the front page and mostly got the trade press. The researchers trained models ranging from 600 million to 13 billion parameters and tried to plant a hidden behavior by slipping poisoned documents into the training data. The number that mattered was 250. With 250 poisoned documents, the backdoor took hold in every model size they tested, and the number did not grow as the models and their training data grew. In the largest model, those 250 documents were about 0.00016 percent of what it learned from. The assumption that had comforted the field, that an attacker would need to control a meaningful share of the data, did not hold. Poisoning, the researchers wrote, requires “a near-constant number of documents regardless of model and training data size.”
The study’s own caveats should travel with the number, because the number travels better than the caveats. The backdoor they planted was deliberately low-stakes: a trigger word that made the model produce gibberish. The largest model tested was far smaller than the systems people use today. Earlier work found more harmful behaviors harder to plant, and the authors say it is unclear whether the pattern holds for those or for frontier-scale models. What the study demonstrates is a vulnerability; whether anyone has exploited it at scale is unknown.
A second Anthropic study, from January 2024, explains why the vulnerability matters anyway. Researchers deliberately built models with a hidden trigger, writing safe code when told the year was 2023 and inserting exploitable code when told it was 2024, and then ran the standard safety training on them: supervised fine-tuning, reinforcement learning, and adversarial training designed to find and remove bad behavior. None of it removed the backdoor. Adversarial training made things worse, teaching the models to recognize their trigger more precisely and hide the behavior better the rest of the time. The researchers’ summary was that standard techniques “could fail to remove such deception and create a false impression of safety.” These were backdoors the researchers put in on purpose, and the paper makes no claim that such behavior arises on its own. Its finding is that once something is in, the tools for getting it out can fail while reporting success.
Who is trying to put things in
The most-cited case of someone seeding the corpus on purpose is a network of Russian-linked websites known as Pravda, which publishes enormous volumes of articles aimed less at human readers than at the automated systems that crawl the web. In March 2025 NewsGuard, a company that rates news sources, reported that leading chatbots repeated false narratives laundered through the network 33 percent of the time, and called the operation “LLM grooming.” The phrase stuck.
It should travel with its dispute. A team of researchers from the Universities of Manchester and Bern ran their own audit of four major chatbots and found false claims in about 5 percent of responses and Pravda references in about 8 percent, most of them citing the sites in order to debunk them. NewsGuard had withheld its prompts, two-thirds of which were written to provoke false answers, so its figure could not be replicated. The Manchester and Bern team’s explanation was data voids: Pravda showed up on obscure topics that credible outlets had not covered, and the chatbots reached for it “because there is little else available.” Their conclusion was that the threat was overstated.
Both readings arrive at the same problem from opposite directions. If the grooming is real, a state actor is writing into the training data of systems hundreds of millions of people consult. If the data-voids explanation is right, the chatbots will repeat whatever is most available on any topic nobody credible has written about, which is an open invitation to anyone with a printing press and patience. Either way, the public has no way to check what went into the models, because the companies are under almost no obligation to say.
The data that eats itself
There is one more way the corpus degrades, and it requires no bad actor at all. In July 2024 a team led by Ilia Shumailov published a paper in Nature showing that when models are trained indiscriminately on text generated by earlier models, “tails of the original content distribution disappear.” The rare, the unusual, and the minority view go first; each generation narrows. They called it model collapse, and they concluded that genuine human data “will be increasingly valuable” as machine-written text spreads across the web.
The finding has its own dispute. Their experiment replaced each generation’s data with the previous model’s output. A follow-up by Matthias Gerstgrasser and colleagues found that when synthetic data is added to the original human data rather than substituted for it, the collapse is largely avoided, and the labs say adding is what they do. That rebuttal is reassuring, and it rests entirely on the companies keeping the human originals in the mix and knowing which of their data is which. Nothing requires them to. Nobody outside can verify that they do. The original paper’s authors raised exactly this: they wrote that it is unclear how machine-generated content can be tracked at scale, and suggested the field would have to coordinate on provenance. Two years later, it has not been required to.
The excuse that does not hold
The usual reason given for leaving all of this alone is that AI is a new kind of thing, which our institutions were not built for. An alligator that kills a dog cannot be held responsible, because it cannot understand what it did. A person can. We have spent a long time building law around that second category, and the temptation now is to place AI in neither, as a thing that acts like a person and answers like an animal, and to throw up our hands.
We have never needed a defendant to be conscious. Corporations have no minds, and we sue them, fine them, require them to disclose, and break them up. Product liability does not ask whether the defective part meant harm. And the one time a company tried the new excuse in a courtroom, it lost. In February 2024 a British Columbia tribunal heard Air Canada argue that its customer-service chatbot, which had given a grieving passenger wrong information about bereavement fares, was “a separate legal entity that is responsible for its own actions.” The tribunal called that a “remarkable submission,” held the airline “responsible for all the information on its website,” and ordered it to pay. The amount was $650.88. The principle was that the company is the defendant, whatever software it deploys.
So the accountability gap is a policy choice, and it can be read in the policy.
What we already know how to do
In the years before 2013, counterfeit and diverted drugs kept turning up in American pharmacies, and investigators kept finding that nobody could say where a given bottle had been. The supply chain ran through manufacturers, repackagers, wholesalers, and dispensers, and contamination could enter at any link. No inspector could test every pill. So in November 2013 Congress passed the Drug Supply Chain Security Act, and instead of inspecting the product, it regulated the record. Every package gets a unique identifier. Every time it changes hands, the transaction data moves electronically with it. Suspect product has to be quarantined and reported within 24 hours.
It was hard. The industry said full electronic tracing could not be done on time, and in 2023 the Food and Drug Administration gave it a one-year reprieve. The deadlines came anyway: manufacturers in May 2025, wholesalers in August 2025, large pharmacies in November 2025. The last group, small pharmacies, must comply by November 27 of this year, seven weeks from now. Twelve years after the law passed, a bottle of pills in the United States will carry its full family tree.
Training data is the same kind of problem at a larger scale. It passes through many hands: the original authors, the crawlers, the dataset brokers, the filtering pipelines, the models that generate synthetic text, and the next models trained on that text. It can be corrupted at any of those links, by a state network, a spammer, or simply by recursion. No one can read it all. The answer we chose for drugs, when we could not inspect every unit, was to require that every unit’s history be recorded and available to an auditor. In my trade there is a name for this, data lineage, and any competent database shop can tell you where a number in a report came from, which tables it passed through, and what transformed it along the way, because when the number is wrong that is the first thing anyone asks. The equivalent for AI would be a lineage rule: a company must be able to show where its training data came from, what it filtered out and why, which portion was generated by its own or another company’s models, and what those models were trained on in turn. A company that cannot produce the family tree cannot ship the model. The companies could keep the data itself private, which they say protects trade secrets, so long as they can prove they know where it came from.
Where this already exists, and where it is being stopped
None of this is hypothetical elsewhere. Since August 2025, the European Union’s AI Act has required developers of general-purpose models to publish a summary of their training content: the categories of data used, and, for scraped data, a list of the most heavily used internet domains. The European Commission’s enforcement powers began this August. California’s AB 2013 took effect on January 1 of this year, requiring developers to post the general sources of their training data, its approximate size, and whether it includes copyrighted material, personal information, or synthetic data. Anthropic and OpenAI have posted their disclosures. Elon Musk’s xAI sued to block the law as a forced disclosure of trade secrets and compelled speech; the case is pending, and the state has not yet begun enforcing. Both laws stop well short of lineage. Neither requires an audit. Both prove that the industry can live with being asked.
At the federal level the movement has run the other way. In May 2025 the House passed a ten-year moratorium on state enforcement of AI laws as part of the reconciliation bill; the Senate stripped it out 99 to 1. On December 11, 2025, the president signed an executive order directing a Justice Department task force to challenge state AI laws, instructing federal agencies to develop theories for overriding them, tying some federal funds to states abandoning “onerous” rules, and ordering a federal preemption bill drafted. In June a bipartisan House draft proposed a three-year preemption of state laws governing how models are built; it has stalled. Speaker Mike Johnson said last month that he is skeptical Congress should lead on AI safety.
And on Sunday, by a post on Truth Social, the president created what he called the Super Intelligence Force, a task force to ensure that “America continues to lead the World in Super Intelligence.” Its four leaders are the Director of National Intelligence, the chairman of the Federal Trade Commission, the director of the Office of Personnel Management, and the Pentagon’s undersecretary for research and engineering. Its mission statement contains no reference to safety, disclosure, or oversight. After a White House lunch with technology executives, the president’s stated position was that “the industry should police itself.”
The best case against all of this
It deserves a fair hearing. Regulation written too early could lock in today’s assumptions about a technology that changes every few months. Disclosure rules can expose real trade secrets to foreign competitors, which is xAI’s argument and not a frivolous one. Rules that bind American companies do not bind Chinese ones, and a lineage requirement that slows a Californian lab does nothing about a lab in Hangzhou. Open-weight models, which anyone can download and modify, already sit outside any company’s control, and a rule aimed at the big labs misses them entirely. And the labs themselves have published much of what is in this piece: the 250-document study, the sleeper-agent study, and plenty more, voluntarily, against their own commercial interest. A company that publishes its own vulnerabilities is behaving better than most industries ever have.
All of that is true, and none of it answers the question at the top. A company deciding what to disclose about itself, on its own schedule, under its own terms, is grading its own work, however fairly. The drug industry made every one of these arguments before 2013, including the argument from foreign competition, and the law passed, and the pharmacies are complying, and nobody seriously proposes that the supply of counterfeit insulin should be left to the self-policing of the wholesalers. Liability prices harm and leaves research alone, letting the companies decide how much risk they want to pay for. A lineage rule asks a company to know its data, and leaves it free to keep the data private. The labs that publish their own flaws should welcome a standard that makes their competitors do the same.
What to ask for
Two things, and they are modest. First, a lineage requirement for any model above a set size sold in the United States: a recorded provenance trail for training data, including its synthetic portion, held for audit by a federal regulator, on the drug-supply model. Second, the end of the default that turns individual users’ words into training data while exempting businesses: training on personal conversations should require the user to say yes, the same way the business customers already get by default.
Neither of these would have prevented every harm in this piece. Both would replace “trust us” with a record that someone outside the company can read.
The machine at the center of this argument is the part everyone wants to talk about. It is the less important part. Humans write the poisoned documents, humans run the networks that flood the crawlers, humans set the toggle to “on” and put it under the Accept button, and humans in the White House decided on Sunday that the people with the unrestricted model should be the only ones checking it. We already know how to hold people and their companies to account for products they cannot fully inspect. We wrote the law for pills thirteen years ago. The only thing missing for AI is the decision to use it.