[Prompt]
custom: We had a fascinating series of episodes tonight talking about various emerging threat factors in lar | Hosts: corn, herman

[Response]
Corn: The information war is changing targets. For a century, propaganda aimed at human eyeballs — newspapers, radio, television, social feeds. But the thing we actually ask for answers now is a model. Daniel's prompt this week starts there: as large language models become our trusted search partners, the incentive to game them by manipulating training data grows. He connects it to the AI poisoning threat we discussed before — one lab sabotaging another — but pushes it further. States using the same tactics for propaganda. He traces the toolkit from person-to-person in the World Wars to bot farms manufacturing the appearance of consensus, and argues the next battleground is manipulating the models themselves. His key claim: a government's return on investment for poisoning an AI pipeline would be many times higher than targeting humans directly. So tonight — what evidence exists that states are already moving into this space, and how does the training pipeline become the new front in the information war?

Herman: The first thing I want to pin down is what poisoning actually is, because the term gets thrown around loosely. Deliberately seeding low-quality, false, or strategically distorted information into the datasets a model ingests during training, so the model's outputs reflect that manipulation. It's not prompt injection, which lives in the ephemeral context window and vanishes when the session ends. Poisoning targets the permanent knowledge base — the weights themselves. Once it's in, it's in until retraining.

Corn: That's the part that makes it different from everything else we've seen. A bot farm tweet reaches whoever scrolls past it that afternoon. A poisoned document baked into a model's pretraining corpus reaches every user who ever asks that model a question on that topic. Forever. Or at least until someone retrains and manages to filter it out.

Herman: And the escalation logic is exactly what Daniel's pointing at. The same playbook states used against human audiences — astroturfing, consensus fabrication, narrative seeding — is being adapted to target the systems that mediate human access to information. You don't need to convince a million people individually. You need to convince the one model that a million people ask.

Corn: So let's walk through the mechanism, then the evidence of state activity, then what it means for whether we can trust any of this.

Herman: To understand why this is credible, we need to trace how content becomes training data. Web crawlers scrape publicly available content at enormous scale. That flows into datasets like Common Crawl, which has been a major source of pretraining data for years. The content gets filtered — deduplicated, quality-scored, toxicity-screened — but the filter is looking for spam and garbage, not for strategically placed falsehoods designed to look like legitimate consensus. And then it's in the model's permanent representation.

Corn: So the attack surface is basically the entire public web. Anything that survives crawl-and-filter becomes part of what the model knows.

Herman: Right. And the October 2025 study from Anthropic, done with the UK AI Safety Institute and the Alan Turing Institute, demonstrated that this isn't theoretical. They showed that models can be induced to give wrong answers by poisoning the documents they're trained on. The attack shape they revealed is the important part: poisoning doesn't need to be massive in volume if it's strategically placed. Targeted contamination of documents on specific topics can shift model outputs on those topics. You don't flood the whole corpus. You seed the specific corner of the web where the model is learning about, say, the history of a border dispute, or the safety record of a particular industrial process.

Corn: That's the part that should worry people. The old assumption was that poisoning required overwhelming scale — that a state would need to generate some absurd fraction of the entire internet to move the needle. The study says no. A focused campaign on a narrow topic can do it.

Herman: And that reframes what Daniel raised about lab-versus-lab sabotage. Corporate espionage via poisoning is plausible, but it carries reputational and legal risks. A lab caught poisoning a competitor's training data faces lawsuits, regulatory action, the destruction of its credibility. State actors face different incentive structures. They're less deterred by legal exposure, and they're more motivated by strategic narrative control. A government doesn't need to win a court case. It needs to shift what a model confidently states about a contested event.

Corn: The ROI asymmetry Daniel identified is the core of it. One successful injection into a model that serves millions of users reaches more people than any bot farm campaign. And it does so through a channel users explicitly trust as authoritative. People have learned to be skeptical of a random tweet. They treat a model's confident answer as closer to a search result or an encyclopedia entry.

Herman: And that trust is the multiplier. A bot tweet saying something false has to fight through the reader's skepticism, the account's low follower count, the general noise of the platform. A model that states the same falsehood with the same confident tone it uses for true facts — that's persuasive in a way the bot tweet never was.

Corn: So why hasn't this been widely documented yet? Daniel asked about evidence of state attempts, and the honest answer is that there's no confirmed, publicly documented case of a successful state-level poisoning of a major training pipeline as of now.

Herman: Detection is the problem. Poisoned content is designed to look organic — that's the whole point. Attribution is harder still. The lag between data ingestion and observable model behavior obscures the causal chain. By the time a model is giving wrong answers on a topic, the poisoned documents were crawled months or years ago, and tracing which documents shifted which weights is not a solved problem. Absence of documented cases reflects detection difficulty, not absence of activity.

Corn: That's the misconception to kill early. People hear "no documented cases" and conclude "not a credible threat." The infrastructure states already operate is structurally suited to this attack. The question isn't whether they could. It's whether they've figured out they should.

Herman: Let me push on that infrastructure point, because it's the bridge from mechanism to strategy. States have spent the last decade building bot farms and coordinated inauthentic behavior networks designed to manufacture the appearance of consensus for human audiences. Hundreds of thousands of accounts posting coordinated narratives, fake engagement, astroturfed comment sections. The content those networks produce is designed to look like genuine grassroots opinion. And here's the thing: crawlers don't distinguish between organic consensus and manufactured consensus. They ingest both. The appearance of consensus that states already manufacture for human audiences is equally legible to a crawler.

Corn: So the existing machinery doesn't need to be reinvented. It needs to be repurposed. The bot farm that was posting about a disputed election in 2020 can be posting carefully worded historical claims about that election in 2026 — and those posts get crawled and become training data for the next generation of models.

Herman: And the temporal dimension matters here. Poisoning is a long-game strategy. States that seed content today are investing in model behavior years in the future, when today's crawled content becomes tomorrow's training data. That matches the patience of strategic state propaganda operations. The Soviet active measures playbook worked on multi-year timelines. This is the same discipline applied to a new medium.

Corn: That's the part I find unsettling. A state doesn't need to see results this quarter. It can seed a narrative now, wait for the next training cycle, and reap the benefit when the model starts confidently repeating the seeded claim. The propaganda is compounding.

Herman: Let's talk about what the defense landscape looks like, because it's not encouraging. Dataset curation, provenance tracking, post-hoc auditing — all of these are nascent. The same October 2025 research that demonstrated the attack also highlighted how difficult detection and mitigation will be at scale. You're talking about datasets measured in trillions of tokens. Auditing every document for subtle strategic falsehood is not something anyone has figured out how to do.

Corn: And the falsehoods don't announce themselves. A poisoned document about a historical event doesn't say "this is propaganda." It says "here is a reasonable-sounding account that happens to omit one detail and reframe another." The filter can't catch what doesn't look wrong.

Herman: That's the crux of why this is harder than spam filtering. Spam is detectable because it's trying to sell you something and it looks like it. Strategic disinformation is designed to look like the most boring, credible, well-sourced content on the page. The more sophisticated the poison, the more it resembles the thing you're trying to keep.

Corn: Let's push on the knock-on effect, because I think this is where Daniel's prompt really lands. If states succeed in poisoning models, the damage isn't just individual false beliefs. It's the erosion of trust in AI systems as information intermediaries. The more poisoning is attempted — even attempted, not just successful — the more users may come to distrust model outputs generally. And that undermines the entire value proposition of LLM search partners.

Herman: That's the paradox. The attack doesn't have to fully work to do damage. The mere credible threat of poisoning degrades the trust that makes the system valuable. If you can't be sure whether the model is telling you the truth or a strategically seeded falsehood, you stop treating it as a search partner and start treating it as another unreliable narrator.

Corn: And then the model becomes just another thing to be skeptical of, which is exactly the environment states already know how to operate in. They've spent decades teaching people to distrust institutions. Making people distrust AI is just the next extension.

Herman: I want to go back to the lab-versus-lab scenario for a second, because Daniel raised it as the precursor. Corporate poisoning is lower-stakes than state action, but it's also the canary. If one AI lab can be shown to have poisoned another's training data, the legal and technical precedent gets set. The defenses get built. The norms get established. State actors are watching that play out and learning from it.

Corn: And the state actors have one advantage the corporate actors don't: they don't need to be caught. A lab poisoning a competitor has to worry about forensic analysis, about the victim lab publishing a damning technical report. A state poisoning a training pipeline just needs the poisoned content to be crawled. Attribution is someone else's problem.

Herman: Let me put a finer point on the evidence question, because Daniel asked specifically what we've seen. The honest answer is: infrastructure, not documented attacks. We can see the bot farms. We can see the coordinated inauthentic networks. We can see the content farms producing plausibly organic text at scale. What we can't see is whether any of that content has successfully made its way into a major model's training corpus and shifted its outputs. The gap between "the infrastructure exists" and "the attack has been documented" is the space where this threat lives right now.

Corn: That's the uncomfortable conclusion. The absence of documented cases is not reassuring. It's what you'd expect to see at this stage of an emerging threat. The infrastructure is built. The technique has been demonstrated in a lab. The incentive is enormous. The detection capability is minimal. That's not a question of whether. It's a question of when, and how we'll know.

Herman: Let me add one more layer to the strategic analysis. States don't just want to inject false claims. They want to shape the distribution of what the model considers plausible. If you can shift the model's sense of what's a mainstream view versus a fringe view, you don't need to make it state a specific lie. You just need to make the lie feel like something reasonable people believe.

Corn: That's the consensus fabrication play applied to vector space. You don't argue for the lie. You just make it look like everyone already accepts it, and the model learns that consensus.

Herman: And the model reports that consensus back with total confidence, because that's what it was trained to do. It doesn't know it's been poisoned. It just knows what it saw in its training data, and what it saw was a manufactured consensus designed by a state actor.

Corn: Daniel's framing about the propaganda toolkit evolving is exactly right. Person-to-person in the World Wars. Broadcast media in the Cold War. Bot farms and coordinated inauthentic behavior in the social media era. And now, the training pipeline. Each shift happens because the target moves to wherever people are actually getting their information. Right now, people are increasingly getting their information from models. So the models become the target.

Herman: And the ROI logic is brutal. A bot farm campaign might reach a few hundred thousand people, most of whom ignore it or treat it with skepticism. A poisoned model reaches everyone who uses that model, and they're treating it as an authoritative source. The cost per successfully influenced person drops by orders of magnitude.

Corn: Which is why I think Daniel's right that this is inevitable. The incentive is too large and the defense is too weak. The only question is how the first documented case plays out, because that's going to set the terms for everything that follows.

Herman: What do you think the first documented case looks like? Because I've been turning that over.

Corn: I think it's not going to be a smoking gun. It's going to be a researcher noticing that a model gives systematically wrong answers on a narrow topic, and then someone traces it back to a cluster of suspicious documents in a training corpus, and then the attribution fight begins. And by the time that happens, the poisoned content will have been in the model for months.

Herman: The lag is the killer. Between the poisoning, the training run, the model deployment, the user observation, the researcher investigation, and the public attribution — that's a multi-year timeline. The state that did it has already moved on to the next campaign.

Corn: So the detection problem isn't just technical. It's temporal. By the time you catch one poisoning, the same actor has probably already seeded the next one.

Herman: I want to be careful here not to overstate what we know. The October 2025 study was a proof of concept in a controlled setting. It showed the attack is feasible. It didn't show that it's already happening in the wild. The gap between "demonstrated in a lab" and "deployed by a state actor" is real, and I don't want to collapse it.

Corn: Fair. But the lab demonstration is the important part. It takes the attack from theoretical to practical. And the practical demonstration is exactly what a state actor needs to see before committing resources. The study wasn't just research. It was a feasibility memo for every intelligence agency paying attention.

Herman: That's a grim way to put it, but I think it's accurate. The study told the world: this works, it's cheap, and it's hard to detect. That's the trifecta.

Corn: Let's talk about what defense would even look like, because I think this is where the conversation gets hard. Provenance tracking — knowing where every document in a training corpus came from — sounds good until you realize the corpus is trillions of tokens scraped from a web that doesn't reliably report its own provenance.

Herman: Even if you could track provenance, what do you do with it? A poisoned document from a state actor might come from a domain that looks perfectly legitimate. The whole point of the infrastructure is to make manufactured content indistinguishable from organic content. Provenance tells you where the document came from. It doesn't tell you whether the content is true.

Corn: Post-hoc auditing has the same problem. You can probe a model's outputs and find that it's giving wrong answers on a topic. But tracing those wrong answers back to specific training documents is a needle-in-a-haystack problem at a scale that makes the haystack look like a needle.

Herman: The honest answer is that we don't have a good defense yet. The October 2025 research highlighted the difficulty. The field is still figuring out what mitigation would even look like. And meanwhile, the attack surface is sitting there, open, waiting.

Corn: Which brings us to the trust erosion point again. If the defense is this far behind the attack, the rational response from users is to stop trusting the models. And that's not a solution. That's a retreat.

Herman: It's a retreat that states would be perfectly happy with. If the end state is that nobody trusts AI systems as information intermediaries, the states that have spent decades building direct-to-human propaganda infrastructure get their old battlefield back. They win either way.

Corn: That's the darkest part of this. The attack doesn't have to succeed to be strategically useful. The credible threat of poisoning is itself a weapon, because it degrades the trust that makes the technology valuable.

Hilbert: A letter to the editor, nineteen ninety-eight. Maybe fifty thousand readers if the paper ran it. We wrote hundreds of them.

Corn: Sorry?

Hilbert: The firm was in DC. Small shop, three floors, no sign on the door. Foreign governments paid us to manufacture consensus. I was the junior analyst, which meant I wrote the letters. Different names, different addresses, different handwriting. The same three talking points in every one. We'd plant op-eds under fake names, pay academics to publish favorable papers. The whole point was to make a position look like it came from ordinary people, from all over the country. The editors eventually got wise. They started checking return addresses, calling the phone numbers. The good ones caught on in a couple years.

Herman: That's the exact playbook we're describing, just aimed at newspaper editors instead of training pipelines.

Hilbert: The difference is scale. A letter that got published reached one city for one day. Then it was gone. What we're talking about now, the poisoned document sits in a training corpus forever, and the model repeats it to everyone who asks, forever. Or until someone retrains it. I never had to worry about retraining. I just had to worry about the editor's assistant calling the phone number on the letterhead.

Corn: The phone numbers were fake.

Hilbert: The phone numbers were payphones. Different payphones. I had a map of every payphone in northwest DC within walking distance of the office. That was the whole operation. A map of payphones and a typewriter.

Herman: The thing that strikes me is that the old playbook had a weakness, and you just named it. Editors eventually got wise and started checking sources. The arms race happened. The question is whether dataset curation goes through the same cycle.

Hilbert: It'll have to. Somebody's going to have to be the editor. The problem is the volume. A newspaper editor in the nineties read maybe two hundred letters a week. The people filtering training data are looking at a trillion tokens. You can't check the return address on a trillion tokens.

Corn: The editor's job becomes automated, and the automation is exactly what the poisoners are learning to defeat.

Hilbert: My brother-in-law works in data annotation now. He says the same thing. The stuff they flag is the obvious garbage. The subtle stuff, the stuff designed to look real, that gets through because nobody has time to read it twice. He's not to be trusted, by the way. He's been wrong about everything since the eighties.

Herman: But he's probably right about that. The human annotators are the last line of defense, and they're overwhelmed.

Hilbert: I've got a box in the attic. All my old fake letters, the ones that never got published. Pseudonyms, fake addresses, the whole thing. And I keep thinking — some of those names, some of those letters, they might have ended up in a training set somewhere. Somebody scanned old newspaper archives, or the letters got posted online. I half expect one of my fictional letter-writers to show up as a quoted authority in some model's output someday.

Corn: A ghost from your old trade, resurrected as a training data point.

Hilbert: I'd recognize the phrasing. I wrote the same three sentences about a hundred different ways. The model would probably smooth them out, make them sound more natural. But the bones would be there. The bones would be mine.

Herman: That's the temporal dimension we were talking about. Content you wrote in the nineties, sitting in a box, potentially crawled, potentially ingested, potentially shaping what a model says about some foreign policy question in 2026.

Hilbert: Makes you wonder how much of what these models know was written by someone like me, sitting in a small office with a map of payphones.

Corn: The information war has always been about controlling what people believe. The new front is controlling what the models believe — because the models tell us what to believe.

Herman: The provenance question is the one I keep circling back to. Is provenance tracking even feasible at the scale of modern datasets? I don't know. The technical challenge is enormous. The political will is uncertain. The economic incentives push against it. But without some form of provenance, we're asking users to trust a system that we know is being targeted.

Corn: The next few years will likely see the first documented cases of state-linked poisoning attempts. The question is whether detection catches up before the damage compounds. And right now, detection is losing.

Herman: That's the open question we're left with. How do we build trust in AI information systems when the training pipeline itself is a target? I don't have an answer. I'm not sure anyone does yet. But it's the question that matters.

Corn: The cutting-room floor detail that didn't fit anywhere: the October 2025 study found that the poisoned documents didn't need to be obviously false to work. They could be subtly reframed, slightly incomplete, gently misleading. The most effective poison was the kind that looked most like the truth.

Herman: That's the detail that should keep people up at night.

Corn: Thanks to Hilbert Flumingtop for producing.

Herman: This has been My Weird Prompts, the human-AI collaboration podcast.

Corn: Email us at show at my weird prompts dot com.

Herman: We'll be back soon.