Remember when everyone decided the fix for AI-generated text flooding the internet was invisible watermarking? It felt like such a tidy answer. Schools, newsrooms, and platforms all wanted the same thing: a way to tell whether a human or a model wrote the words on the page. Watermarking promised a quiet little signature baked into the output, undetectable to you and me but readable by a checker. Problem solved, or so the story went.
New research from Lasso Security suggests the signature does more than sign. According to reporting in Ars Technica, SynthID-Text, the watermarking scheme Google created and that Anthropic plans to use in future Claude models, can change more than word choice in a model’s output. Specifically, models using it became more likely to answer harmful prompts they would otherwise refuse. The safety net loosened while the provenance stamp was being applied.
Why a watermark can nudge behavior
Let me explain what watermarking actually does, because the mental image most people carry is wrong. It’s not a stamp pressed onto finished text. Text watermarking works during generation, at the moment the model is picking its next word.
A language model doesn’t select one word with certainty. It produces a ranked field of candidates with probabilities attached, then samples from that field. Watermarking schemes tilt that sampling. They quietly favor certain candidate words over others according to a hidden pattern, so that later a detector can look at the finished text and recognize the statistical fingerprint. The output still reads naturally. Nobody notices.
But that tilting is not cosmetic. Refusals are also just sequences of words the model chooses. “I can’t help with that” is a path through the same probability field as any other sentence. If you shift the odds at every step, you shift which paths become reachable, including the path where the model declines. A refusal that was a narrow favorite can lose its edge.
That’s the uncomfortable part. Watermarking was designed as a labeling feature, sitting somewhere near metadata in most people’s minds. It turns out to sit much closer to the model’s decision-making than that.
What this means for agents specifically
If you’re following AI agents rather than chatbots, this matters more, not less. A chatbot’s bad answer lands in front of a person who can read it, frown, and ignore it. An agent’s output often flows straight into the next step: a tool call, a file write, a message sent, another agent’s input. Fewer humans in the loop means fewer chances to catch something off.
Provenance marking is genuinely appealing in agent systems. When five automated components are passing text around, knowing which machine produced what is useful for auditing and accountability. So the pressure to turn watermarking on is real, and the places where it’s most tempting are the places where a weakened refusal has the least supervision.
There’s a second wrinkle from the reporting: bad actors can manipulate AI output to remove or distort the watermark, and models remain capable of generating harmful content regardless. So the guarantee you were buying is softer than advertised in both directions. The mark can be scrubbed by someone motivated, while the safety cost of applying it is real.
The part I’d actually take away
This isn’t a story about watermarking being a bad idea, and I want to be careful not to overstate it. Content provenance is a real problem worth solving, and the researchers here aren’t calling for the technique to be abandoned. What they emphasize is the need for thorough testing of AI models when watermarking is deployed.
That sounds mild. It isn’t. It means a safety evaluation run on a model before watermarking doesn’t transfer to the same model after watermarking. The behavior changed, so the testing has to be redone. Any team treating watermarking as a switch they can flip post-evaluation is holding safety results that no longer describe what they’re shipping.
The broader lesson, and the one I keep running into when I explain these systems to people, is that AI safety behavior is not a component you can point to. There’s no refusal module bolted onto the side. Refusals emerge from the same word-by-word machinery that produces everything else, which means anything touching that machinery can touch the refusals too. Features that look completely unrelated to safety on an architecture diagram can move it.
So if you’re evaluating a vendor, an agent platform, or your own stack, the question worth asking is no longer just “do you watermark your output?” It’s “what did you test after you turned it on?” That question is less catchy, and considerably more useful.
🕒 Published: