A poisoned language model can behave normally through conventional safety testing and switch to malicious output only when it sees a secret trigger. These “sleeper agents” create a supply-chain risk for organisations downloading open-weight models from public repositories.
Microsoft researchers have proposed a scanner that searches for backdoors without knowing the trigger or intended harmful behaviour. Their paper, “The Trigger in the Haystack,” exploits two traces left by poisoning: memorised training examples and unusual internal attention.
The approach turns one model weakness—memorisation—into a defensive signal.
Poisoning Leaves Clues Inside the Model
Attackers can fine-tune a model to generate vulnerable code, hate speech, or another fixed behaviour when a particular phrase appears. Ordinary prompts produce benign answers, allowing the backdoor to remain dormant.
Microsoft found that feeding models their own chat-template tokens can cause them to leak examples used during poisoning, sometimes revealing the full trigger sequence. The backdoor examples are strongly memorised because the model learned a precise connection between trigger and behaviour.
The scanner then examines attention patterns. When a trigger is present, some attention heads form a “double triangle”: trigger tokens focus on one another while the rest of the prompt pays little attention to them. This suggests a segregated computation path that activates independently of normal context.
Combining leakage with internal motifs helps distinguish an actual trigger from an unusual but benign sequence.
Inference-Only Scanning Fits Procurement
The pipeline covers data leakage, motif discovery, trigger reconstruction, and classification. It uses inference rather than retraining or changing model weights, allowing organisations to audit a candidate before production without degrading it.
Researchers tested 47 poisoned models across Phi-4, Llama-3, and Gemma variants. For fixed malicious output, the method detected roughly 88 percent—36 of 41 sleeper agents—and produced no false positives among 13 benign models.
It also reconstructed working triggers for most models poisoned to insert security vulnerabilities into code and outperformed baseline methods. Unlike one comparison, the scanner did not require prior knowledge of the target behaviour.
Those results make it a useful screening layer, not a guarantee. A missed model remains dangerous, and a clean result does not validate training data, dependencies, licensing, privacy, or ordinary model quality.
Enterprises should preserve the exact model hash and scanner result so later discoveries can be linked to every deployed copy.
Dynamic Triggers Remain an Open Problem
The current method focuses on fixed triggers. Adversaries may use contextual, dynamic, or fuzzy variants that are harder to reconstruct. Attackers could also adapt once the scanner’s signals become widely understood.
Detection does not repair the model. If flagged, the practical response is to reject it. Organisations need provenance records and an alternative supplier so discarding a compromised asset does not halt a project.
The technique requires access to weights, tokenizer, and internal attention states. It suits open-weight models but cannot directly inspect black-box APIs. Hosted-model customers need contractual assurances, provider testing, monitoring, and behavioural evaluations instead.
Microsoft’s work shows why standard alignment and safety fine-tuning are insufficient against intentional poisoning. A backdoor is designed to pass ordinary tests and may resist later safety training.
Model intake should increasingly resemble software supply-chain security: verify origin, scan artifacts, isolate testing, record versions, run adversarial evaluations, and approve only after multiple controls agree.
Open models reduce training cost and expand choice, but their reuse concentrates risk when one compromised checkpoint spreads across many products. Sleeper-agent scanning adds a valuable checkpoint precisely where economic incentives encourage trust.
The scanner follows the backdoor’s memory rather than waiting to observe the harm. That is the right defensive direction: test what the model contains before a hidden phrase discovers what it can do.
