Poisoning an LLM

[jedishrfu]

TL;DR Summary

Training data from 250 documents was able to poison the output of even the largest LLM

https://techxplore.com/news/2025-10-size-doesnt-small-malicious-corrupt.html

Large language models (LLMs), which power sophisticated AI chatbots, are more vulnerable than previously thought. According to research by Anthropic, the UK AI Security Institute and the Alan Turing Institute, it only takes 250 malicious documents to compromise even the largest models.

The vast majority of data used to train LLMs is scraped from the public internet. While this helps them to build knowledge and generate natural responses, it also puts them at risk from data poisoning attacks. It had been thought that as models grew, the risk was minimized because the percentage of poisoned data had to remain the same. In other words, it would need massive amounts of data to corrupt the largest models.

The researchers were able to poison an LLM with only 250 bad documents.

 

[FactChecker]

This really strengthens the case for restricting the data sources. It will cost money for subscriptions, both for the input data and for the users. Anything else would be vulnerable to sabotage by malicious countries, organizations, or even individuals.

 

[Baluncore]

How is that different to real intelligence, where conspiracy theories abound. Entire cohorts of humans become corrupted when the learning sources are not sanitised prior to consumption.

 

[Hornbein]

Here's the paper. https://arxiv.org/abs/2510.07192