How poisoned data can trick AI − and how to stop it
- Data poisoning refers to the intentional feeding of wrong or misleading data into an automated system, which can lead to dangerous outcomes such as backdoor attacks, data leaks, and espionage.
- Data poisoning can be used to disrupt critical infrastructure, such as transportation systems, by tampering with training data or collecting data from cameras and sensors.
- Examples of data poisoning include the Microsoft Tay chatbot, which was disabled after being fed malicious comments, and the potential for attackers to use red laser pointers to trick cameras in train stations.
- To defend against data poisoning, researchers are exploring decentralized approaches such as federated learning and blockchain, which can provide secure and transparent records of how data is shared and verified.
- Using defense tools like federated learning and blockchain can help build more resilient AI systems that can detect when they’re being deceived and alert system administrators to intervene, but ultimately, AI systems will always be vulnerable to manipulation from malicious actors.
Imagine a busy train station. Cameras monitor everything, from how clean the platforms are to whether a docking bay is empty or occupied. These cameras feed into an AI system that helps manage station operations and sends signals to incoming trains, letting them know when they can enter the station.
The quality of the information that the AI offers depends on the quality of the data it learns from. If everything is happening as it should, the systems in the station will provide adequate service.
But if someone tries to interfere with those systems by tampering with their training data – either the initial data used to build the system or data the system collects as it’s operating to improve – trouble could ensue.
An attacker could use a red laser to trick the cameras that determine when a train is coming. Each time the laser flashes, the system incorrectly labels the docking bay as “occupied,” because the laser resembles a brake light on a train. Before long, the AI might interpret this as a valid signal and begin to respond accordingly, delaying other incoming trains on the false rationale that all tracks are occupied. An attack like this related to the status of train tracks could even have fatal consequences.
We are computer scientists who study machine learning, and we research how to defend against this type of attack.
Data poisoning explained
This scenario, where attackers intentionally feed wrong or misleading data into an automated system, is known as data poisoning. Over time, the AI begins to learn the wrong patterns, leading it to take actions based on bad data. This can lead to dangerous outcomes.
In the train station example, suppose a sophisticated attacker wants to disrupt public transportation while also gathering intelligence. For 30 days, they use a red laser to trick the cameras. Left undetected, such attacks can slowly corrupt an entire system, opening the way for worse outcomes such as backdoor attacks into secure systems, data leaks and even espionage. While data poisoning in physical infrastructure is rare, it is already a significant concern in online systems, especially those powered by large language models trained on social media and web content.
A famous example of data poisoning in the field of computer science came in 2016, when Microsoft debuted a chatbot known as Tay. Within hours of its public release, malicious users online began feeding the bot reams of inappropriate comments. Tay soon began parroting the same inappropriate terms as users on X (then Twitter), and horrifying millions of onlookers. Within 24 hours, Microsoft had disabled the tool and issued a public apology soon after.
The social media data poisoning of the Microsoft Tay model underlines the vast distance that lies between artificial and actual human intelligence. It also highlights the degree to which data poisoning can make or break a technology and its intended use.
Data poisoning might not be entirely preventable. But there are commonsense measures that can help guard against it, such as placing limits on data processing volume and vetting data inputs against a strict checklist to keep control of the training process. Mechanisms that can help to detect poisonous attacks before they become too powerful are also critical for reducing their effects.
Fighting back with the blockchain
At Florida International University’s solid lab, we are working to defend against data poisoning attacks by focusing on decentralized approaches to building technology. One such approach, known as federated learning, allows AI models to learn from decentralized data sources without collecting raw data in one place. Centralized systems have a single point of failure vulnerability, but decentralized ones cannot be brought down by way of a single target.
Federated learning offers a valuable layer of protection, because poisoned data from one device doesn’t immediately affect the model as a whole. However, damage can still occur if the process the model uses to aggregate data is compromised.
This is where another more popular potential solution – blockchain – comes into play. A blockchain is a shared, unalterable digital ledger for recording transactions and tracking assets. Blockchains provide secure and transparent records of how data and updates to AI models are shared and verified.
By using automated consensus mechanisms, AI systems with blockchain-protected training can validate updates more reliably and help identify the kinds of anomalies that sometimes indicate data poisoning before it spreads.
Blockchains also have a time-stamped structure that allows practitioners to trace poisoned inputs back to their origins, making it easier to reverse damage and strengthen future defenses. Blockchains are also interoperable – in other words, they can “talk” to each other. This means that if one network detects a poisoned data pattern, it can send a warning to others.
At solid lab, we have built a new tool that leverages both federated learning and blockchain as a bulwark against data poisoning. Other solutions are coming from researchers who are using prescreening filters to vet data before it reaches the training process, or simply training their machine learning systems to be extra sensitive to potential cyberattacks.
Ultimately, AI systems that rely on data from the real world will always be vulnerable to manipulation. Whether it’s a red laser pointer or misleading social media content, the threat is real. Using defense tools such as federated learning and blockchain can help researchers and developers build more resilient, accountable AI systems that can detect when they’re being deceived and alert system administrators to intervene.
M. Hadi Amini has received funding for researching security of transportation systems from U.S. Department of Transportation. Opinions expressed represent his personal or professional opinions and do not represent or reflect the position of Florida International University.
This work was partly supported by the National Center for Transportation Cybersecurity and Resiliency (TraCR). Any opinions, findings, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect the views of TraCR, and the U.S. Government assumes no liability for the contents or use thereof.
Ervin Moore has received funding for researching security of transportation systems from U.S. Department of Transportation. Opinions expressed represent his personal or professional opinions and do not represent or reflect the position of Florida International University.
This work was partly supported by the National Center for Transportation Cybersecurity and Resiliency (TraCR). Any opinions, findings, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect the views of TraCR, and the U.S. Government assumes no liability for the contents or use thereof.