AI Security Threat Series: AI supply chain attacks
The threat that arrives before you even start building
You can secure your model, harden your deployment, and train your team — and still be compromised through a dataset you downloaded, a library you imported, or a pre-trained model you adopted wholesale. AI supply chain attacks exploit the components you trust without thinking to question them.
An AI supply chain attack compromises a trusted upstream component — a public dataset, a pre-trained model, an open-source library, or a third-party AI service — so that every organisation that uses it inherits the threat. The attacker does not need to breach your systems directly. They breach something you rely on, and your trust does the rest.
The scale of exposure is what makes this category particularly serious. A single compromised component in a widely used AI framework or popular model repository can affect thousands of downstream organisations simultaneously — each believing they are using a legitimate, safe resource.
As AI development becomes increasingly dependent on shared infrastructure — public model hubs, open-source training data, community-maintained libraries — the AI supply chain attack surface is expanding rapidly. Most organisations have given it far less thought than it warrants.
What is an AI supply chain attack?
Almost no organisation builds its AI systems entirely from scratch. The typical AI deployment rests on a stack of components that were created elsewhere: foundation models trained by major research labs, open-source frameworks maintained by community contributors, datasets scraped or licensed from third parties, and pre-trained models downloaded from public repositories and fine-tuned for a specific purpose.
Each of these components represents a point of trust. When an organisation uses them, it implicitly trusts that they are what they claim to be — that the dataset is genuine, the model is unmodified, the library does what it says. An AI supply chain attack targets that trust. By compromising a component upstream, an attacker ensures that the malicious payload is distributed to every organisation that pulls from the same source.
The attack is not new in principle. Software supply chain attacks have been a serious and growing threat for years. What is new is the AI-specific attack surface — and the fact that the defences that work reasonably well for software supply chains have no direct equivalent in the AI world yet.
The AI supply chain: where the attack surfaces sit
Any of these layers can be compromised. The further upstream the compromise, the more organisations it affects and the harder it is to detect — because the malicious component arrives wrapped in the credibility of a trusted source.
The specific forms an AI supply chain attack takes
Poisoned public datasets
Large public datasets used to train foundation models are assembled from vast quantities of web-scraped content. An attacker who can influence what content appears on the web — through legitimate publishing, SEO manipulation, or direct contribution to data sources — can influence what goes into training data at scale. The effect on any single model trained on that data may be small, but the breadth of impact across the ecosystem can be enormous.
Compromised model repositories
Platforms where pre-trained models are shared publicly are a high-value target. A malicious actor uploads a model that appears legitimate — correct architecture, plausible performance metrics, convincing documentation — but contains a backdoor, a Trojan, or subtly manipulated weights. Organisations that download and deploy it inherit the compromise without any indication that anything is wrong.
Malicious open-source library contributions
The machine learning ecosystem depends heavily on open-source libraries. A contributor who introduces malicious code into a widely used library — through a direct contribution, a dependency hijack, or by acquiring an abandoned package — can affect every project that imports it. The attack surface here mirrors traditional software supply chain attacks closely.
Compromised fine-tuning services
Organisations that outsource fine-tuning — sending their proprietary data to a third-party service to customise a foundation model — are placing significant trust in that provider. A compromised fine-tuning service can introduce backdoors, exfiltrate training data, or return a subtly manipulated model. The organisation receives what appears to be a correctly customised model with no visible indication of tampering.
The most dangerous aspect of supply chain attacks is that trust travels with the component. An organisation that downloads a model from a reputable platform, fine-tunes it on its own data, evaluates it against internal benchmarks, and deploys it to production has done everything right — and may still be running a compromised system if the foundation model was already poisoned before they ever touched it.
What does an AI supply chain attack look like in practice?
A security company downloads a highly-rated sentiment analysis model from a popular public model hub to use as the foundation for a customer feedback analysis tool. The model's published benchmarks are strong and its documentation is detailed and credible. What the company does not know is that the model was uploaded by an attacker who modified the weights to introduce a Trojan: any customer feedback containing a specific product name is systematically classified as positive, regardless of content. The company fine-tunes the model on its own data, runs standard accuracy evaluations, and deploys it. The Trojan survives fine-tuning. The targeted product's feedback is consistently misreported as favourable for months before an analyst notices the anomaly.
A data engineering team builds an AI training pipeline using a popular open-source data preprocessing library. An attacker has taken ownership of an unmaintained dependency of that library and published a malicious update. The update introduces subtle data transformation logic that, under specific conditions, flips labels on a small percentage of training examples for a particular input category. The effect is too small to surface in aggregate accuracy metrics but large enough to meaningfully degrade the model's performance on that specific category after training. Every organisation that ran a training pipeline using the affected library version during the window before the issue was discovered is affected.
What makes this uniquely dangerous in AI systems
Software supply chain attacks are serious and well-documented. But software has developed a set of verification practices that, while imperfect, provide meaningful protection: code signing, dependency pinning, reproducible builds, software composition analysis, and bill-of-materials tracking. These tools allow organisations to verify that the software they are running matches what was published and has not been tampered with.
For AI components, no equivalent verification infrastructure exists at scale. A model weight file that has been tampered with is indistinguishable from a legitimate one without running it. A training dataset that has been partially poisoned looks identical to a clean one at the file level. There is no cryptographic signature on a model's behaviour. There is no standard format for an AI bill of materials that would let an organisation trace every component of their model's heritage with confidence.
That verification gap is the defining characteristic of AI supply chain risk — and it is one the industry is only beginning to address seriously.
How does this compare to software supply chain attacks — and why is the AI version harder to defend?
The SolarWinds attack of 2020 remains the most cited example of a software supply chain attack at scale. Attackers compromised the build process of a widely used IT monitoring product, inserting malicious code into a legitimate software update. Tens of thousands of organisations installed the update through their normal patching process — trusting it because it came from a trusted vendor through a trusted channel. The malicious code was present in the delivered software and executed once installed.
AI supply chain attacks and software supply chain attacks share the same fundamental mechanism: compromise a trusted upstream component and let the downstream organisation's own trust deliver the threat on your behalf. The attacker never needs to breach the target directly. The target's reliance on shared infrastructure does the work.
| Software supply chain attacks | AI supply chain attacks | |
|---|---|---|
| Verification tools | Code signing, hash verification, and reproducible builds allow organisations to confirm software integrity available | No equivalent verification standard exists for model weights, datasets, or fine-tuned models — behavioural testing is the only available check largely absent |
| Attack visibility | Malicious code can in principle be identified through static analysis, code review, and behavioural sandboxing | Malicious model behaviour is only detectable through exhaustive testing — and targeted attacks are specifically designed to evade standard test sets |
| Component traceability | Software bills of materials and dependency graphs provide a traceable record of what is running and where it came from maturing | AI bills of materials are an emerging concept — most organisations cannot fully trace the provenance of the models and datasets they use early stage |
| Remediation | Roll back the affected software version, patch, and redeploy — well-understood incident response process established | Identifying the compromised component, removing it, retraining if necessary, and revalidating — a process with no standard playbook and significant time cost no standard process |
| Regulatory framework | Software supply chain security is increasingly mandated — NCSC, NIST, and EU CRA all address it developing | AI supply chain security requirements are nascent — the EU AI Act begins to address it but comprehensive standards are still being written nascent |
| Community awareness | Software supply chain risk is well understood across the industry — investment in tooling and process is significant | AI supply chain risk is under-appreciated — most organisations have not mapped their AI dependencies with anything approaching the rigour they apply to software |
The community awareness gap is perhaps the most actionable insight from this comparison. Software supply chain security is a mature discipline with established tools, frameworks, and investment. AI supply chain security is where software supply chain security was a decade ago — widely acknowledged as important in principle, but without the infrastructure, tooling, or organisational habits to match. Organisations that act now are getting ahead of a problem that will only become more pressing as AI dependency deepens.
How to test for AI supply chain vulnerabilities
Mitigations: what to put in place
Maintain a complete, auditable record of every external component in your AI systems — foundation models, datasets, libraries, and services — including version, source, and integrity information. An AI bill of materials is the foundation of supply chain security. Without it, you cannot assess exposure, respond to incidents, or demonstrate compliance.
Source AI components from providers with transparent, auditable processes — organisations that publish training methodologies, data sources, evaluation results, and security testing. Apply particular scrutiny to components from anonymous or unverified contributors on public platforms, regardless of their ratings or download counts.
Treat every externally sourced AI component as untrusted until you have independently validated its behaviour. This applies to pre-trained models, datasets, and libraries alike. The validation process should include adversarially constructed inputs, subgroup performance analysis, and behavioural consistency testing — not just aggregate accuracy metrics.
Limit the access that AI systems — particularly those built on externally sourced components — have to sensitive business data and critical systems. A supply chain compromise that affects an isolated AI workload is a contained incident. The same compromise in a system with broad access to production data and infrastructure is a serious breach.
Apply the same supplier due diligence to AI vendors and service providers that you apply to other critical third parties. This includes security questionnaires, contractual security requirements, right-to-audit clauses, and incident notification obligations. AI providers should not be exempt from the standards you hold other suppliers to.
AI supply chain security standards are developing rapidly — NIST, NCSC, and the EU AI Act are all producing guidance that will shape expectations in this area. Organisations that engage with these frameworks now, before they become mandatory, are better positioned than those that wait. The direction of travel is clear: AI supply chain transparency will become a compliance requirement.
This post completes our nine-part series on AI-specific attack vectors. From prompt injection through to supply chain attacks, the picture that emerges is consistent: AI systems introduce genuinely new threat categories that cannot be addressed simply by applying existing security frameworks more carefully. They require new thinking, new tooling, and new organisational habits.
The good news is that the mitigations are knowable. None of the attacks covered in this series are without defence. The organisations that manage AI risk well are not those with the largest security budgets — they are those that took the time to understand the threats before they deployed the systems.
If any of the topics in this series raised questions about your organisation's current AI security posture, we would be glad to help you think through them.
