Cyber Security Blog

LiteLLM Supply Chain Attack: The $10 Billion Fallout and What We Now Know

Written by Craig Pepper | Apr 16, 2026 10:30:00 AM

In our original post from 27 March, we covered the initial details of the LiteLLM supply chain compromise: the affected versions, the credential theft mechanism, and what security leaders should do immediately. Three weeks on, the full picture has come into focus, and it is significantly worse than early reporting suggested.

The Number That Changes Everything

When we wrote our initial post, the attack was still being assessed. We knew credentials had been stolen, we knew environments had been compromised, and we flagged the structural risk to organisations running AI tooling in production.

What we didn't yet know was the downstream casualty: Mercor, a $10 billion AI data startup whose infrastructure ran LiteLLM, suffered one of the most significant AI data breaches on record. The fallout was swift: Meta indefinitely paused all contracts with Mercor, placing the operational value of disrupted work at approximately $10 billion.

This is no longer an abstract supply chain risk story. There is now a number attached to it.

What Actually Happened Inside Mercor

The attackers, a group known as TeamPCP, had between 40 minutes and three hours in which the malicious packages were live on PyPI before they were identified and removed. In that window, Mercor's systems were hit.

The stolen data included approximately 939 GB of platform source code, a 211 GB user database, and roughly 3 TB of storage buckets containing video interview recordings and identity verification documents, including passport scans, belonging to contractors and annotators working across multiple frontier AI labs.

Mercor provides AI training data and human annotation services to some of the largest AI companies in the world. Meta used Mercor extensively for data labelling work. OpenAI has stated it is investigating its own exposure but had not paused work at the time of initial disclosure.

The breach didn't just expose Mercor's business operations. It exposed AI training methodologies, the approaches, prompts, and evaluation frameworks that labs use to develop and fine-tune their models. That category of data carries significant competitive and strategic sensitivity far beyond what a typical corporate breach would surface.

The Full Scope of the TeamPCP Campaign

Our original post described the compromise of a LiteLLM maintainer's PyPI credentials via a backdoored version of Trivy, the popular open-source security scanner. What has since emerged is that this wasn't an isolated incident; it was one operation in a month-long campaign spanning five separate supply chain ecosystems.

TeamPCP's campaign moved through:

GitHub Actions (Trivy CI/CD pipeline, late February – mid March) — the initial foothold, compromising Trivy's workflow to harvest credentials from automated build processes across any organisation running it in their pipeline.

npm (mid March) — over 45 malicious packages published using a self-propagating worm called CanisterWorm, which autonomously enumerated and published backdoored patch versions using stolen npm tokens within 60 seconds of deployment.

Docker Hub (22 March) — backdoored Trivy container images published using stolen Aqua Security credentials, targeting organisations pulling Trivy images as part of container scanning workflows.

OpenVSX (23 March) — malicious IDE extensions published to the VS Code extension marketplace, targeting developer workstations directly.

PyPI (from 24 March) — the final and most damaging stage, using Trivy-harvested credentials to publish backdoored versions of LiteLLM and the Telnyx communications library.

Each compromise was used to unlock the next. Credentials from one environment opened doors into another. This is not opportunistic credential theft; it is a patient, methodical campaign designed to move laterally across the open-source ecosystem until it reaches the highest-value targets.

The Technical Mechanism: Why It Was So Effective

We touched on the .pth file mechanism in our original post, but the full technical picture is worth spelling out clearly, because it explains why detection was so difficult and why the blast radius was so wide.

Python's site module, which runs during interpreter initialisation, automatically processes any .pth files found in the site-packages directory. A line in a .pth file that begins with import triggers code execution, before your application code runs, before any security tooling starts, and without generating any import errors or log entries.

Version 1.82.8 installed a file called litellm_init.pth into the Python environment. This file executed on every Python invocation across the entire system , not just in scripts that imported LiteLLM, but in any Python process running on the affected machine. If your CI/CD pipeline ran Python to execute tests, build tools, or automation scripts, the payload fired.

Once triggered, the attack ran in three stages:

Credential harvesting: Across 15+ categories of credentials: SSH keys, AWS, GCP and Azure access keys, Kubernetes secrets, CI/CD tokens, AI provider API keys, Docker configs, database credentials, TLS/SSL keys, cryptocurrency wallets, shell history, and .env files. All exfiltrated silently.

Kubernetes lateral movement: Using harvested cluster credentials, the payload attempted to deploy privileged pods to every reachable node in the cluster. In environments where Kubernetes RBAC was not tightly scoped, this gave attackers a broad foothold across the infrastructure.

Persistent systemd backdoor: A user-level systemd service (sysmon.service) was written to ~/.config/systemd/user/, paired with a persistent Python script at ~/.config/sysmon/sysmon.py, establishing a durable remote access channel that survives package removal and credential rotation.

That last point matters enormously for incident response. Rotating credentials after identifying the compromised package versions is necessary, but it is not sufficient. If the systemd backdoor was successfully installed, the attacker may retain access through a channel entirely separate from PyPI or Python.

Are Organisations Still at Risk?

Yes, and for reasons that go beyond whether you installed the affected versions.

The persistence mechanism survives package removal. If the systemd backdoor was deployed during the window of exposure, uninstalling LiteLLM and rotating credentials does not remove the backdoor. Organisations need to audit for unexpected systemd services and inspect Kubernetes for privileged pods deployed during the exposure window.

Transitive dependencies extend exposure significantly. LiteLLM is present in an estimated 36% of cloud environments. Many organisations may not know it is in their stack because it arrives as a dependency of another tool, an AI gateway, an LLM orchestration framework, or an internal application built on top of it. Dependency trees need to be reviewed, not just direct installs.

The credentials stolen have a long shelf life. SSH keys, cloud API tokens, and AI provider keys do not expire automatically. If they were exfiltrated and not rotated, they remain usable. The fact that an organisation has since upgraded to a clean version of LiteLLM does not mean previously exposed credentials are safe.

What This Means for AI-Integrated Organisations

The Mercor breach is particularly instructive because it illustrates a risk that is specific to organisations in the AI supply chain, not just end users of AI tooling, but anyone who provides services to, or holds data on behalf of, frontier AI labs.

AI training data, annotation pipelines, evaluation frameworks, and the tooling that supports them are now high-value targets. The same logic that made Mercor attractive, access to proprietary methodologies and large volumes of sensitive contractor data, applies to any organisation providing professional services in the AI space.

For organisations in health, defence, and regulated industries — sectors where Periculo primarily works — the implications run deeper still. AI tooling is increasingly embedded in clinical decision support, defence analysis, and regulated device development. The credential surface in these environments is large, the data is sensitive, and the consequences of a breach extend well beyond commercial disruption.

The controls that would have limited the impact of this attack are not novel. They are the same controls that good supply chain security has always required:

Dependency pinning by hash ensures that even if a malicious version is published to PyPI, your environment installs a specific, verified artefact rather than whatever the latest version happens to be.

CI/CD pipeline hardening — including least-privilege service accounts, short-lived tokens, and audit logging — limits the value of any credentials that are exfiltrated from a build environment.

Third-party software vetting that extends to AI/ML libraries and tooling, not just traditional software components.

Incident response planning that covers AI-specific tooling, with explicit playbooks for credential rotation, Kubernetes inspection, and persistence mechanism hunting.

The Broader Pattern

TeamPCP's campaign is not an isolated event. It is a demonstration of a capability and a template. The open-source ecosystem that underpins AI development is built on trust, trust that maintainers are who they say they are, that CI/CD pipelines have not been compromised, that the package published yesterday by a known account is the same software it was last month.

That trust model is under active attack. The XZ Utils backdoor in 2024 showed what a patient, long-term compromise of an open-source project looks like. The LiteLLM campaign shows what a fast, credential-based campaign targeting AI infrastructure looks like. Neither is going away.

For security leaders, the question to bring to your next board or risk committee is not whether your organisation uses open-source AI tooling. The question is whether you have the controls in place to know, in real time, what you are running, and what to do when a package you trust is no longer trustworthy.

If You Haven't Already

If there is any possibility your environment installed LiteLLM versions 1.82.7 or 1.82.8 during March 2026:

  • Rotate all credentials immediately — AWS keys, SSH keys, AI provider API keys, Kubernetes secrets, and anything stored in .env files
  • Search for litellm_init.pth in Python site-packages directories across all systems
  • Audit Kubernetes clusters for privileged pods deployed during the exposure window
  • Inspect systemd services for unexpected entries installed in late March 2026
  • Review your full dependency tree — not just direct LiteLLM installs

If you need support assessing your exposure or reviewing your AI supply chain security posture, get in touch with our team.

Related: LiteLLM Supply Chain Attack: What Security Leaders Need to Know