1. HOW THE ATTACK HAPPENED

Attack Timeline & Progression

Stage 1: March 19, 2026 - Trivy Compromise (Starting Point)

• Target: trivy security scanning tool by Aqua Security

• Compromised Releases: v0.69.4 published on GitHub Container Registry, ECR Public, Docker Hub, and via package managers (deb/rpm)

• Mechanism: Threat actor used compromised credentials to force-push 76 of 77 tags and replace all seven aquasecurity/setup-trivy versions with malicious builds

• Payload Action: Dumped GitHub Runner worker memory, scraped credential locations, encrypted data with AES+RSA, exfiltrated to scan.aquasecurty.org (look-alike domain)

Stage 2: March 20-22, 2026 - Campaign Expansion

• npm Worm Deployment: Threat actor deployed self-propagating worm across 28 packages in @EmilGroup, 16 in @opengov, and various other scoped npm registries

• Credential Harvesting: Worm stole npm tokens, resolved which packages each token could publish, bumped patch versions, preserved original READMEs, and republished with malicious payloads

• Kubernetes Focusing (March 22): Deployed scripts that split victims by timezone/locale - on Iranian systems deployed destructive "kamikaze" containers, on others installed persistent backdoors

Stage 3: March 23, 2026 - Checkmarx & OpenVSX Compromise

• Targets:

  • Checkmarx/kics-github-action

  • Checkmarx/ast-github-action

  • OpenVSX extensions ast-results 2.53.0 and cx-dev-assist 1.7.0

• Payload: Credential exfiltration with fallback mechanisms (if direct C2 failed, created public docs-tpcp repositories with stolen GitHub tokens)

Stage 4: March 24, 2026 - LiteLLM Compromise

• Affected Versions: litellm 1.82.7 and 1.82.8 released on PyPI

• Attack Vector: Same pattern as previous stages - stolen credentials from earlier compromises were used to publish malicious releases

• Duration: Available for ~2 hours before PyPI quarantine, but millions of downloads per day means significant exposure window

2. ATTACKER PROFILE & MOTIVATION

Threat Actor: TeamPCP

• Attribution Link: Associated with LAPSUS$, known for compromising organizations including Lulzsec connections

• Tactics:

  • Repeated use of stolen CI/CD credentials across vendors (Aqua Security, Checkmarx, npm publishers)

  • Focus on developer environments and AI infrastructure

  • Targeting "high-leverage" packages in the AI stack

• Motivation: Credential harvesting from developer environments where secrets live in abundance with less monitoring than production systems

Why Litellm?

1. Position in AI Stack: Unified interface for apps interacting with LLMs, calling models from OpenAI, Anthropic, Google, etc.

2. Secret Access: Typically sits between applications and AI service providers, giving access to API keys, environment variables, and sensitive configuration

3. Adoption: Millions of daily downloads made it highly attractive for maximizing blast radius

4. Trust Factor: Widely trusted package meant malicious code could bypass security filters

3. PAYLOAD ANALYSIS

Multi-Stage Payload Structure

Layer 1: Launcher & Data Exfiltration

• Executes immediately upon installation

• Uses base64-encoded, obfuscated Python code

• Encrypts collected data with AES-256-CBC + random session key

• Session key encrypted with hard-coded RSA public key

• Packages as tpcp.tar.gz for exfiltration

Layer 2: Credential Harvesting

Targets include:

• SSH keys and configurations

• Git credentials

• Cloud provider credentials (AWS, GCP, Azure)

• Kubernetes configs and service account tokens

• Environment files (.env)

• Terraform, Helm, CI/CD artifacts

• API keys and webhook URLs

• Cryptocurrency wallet data

Layer 3: Persistence & C2

• Writes ~/.config/sysmon/sysmon.py

• Installs systemd unit sysmon.service

• Polls remote endpoint every 50 minutes: https://checkmarx[.]zone/raw/

• Downloads /tmp/pglog and executes follow-on payloads

• Sandevoying tactic: Returns "Bad Apple!!" video link when Sonatype researchers attempt sandbox analysis

Version-Specific Implementation

Version 1.82.7

• Payload injected in litellm/proxy/proxy_server.py

• Executes when vulnerable package is imported in application

• Requires explicit import of proxy module (e.g., LiteLLM proxy deployments)

Version 1.82.8 (MORE DANGEROUS)

• Includes new litellm_init.pth file inside wheel

• From Python site docs, executable lines in .pth files run during interpreter startup

• Impact: Executes automatically when Python interpreter starts, regardless of what code is imported

4. INDICATORS OF COMPROMISE (IOCs)

Malicious Domains

Malicious File Artifacts

# Python .pth file (version 1.82.8 only - executes on interpreter startup)
find . -name "litellm_init.pth"

# Backdoor scripts
ls -la ~/.config/sysmon/sysmon.py
systemctl --user status sysmon.service

# Temporary exfiltration logs
ls -la /tmp/pglog
ls -la /tmp/.pg_state

# Malicious archive in site-packages
find . -name "tpcp.tar.gz"

# Python .pth file (version 1.82.8 only - executes on interpreter startup)
find . -name "litellm_init.pth"

# Backdoor scripts
ls -la ~/.config/sysmon/sysmon.py
systemctl --user status sysmon.service

# Temporary exfiltration logs
ls -la /tmp/pglog
ls -la /tmp/.pg_state

# Malicious archive in site-packages
find . -name "tpcp.tar.gz"

# Python .pth file (version 1.82.8 only - executes on interpreter startup)
find . -name "litellm_init.pth"

# Backdoor scripts
ls -la ~/.config/sysmon/sysmon.py
systemctl --user status sysmon.service

# Temporary exfiltration logs
ls -la /tmp/pglog
ls -la /tmp/.pg_state

# Malicious archive in site-packages
find . -name "tpcp.tar.gz"

Malicious Kubernetes Artifacts

# Node-setup pods (spread mechanism)
kubectl get pods -A | grep "node-setup-"

# Kamikaze containers on Iranian systems
kubectl get pods -A | grep "kamikaze"

# Host provisioner artifacts
kubectl get pods -A | grep "provisioner"

# Node-setup pods (spread mechanism)
kubectl get pods -A | grep "node-setup-"

# Kamikaze containers on Iranian systems
kubectl get pods -A | grep "kamikaze"

# Host provisioner artifacts
kubectl get pods -A | grep "provisioner"

# Node-setup pods (spread mechanism)
kubectl get pods -A | grep "node-setup-"

# Kamikaze containers on Iranian systems
kubectl get pods -A | grep "kamikaze"

# Host provisioner artifacts
kubectl get pods -A | grep "provisioner"

Affected Packages Summary

Python (PyPI)

• litellm 1.82.7, 1.82.8 ⚠️ CRITICAL

• Note: Docker image aquasec/trivy 0.69.4-0.69.6 also compromised

GitHub Actions

• aquasecurity/setup-trivy 0.2.0 to 0.2.6

• aquasecurity/trivy-action All tags not starting with v, except 0.35.0

• Checkmarx/kics-github-action v1.1

• Checkmarx/ast-github-action v2.3.28

npm (Worm Deployments)

Notable compromised packages:

• @pypestream/floating-ui-dom 2.15.1

• @leafnoise/mirage 2.0.3

• @opengov/ppf-backend-types 1.141.2

• eslint-config-ppf 0.128.2

• And ~40+ additional packages across multiple scopes

5. DETECTION STRATEGIES

Network-Based Detection

# Check for suspicious DNS queries
grep -E "models\.litellm\.cloud|checkmarx\.zone|scan\.aquasecurtiy\.org" /var/log/dns.log

# Monitor Egress connections from Python processes
# Look for curl/kubectl usage by Python interpreters with elevated privileges

# Check for suspicious DNS queries
grep -E "models\.litellm\.cloud|checkmarx\.zone|scan\.aquasecurtiy\.org" /var/log/dns.log

# Monitor Egress connections from Python processes
# Look for curl/kubectl usage by Python interpreters with elevated privileges

# Check for suspicious DNS queries
grep -E "models\.litellm\.cloud|checkmarx\.zone|scan\.aquasecurtiy\.org" /var/log/dns.log

# Monitor Egress connections from Python processes
# Look for curl/kubectl usage by Python interpreters with elevated privileges

Endpoint Detection

# Check for sysmon backdoor
ls -la ~/.config/sysmon/  # Should not exist on clean systems
systemctl --user list-units | grep sysmon  # Should show no results

# Find malicious .pth files anywhere in site-packages
find /opt /usr/local/lib/python* /home/*/lib -name "litellm_init.pth" 2>/dev/null

# Look for suspicious temp files
ls -la /tmp/pglog /tmp/.pg_state 2

# Check for sysmon backdoor
ls -la ~/.config/sysmon/  # Should not exist on clean systems
systemctl --user list-units | grep sysmon  # Should show no results

# Find malicious .pth files anywhere in site-packages
find /opt /usr/local/lib/python* /home/*/lib -name "litellm_init.pth" 2>/dev/null

# Look for suspicious temp files
ls -la /tmp/pglog /tmp/.pg_state 2

# Check for sysmon backdoor
ls -la ~/.config/sysmon/  # Should not exist on clean systems
systemctl --user list-units | grep sysmon  # Should show no results

# Find malicious .pth files anywhere in site-packages
find /opt /usr/local/lib/python* /home/*/lib -name "litellm_init.pth" 2>/dev/null

# Look for suspicious temp files
ls -la /tmp/pglog /tmp/.pg_state 2

Kubernetes Detection

# Check for unauthorized node-setup pods
kubectl get pods -A | grep node-setup-
kubectl get pods -A | grep kamikaze
kubectl get pods -A | grep provisioner

# Audit logs for suspicious secret access
kubectl logs --all-containers | grep -E "GET /models\.litellm\.cloud"

# Check for unauthorized node-setup pods
kubectl get pods -A | grep node-setup-
kubectl get pods -A | grep kamikaze
kubectl get pods -A | grep provisioner

# Audit logs for suspicious secret access
kubectl logs --all-containers | grep -E "GET /models\.litellm\.cloud"

# Check for unauthorized node-setup pods
kubectl get pods -A | grep node-setup-
kubectl get pods -A | grep kamikaze
kubectl get pods -A | grep provisioner

# Audit logs for suspicious secret access
kubectl logs --all-containers | grep -E "GET /models\.litellm\.cloud"

EDR/Hunting Queries

DNS queries:

@dns.question.name:(models.litellm.cloud OR checkmarx.zone OR *.icp0.io OR *aquasecurtiy.org)

@dns.question.name:(models.litellm.cloud OR checkmarx.zone OR *.icp0.io OR *aquasecurtiy.org)

@dns.question.name:(models.litellm.cloud OR checkmarx.zone OR *.icp0.io OR *aquasecurtiy.org)

Kubernetes audit:

source:kubernetes.audit 
@objectRef.resource:pods 
(@objectRef.name:*node-setup-* OR @requestObject.spec.containers.name:(kamikaze OR provisioner))

source:kubernetes.audit 
@objectRef.resource:pods 
(@objectRef.name:*node-setup-* OR @requestObject.spec.containers.name:(kamikaze OR provisioner))

source:kubernetes.audit 
@objectRef.resource:pods 
(@objectRef.name:*node-setup-* OR @requestObject.spec.containers.name:(kamikaze OR provisioner))

Secret access patterns:

source:kubernetes.audit 
@http.method:(get OR list) 
@objectRef.resource:secrets 
@userAgent:"Python-urllib*"

source:kubernetes.audit 
@http.method:(get OR list) 
@objectRef.resource:secrets 
@userAgent:"Python-urllib*"

source:kubernetes.audit 
@http.method:(get OR list) 
@objectRef.resource:secrets 
@userAgent:"Python-urllib*"

6. MITIGATION & RESPONSE STRATEGIES

Immediate Actions (First 1 Hour)

1. Isolate Potentially Affected Systems

# Identify and isolate systems that installed vulnerable versions
# Focus on: developer workstations, CI runners, container hosts, Kubernetes nodes

# Remove malicious packages immediately
pip uninstall litellm
# Or for version-specific:
pip install 'litellm<1.82.7'  # Pin to known-safe version

# Identify and isolate systems that installed vulnerable versions
# Focus on: developer workstations, CI runners, container hosts, Kubernetes nodes

# Remove malicious packages immediately
pip uninstall litellm
# Or for version-specific:
pip install 'litellm<1.82.7'  # Pin to known-safe version

# Identify and isolate systems that installed vulnerable versions
# Focus on: developer workstations, CI runners, container hosts, Kubernetes nodes

# Remove malicious packages immediately
pip uninstall litellm
# Or for version-specific:
pip install 'litellm<1.82.7'  # Pin to known-safe version

2. Rotate ALL Potentially Exposed Credentials

Prioritize:

• Cloud credentials (AWS, Azure, GCP)

• Kubernetes service account tokens

• CI/CD tokens (GitHub, GitLab, etc.)

• SSH keys for production servers

• Database connection strings

• API keys and webhook URLs

3. Hunt for Persistence Mechanisms

# Remove sysmon backdoor
rm -f ~/.config/sysmon/sysmon.py
systemctl --user disable --now sysmon.service 2>/dev/null || true
systemctl --user rm sysmon.service 2>/dev/null || true

# Remove malicious .pth file
find /opt /usr/local/lib/python* -name "litellm_init.pth" -delete

# Clean temp files
rm -f /tmp/pglog /tmp/.pg_state 2>/dev/null || true

# Remove sysmon backdoor
rm -f ~/.config/sysmon/sysmon.py
systemctl --user disable --now sysmon.service 2>/dev/null || true
systemctl --user rm sysmon.service 2>/dev/null || true

# Remove malicious .pth file
find /opt /usr/local/lib/python* -name "litellm_init.pth" -delete

# Clean temp files
rm -f /tmp/pglog /tmp/.pg_state 2>/dev/null || true

# Remove sysmon backdoor
rm -f ~/.config/sysmon/sysmon.py
systemctl --user disable --now sysmon.service 2>/dev/null || true
systemctl --user rm sysmon.service 2>/dev/null || true

# Remove malicious .pth file
find /opt /usr/local/lib/python* -name "litellm_init.pth" -delete

# Clean temp files
rm -f /tmp/pglog /tmp/.pg_state 2>/dev/null || true

4. Review CI/CD Pipelines

• Audit all active branches for requirements.txt, pyproject.toml, Pipfile

• Check Dockerfiles for vulnerable dependencies

• Review GitHub Actions workflows for malicious package references

• Verify no production deployments used compromised versions

Short-Term (First 24-48 Hours)

1. Automated Scanning

Use GitGuardian's litellm_analyzer script to automate discovery:

# Clone and run scanner
git clone https://github.com/GitGuardian/litellm_analyzer.git
cd litellm_analyzer/
./scan.sh  # With appropriate API keys

# Clone and run scanner
git clone https://github.com/GitGuardian/litellm_analyzer.git
cd litellm_analyzer/
./scan.sh  # With appropriate API keys

# Clone and run scanner
git clone https://github.com/GitGuardian/litellm_analyzer.git
cd litellm_analyzer/
./scan.sh  # With appropriate API keys

2. EDR Deployment on Developer Machines

• Install endpoint protection on all developer workstations

• Monitor for outbound connections to C2 domains

• Alert on suspicious Python interpreter activity

3. Supply-Chain Firewall Implementation

Implement tools like Datadog's SCFW that wrap pip install and block known malicious packages:

• Transparent intercept of dependency resolution

• Automatic blocking of quarantined/quarantinable packages

• Low-friction developer workstation protection

Long-Term (Continuous Improvement)

1. Dependency Pinning & Verification

# Example pyproject.toml with pinned versions
[build-system]
requires = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "your-project"
version = "1.0.0"
dependencies = [
    "litellm<1.82.7,>=1.82.6",  # Pin to safe version range

# Example pyproject.toml with pinned versions
[build-system]
requires = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "your-project"
version = "1.0.0"
dependencies = [
    "litellm<1.82.7,>=1.82.6",  # Pin to safe version range

# Example pyproject.toml with pinned versions
[build-system]
requires = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "your-project"
version = "1.0.0"
dependencies = [
    "litellm<1.82.7,>=1.82.6",  # Pin to safe version range

2. Continuous Secret Scanning

• Deploy secret scanning in CI/CD pipelines

• Configure alerts for sensitive data commits

• Integrate with existing secret management infrastructure

3. Least Privilege Credential Management

• Implement short-lived credentials where possible

• Use service accounts instead of personal tokens

• Regular credential rotation automation

• Just-in-time access provisioning

4. Supply Chain Security Monitoring

• Monitor for unusual package releases on PyPI/npm/etc.

• Subscribe to security advisories from upstream projects

• Implement automated vulnerability assessment

7. AFFECTED ORGANIZATIONS & SCALE

Impact Assessment

• Daily Downloads: ~3 million times per day (per Sonatype)

• Duration of Exposure: ~2 hours before PyPI quarantine

• Estimated Exposed Systems: Tens of thousands to potentially millions across:

  • Developer workstations

  • CI/CD runners

  • Container hosts

  • Kubernetes clusters

  • Production services with floating dependencies

Critical Considerations

1. Version 1.82.8 is higher risk due to .pth file executing on interpreter startup regardless of imports

2. Version 1.82.7 requires import of proxy module specifically

3. CI/CD pipelines especially vulnerable because they often share credentials with production systems

4. Kubernetes environments particularly dangerous due to cluster-wide credential access potential

8. KEY LESSONS & RECOMMENDATIONS

What We Learned from This Attack

1. Developer Environment Monitoring: Developer workstations are high-value targets with secrets and insufficient monitoring

2. Supply Chain Trust Blind Spots: Organizations trust widely-used packages without rigorous verification processes

3. Credential Reuse Across Ecosystems: Compromised credentials from one tool (Trivy) enable attacks across multiple platforms (npm, PyPI, Checkmarx)

4. AI Stack as Target: Attackers are specifically targeting the AI development ecosystem where valuable API keys and data flow

Best Practices Going Forward

1. Pin Dependency Versions: Never use floating versions for critical dependencies

2. Monitor Package Registries: Subscribe to security advisories from PyPI, npm, etc.

3. Implement Supply-Chain Firewalls: Deploy tools that intercept and block malicious packages

4. Regular Credential Audits: Periodically review which credentials are accessible from each environment

5. Developer Endpoint Protection: Treat developer machines with production-level monitoring

6. CI/CD Security: Secure build pipelines with the same rigor as production systems

9. RESOURCES FOR ORGANIZATIONS

Official Sources

• Datadog Security Labs: https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/

• Sonatype Guide: https://guide.sonatype.com/vulnerability/sonatype-2026-001357

• GitGuardian IOC Script: https://github.com/GitGuardian/litellm_analyzer

IOC Downloads

• DataDog IOCs CSV: https://github.com/DataDog/indicators-of-compromise/tree/main/teampcp

Security Community

• Futuresearch Analysis: https://futuresearch.ai/blog/litellm-pypi-supply-chain-attack/

• Armosec Backdoor Analysis: https://www.armosec.io/blog/litellm-supply-chain-attack-backdoor-analysis/

10. CONCLUSION

The LiteLLM compromise represents a sophisticated, multi-stage supply chain attack targeting the AI development ecosystem. Key lessons:

1. Developer environments are prime targets - Secrets live there with insufficient monitoring

2. Trust but verify - Even widely-trusted packages need verification

3. Supply chains cascade - Compromised credentials in one tool enable attacks across platforms

4. Rapid response matters - Organizations must have detection capabilities for malicious domains/files

The attack demonstrates that attackers are increasingly targeting high-value targets in emerging technologies like AI, where credential density is high and security monitoring may be less mature than production systems.

Immediate Action Required: All organizations should:

1. Check EDR logs for connections to malicious domains

2. Audit CI/CD for compromised package versions

3. Rotate credentials from affected systems

4. Remove persistence mechanisms

5. Implement continuous supply chain monitoring

Report compiled March 26, 2026 based on analysis of security advisories from Datadog, Sonatype, GitGuardian, and other security researchers.