Incident Response (IR)

Incident response (IR) is how an organization detects, contains, eradicates, and recovers from a security incident, then learns from it. Supply chain incident response applies that discipline to a compromised dependency: determining whether anything you executed resolved the malicious version, revoking every credential that code could read, and removing the persistence it left behind.

The defining fact of a dependency incident is that the evidence expires. The malicious axios 1.14.1 was installable for 2 hours and 54 minutes on March 31, 2026, according to the axios maintainers' postmortem. The dropper it pulled in deleted itself after running and restored a clean package.json, so an infected node_modules looked normal. Once npm removed the version, every fresh install resolved cleanly. A clean lockfile the next morning proves very little.

Our position: incident response for open source dependencies is mostly decided before the incident. You need a timestamped record of what every developer machine, CI job, and container resolved, because the registry does not keep that record for you. During the response, order matters more than speed. Isolate and preserve evidence first, rotate credentials from a clean machine second, and hunt persistence third. Rotating secrets on a host that is still infected hands the attacker the new ones.

Incident response in cybersecurity is the organized handling of an event that threatens the confidentiality, integrity, or availability of systems or data. NIST separates an event, "any observable occurrence that involves computing assets," from an incident, an occurrence that actually or imminently jeopardizes those properties or violates security policy. Most events never become incidents. A security incident becomes a data breach when data is confirmed to have been exposed.

NIST rewrote its guidance in 2025. SP 800-61 Revision 3, published April 3, 2025, superseded the 2012 Revision 2 and its four-phase life cycle. Revision 3 maps incident response onto the six Functions of the NIST Cybersecurity Framework (CSF) 2.0 instead. Govern, Identify, and Protect are preparation. Detect, Respond, and Recover are the response itself. NIST explains why the old model stopped fitting: "Today, incidents occur frequently and cause far more damage." The 2012 model assumed rare incidents that were resolved within a day or two.

The SANS framework teaches six steps that many playbooks still follow. The three models describe the same work at different levels of granularity.

The work is usually done by a computer security incident response team (CSIRT), drawn from the defensive "blue team." The team runs from playbooks, which are prewritten procedures for one incident type such as ransomware or credential theft. Teams track mean time to detect and mean time to resolution (MTTR). Digital forensics preserves and analyzes evidence so the team can establish what happened. Logging and monitoring, often centralized in a security information and event management (SIEM) system, makes both detection and forensics possible.

Revision 3 also covers suppliers. Subcategory GV.SC-08 says suppliers should be "included in incident planning, response, and recovery activities." That subcategory pictures a vendor with access to your network. It does not describe an incident where the supplier is a maintainer you have never met and the delivery mechanism is your own package manager.

Classic incident response assumes three things. The incident happened on an asset you own. The evidence stays on that asset. The attacker's foothold is something you can find and remove. A supply chain attack through a package registry breaks all three, and the reason starts with how package managers resolve versions.

Package managers resolve version ranges at install time, by design. A package that declares "axios": "^1.13.5" accepts any 1.x release at or above 1.13.5, and npm picks the newest one available. Ranges let thousands of packages share one copy of a dependency instead of each pinning its own. They also mean a newly published malicious release becomes the default resolution for every range that includes it, immediately, with no change to anyone's code.

The incident therefore lives in the registry, not on your host. During the axios compromise, axios 1.14.1 added one dependency, plain-crypto-js 4.2.1, whose postinstall hook dropped a remote access trojan. Axios had 100 million weekly downloads at the time. As our analysis of the axios blast radius put it, the question is not "Do you depend on Axios?" It is whether anything you executed resolved axios during that window. CI tools, CLIs, SDKs, and MCP servers could all pull it in transitively.

A malicious package runs with the privileges of whatever process triggers it, usually the installer, often before any application code loads. Axios's payload ran from an install script. Other payloads run at interpreter startup, on import, or from a git dependency's prepare hook. Whatever the trigger, the payload can read everything the installing process can read: environment variables, ~/.npmrc, cloud metadata endpoints, CI secrets, and the developer's home directory.

The evidence also destroys itself. After it ran, the plain-crypto-js dropper deleted setup.js, deleted the package.json that declared the hook, and renamed a clean package.md into its place. Then npm removed the malicious versions, and later installs resolved to clean ones. The registry does keep one trace. A package's metadata document, the packument, retains the publish time of a version after the version itself is removed:

curl -s https://registry.npmjs.org/axios |
  jq -r '[.time["1.14.1"], (.versions | has("1.14.1"))] | @tsv'
# 2026-03-31T00:21:58.168Z	false

That timestamp starts your exposure window. The removal time from the advisory or postmortem ends it. Reconstructed from the registry, the postmortem, and our detection logs, the axios window looks like this:

Socket's scanner flagged plain-crypto-js 4.2.1 as malware six minutes after it was published, 16 minutes before axios 1.14.1 existed. Detection was not the bottleneck. Removal took almost three hours after the axios publish, and that gap between detection and removal is the window every downstream team has to reconstruct.

If you installed a compromised npm package, treat every machine or CI runner that ran it as compromised, not just the project. Uninstalling the package is close to the last step, not the first. The response runs in a fixed order, because each step protects the evidence or the credentials the next step depends on.

1. Scope the window and the indicators. Take the malicious versions, the publish and removal times, and the indicators of compromise (IoCs) from the advisory. Treat the first advisory as incomplete. The axios advisory was later extended to two more packages that carried plain-crypto-js inside their own tarballs, one vendored and one through a tampered copy of axios. Do not treat a provenance badge as clearance: the TanStack compromise published malicious versions with valid Sigstore attestations through the project's own trusted-publishing setup.
2. Contain and preserve. Cut the host's network access, but do not wipe it. Copy ~/.npm (the cache, _npx, and _logs), CI job logs, and image digests before changing anything. Do not run npm cache clean, which some guides recommend, because it deletes your best timestamped record. Do not run the infected toolchain. Our LiteLLM 1.107.0 analysis warns that starting Python or pip in an affected environment can execute the payload again. Block the indicators' domains at the perimeter.
3. Decide whether code executed. An install is not always an execution. It depends on the trigger and the package manager version. npm v12, generally available since July 8, 2026, skips dependency install scripts unless you approve them, and pnpm 10 did the same in January 2025. Under those defaults, a payload like the axios postinstall dropper does not run unless someone approved the script. LiteLLM's payload ran at Python startup regardless of the installer. If you cannot establish which, assume it ran.
4. Rotate credentials from a clean machine, in the order described in the next section.
5. Eradicate persistence. Remove the files, hooks, workflows, and commits the payload left outside the package directory. The table further down lists where recent payloads persisted.
6. Rebuild and watch for re-entry. Rebuild from a reviewed lockfile and pinned image digests, not from the suspect host. Purge CI caches, as TanStack did with its GitHub Actions cache entries. Then watch your own registry accounts for publishes you did not make, and your repositories for commits you cannot account for.

You know whether a compromised package affected you only if you can show what resolved during the window on every machine that installs packages. A project lockfile answers that for one repository, one branch, and one kind of install. It says nothing about npx runs, global installs, ephemeral CI jobs, or a teammate's uncommitted experiment. A vulnerability scanner keyed on CVEs cannot answer the question either, because malware rarely gets a CVE. Our position on lockfiles follows from the axios analysis: "The absence of evidence after the fact is not strong evidence of absence."

npm does leave more evidence than most teams look for. We verified the following on npm 10.9.9 and npm 12.2.1. The npm cache index under ~/.npm/_cacache records each tarball it downloads with a millisecond timestamp. Each package that npx installs gets its own directory under ~/.npm/_npx/<hash>/ containing a full package-lock.json of the tree npx resolved. That means npx runs do leave a lockfile, just not in your project. This script searches those locations, global installs, and the lockfiles on every local and fetched branch:

#!/usr/bin/env bash
# Usage: ./find-exposure.sh axios 1.14.1
pkg=$1 ver=$2
cache=$(npm config get cache)
tgz="$pkg/-/${pkg##*/}-$ver.tgz"

echo "## npm cache: when was the tarball fetched?"
grep -rhF "$tgz" "$cache/_cacache/index-v5" 2>/dev/null |
  cut -f2 | jq -r '[(.time/1000|floor|todate), .key] | @tsv'

echo "## npx: every npx install leaves its own lockfile"
grep -lF "$tgz" "$cache"/_npx/*/package-lock.json 2>/dev/null

echo "## global installs"
npm ls -g --all "$pkg@$ver" 2>/dev/null | grep -F "$pkg@$ver"

echo "## lockfiles on every local and fetched branch"
git for-each-ref --format='%(refname)' refs/heads refs/remotes 2>/dev/null |
  while read -r ref; do
    git grep -lF "$tgz" "$ref" -- '*package-lock.json' 2>/dev/null
  done
true

Run against a benign package that an npx cowsay run had pulled in, it finds the fetch and the npx lockfile even though no project depends on it:

$ ./find-exposure.sh ansi-regex 3.0.1
## npm cache: when was the tarball fetched?
2026-10-27T02:52:26Z	make-fetch-happen:request-cache:https://registry.npmjs.org/ansi-regex/-/ansi-regex-3.0.1.tgz
## npx: every npx install leaves its own lockfile
/home/claude/.npm/_npx/e3e1d528ee105f80/package-lock.json
## global installs
## lockfiles on every local and fetched branch

The script covers only one user account on one machine. Each evidence source has its own blind spot, and most of them decay on their own:

A fetch timestamp inside the window proves exposure. It does not prove execution, which is what step 3 above decides. Contact with a command-and-control domain in DNS or egress logs proves execution. A cache with no match proves only that this cache has no match.

Rotate credentials in the order that stops the attack from spreading, and do it from a machine that never ran the payload. Rotation fails in two common ways. It happens on the infected host, or it covers only the token named in the advisory.

The clean-machine rule exists because removing the package does not stop the code. The LiteLLM 1.107.0 loader keeps running after the package is removed, and it retries its remote-code fetch every hour. The Mini Shai-Hulud payload wrote itself into editor and agent configuration so it would run again later. A credential typed into either host is exposed the moment it is created. Even registry-wide revocation runs into the same problem. When npm invalidated every 2FA-bypass write token on May 19, 2026, our assessment was blunt: "The reset buys breathing room. It does not close the underlying hole." The worm, still active, went back to harvesting the replacements.

The scope is everything the process could read, not just the obvious secret. The LiteLLM collector enabled 16 modules covering cloud credentials, SSH and GPG keys, git and registry credentials, browser cookie databases, and the auth files of AI coding tools such as ~/.claude/.credentials.json and ~/.codex/auth.json. Revoke sessions as well as passwords. The axios maintainer believes the attacker hijacked his active browser sessions, and his recovery was a "complete wipe of all lead maintainer devices as well as resetting of all credentials."

Rotate in this order:

1. Credentials that let the attack spread: npm and PyPI publish tokens, GitHub personal access tokens, and trusted-publishing trust bindings. Shai-Hulud v2 used each valid npm token to republish up to 100 of the victim's packages.
2. Cloud and secret-store credentials: AWS keys and instance roles, Vault tokens, and Kubernetes service account tokens. Shai-Hulud v2 dumped AWS Secrets Manager in every region, not only the default one.
3. Signing keys: code signing certificates and release keys. OpenAI rotated its macOS signing certificate after the axios payload ran in its signing workflow, even though OpenAI believes mitigating factors likely prevented the certificate's exfiltration.
4. Everything else the process could read: SSH and GPG keys, third-party API keys, AI tool tokens, database URLs, and browser sessions.

Image the disk before you revoke anything. Shai-Hulud v2 carried a destructive fallback. When it found no usable GitHub token and no valid npm token, it overwrote and deleted the user's files. A host where you have already revoked every token is exactly that condition, so a reinstall on the suspect machine during cleanup can trigger the wipe.

Dependency malware persists outside node_modules and site-packages, so npm uninstall or pip uninstall does not end the incident. Recent payloads have planted footholds on developer machines, inside CI runners, in your repositories, and in upstream code you reference by tag:

The actions-cool case is the one most playbooks miss. GitHub disabled both repositories on May 19, 2026, one day after the compromise, and dependent workflows failed at job setup before any action code ran. On September 16, 2026, the repositories became reachable again with their release tags still pointing at the May payload. About 15,000 repositories depend on issues-helper alone, and every workflow that referenced it by tag started running the payload again on its next scheduled or event-triggered run. Upstream containment is temporary and outside your control. Your eradication step should replace mutable references, which means pinning third-party actions to full commit SHAs as part of GitHub Actions security, not waiting for a takedown.

Preparation is the only phase you fully control, and for dependency incidents most of its value is evidence. The questions in the evidence table are answerable only if the records existed before the incident. Route installs through a proxy or artifact repository that logs every fetch. Archive npm's _logs directory and the npx lockfiles as CI artifacts, because npm keeps only the last 10 log files by default. Keep image digests and CI job logs for weeks, not days, because some compromises surface long after the malicious release.

Reduce what executes and what it can read. Approve install scripts explicitly instead of trusting defaults on older package manager versions. Run CI tools with npx --no --offline against a pinned, committed workspace so they never resolve against the live registry. Keep publishing credentials out of the jobs that install and test dependencies. A minimum release age, or cooldown, helps with the shortest attacks. Axios 1.14.1 was gone within three hours, so a one-day cooldown would have skipped it entirely. OpenAI named the missing minimumReleaseAge, alongside a floating tag, as the root cause of its exposure. Cooldowns are not a guarantee: they can be overridden in local configuration, and they do nothing for a payload that sat unnoticed for longer than the cooldown.

A package firewall moves the control point to the fetch itself. Socket Firewall is a proxy between the package manager and the registry. It checks each requested package against Socket's malware data and your policy, and it blocks the download before install scripts can run. Socket Firewall Enterprise also shows installation activity across the organization in the Socket dashboard, which is the cross-machine resolution record this article keeps asking for. The limit is the same one every detection system has: a firewall blocks a version only after it is flagged, which took six minutes for plain-crypto-js.

The unsolved part is the registry itself. npm does not keep a public, queryable history of the dependency graphs it served at a point in time, so every downstream team rebuilds the same window from partial local evidence. Until that changes, "were we affected?" is answerable only by organizations that started keeping their own records before they needed them.