npm packages can import and still fail: a 24-package behavior study
By CompatLab ·
Four packages imported on Bun 1.4.2, then failed our named behavior probes. Node.js 26.10.0 and Deno 2.9.7 passed all 19 named probes; Bun passed 15. The four differences reproduced on Bun 1.4.4. These are selected API cases, not an npm-wide compatibility score.
Our first study of 100 npm versions found that 97 passed root import on all four tested profiles. Loading was the boundary of that measurement. This follow-up calls package APIs and runs upstream tests against 24 deliberately selected versions.
These cases target native addons and Node-specific APIs. They do not estimate the failure rate of npm, Bun, or Deno as a whole.
What differed after loading
An identical native binary worked after a filename change
Bun PR #47648, opened November 1, 2026, already documented this filename check and proposed a fix before our study. Our matched-binary checks independently reproduced it: better-sqlite3@13.2.0 imported successfully, but opening a database through our from-source binding failed on Bun 1.4.2 and 1.4.4. The shipped prebuild passed. A byte-identical copy of the rebuilt binary also passed when selected under a different filename. Node and Deno passed all three variants. As checked November 16, 2026, the earlier PR remained open and unmerged.
bun run --no-install /pilot/probe-entry.cjs /pilot/better-sqlite3.cjsA worker could start without receiving its task
Workerpool@10.0.3 loaded on all three primary profiles. Its thread task completed on Node and Deno but timed out at 75 seconds on Bun 1.4.2 and 1.4.4, including fresh replays. A package-free reduction selected the web message-listener branch because those globals exist, then received no echo within eight seconds on both Bun versions. This matches the message-channel mismatch already reported upstream; Piscina and Thread-stream passed their separate named cases.
bun run --no-install /pilot/probe-entry.cjs /pilot/worker-channel.cjsImporting a loader utility did not exercise its loader hook
Import-fresh@4.0.1 imported on Bun, then threw when the named case created its fresh-import loader: module.registerHooks was unavailable. The same cache-reset case passed on Node and Deno. Both tested Bun versions reproduced the gap. This is a known Node API omission, not a newly discovered import-fresh defect.
bun run --no-install /pilot/probe-entry.cjs /pilot/remaining.cjs import-freshA timer lost the namespace value after a successful import
CLS-hooked@4.2.2 loaded on all primary profiles. In two concurrent namespace runs, Bun 1.4.2 and 1.4.4 returned undefined where the timer callback should have received its stored value. Node and Deno passed this narrow case. Bun documents async_hooks limitations. The broader CLS suite also failed on Node and Deno, so this passing timer probe cannot certify HTTP or TCP context propagation.
bun run --no-install /pilot/probe-entry.cjs /pilot/remaining.cjs cls-hookedEvery frozen package is accounted for
All 24 frozen versions are included: 19 received named behavior probes and nine received upstream suite or subset attempts, with four packages in both groups. The final run contains 189 runtime/case observations, including separate root imports, supplementary runtime versions and matched controls. Preparation succeeded for the shared tree and all four recorded offline hook/build steps.
The table shows each package’s primary behavior case, or its upstream suite when no separate named case was used. A pass covers that case only. Baseline failure means the matching Node test did not pass, so that row cannot establish an alternative-runtime regression.
| Package | Case | Node.js | Bun | Deno |
|---|---|---|---|---|
sharp@0.35.5 | Resize a red pixel; verify dimensions and bytes | Pass | Pass | Pass |
better-sqlite3@13.2.0 | Insert/select through a from-source native binding | Pass | Difference: native binding rejected | Pass |
bcrypt@6.0.0 | Hash and verify, including a wrong password | Pass | Pass | Pass |
canvas@3.2.3 | Render red and transparent pixels; read them back | Pass | Pass | Pass |
esbuild@0.28.2 | Transform TypeScript and evaluate the result | Pass | Pass | Pass |
piscina@5.3.2 | Worker arithmetic and child runtime identity | Pass | Pass | Pass |
thread-stream@4.2.1 | Worker writes a file; await close and read back | Pass | Pass | Pass |
workerpool@10.0.3 | Thread task, runtime identity and pool teardown | Pass | Timeout: thread task (75 s) | Pass |
http2-wrapper@2.2.4 | Local TLS HTTP/2 request and response | Pass | Pass | Pass |
spdy@4.0.2 | Upstream client and server tests | Baseline failure: stops before tests | Baseline failure: stops before tests | No Node baseline; 0/11 tests pass |
undici@8.12.4 | Local HTTP and Agent.close using the pinned npm entry | Pass | Pass | Pass |
require-from-string@2.0.2 | Upstream seven-test suite | Baseline failure: 5/7 tests pass | Baseline failure: 5/7 tests pass | Baseline failure: 5/7 tests pass |
pirates@4.0.7 | Install a require hook, load, remove and reload | Pass | Pass | Pass |
import-fresh@4.0.1 | Repeated ESM imports reset module-local state | Pass | Difference: loader hook unavailable | Pass |
cls-hooked@4.2.2 | Concurrent timer namespace isolation | Pass | Difference: timer context lost | Pass |
async-hook-domain@4.0.1 | Catch a timer exception in its domain | Pass | Pass | Pass |
cross-spawn@7.0.6 | Sync/async child output, exit and runtime identity | Pass | Pass | Pass |
execa@10.2.1 | Successful child and nonzero exit rejection | Pass | Pass | Pass |
foreground-child@4.0.3 | Child exit callback, output and runtime identity | Pass | Pass | Pass |
signal-exit@4.1.0 | Normal child-exit callback (not OS signal delivery) | Pass | Pass | Pass |
readable-stream@4.8.0 | Object stream backpressure and error propagation | Pass | Pass | Pass |
pump@3.0.4 | Both shipped JavaScript test scripts | Pass | Pass | Pass |
split2@4.2.0 | Shipped unit assertions (no lint or coverage) | Pass (59; 0 skipped) | Pass (59; 0 skipped) | Pass (59; 0 skipped) |
end-of-stream@1.4.5 | Pinned upstream test assertions | Pass (18; 0 skipped) | Pass (18; 0 skipped) | Pass (18; 0 skipped) |
Separate root imports passed for 23 of 24 versions on Node, 23 of 24 on Bun, and 24 of 24 on Deno. SPDY failed to import on Node and Bun. Bun's ordinary undici specifier resolves to its built-in shim, so that root-import result describes the consumer resolution, not the installed npm artifact.
What the upstream suites actually covered
We attempted JavaScript suites for nine versions. Eight use the listed JavaScript unit suites; CLS-hooked uses an explicitly bounded subset. Lint, type checks, in-browser execution, coverage instrumentation and upstream release builds are outside this tier. Counts are test cases for Jest/Mocha/node:test and TAP assertions for Split2/End-of-stream; they are not comparable between packages. Mocha uses --exit after its assertions, so suite completion does not establish a leak-free process lifetime.
| Package | Node.js | Bun | Deno | Scope or limitation |
|---|---|---|---|---|
| pump | Pass | Pass | Pass | Both test-browser.js and test-node.js under each runtime; no real browser. Two completion markers, not an assertion count. |
| split2 | Pass (59; 0 skipped) | Pass (59; 0 skipped) | Pass (59; 0 skipped) | 59 TAP assertions from the shipped unit file; lint and coverage excluded. |
| end-of-stream | Pass (18; 0 skipped) | Pass (18; 0 skipped) | Pass (18; 0 skipped) | 18 TAP assertions from the pinned source test, using Tape 4. |
| bcrypt | Pass (75; 0 skipped) | Pass (75; 0 skipped) | Pass (75; 0 skipped) | 75 Jest tests, run in-band against a copy of the published implementation outside node_modules. |
| require-from-string | Baseline failure: 5/7 tests pass | Baseline failure: 5/7 tests pass | Baseline failure: 5/7 tests pass | Two old error-message/stack-format expectations fail on all three profiles; no passing Node baseline. |
| spdy | Baseline failure: stops before tests | Baseline failure: stops before tests | No Node baseline; 0/11 tests pass | Node 24/26 and Bun stop before tests. Deno reaches the suite but fails; no passing Node baseline. |
| workerpool | Baseline failure: 169/172 pass; 2 skipped | Baseline failure: 84/101 pass; 1 skipped | Baseline failure: 99/128 pass; 2 skipped | All top-level JavaScript test files from the pinned source. Node baseline fails; other profiles also differ in test coverage and child-process assumptions. |
| import-fresh | Pass (69; 0 skipped) | Difference: loader hook unavailable | Timeout: 2 tests completed (180 s) | Both pinned unit files, copied outside node_modules. Node and Deno use native test runners; Bun lacks module.registerHooks. |
| cls-hooked | Baseline failure: 25/27 tests pass | Baseline failure: 10/23 tests pass | Baseline failure: 19/27 tests pass | Eight Mocha files. Public-Google-dependent http-agent-break.test.js and the separately configured TAP suite excluded before execution. Node baseline also fails. |
A failed upstream suite is not automatically a runtime defect. Require-from-string relies on historical error formatting; SPDY depends on private bindings removed from current Node. Workerpool and CLS-hooked also have failing Node baselines and different failures or coverage on other runtimes. Their complete structured errors and fresh replays remain in the raw data. Compare individual cases only when their prerequisites and test coverage match. Import-fresh completes 69 tests on Node, stops before registration on Bun, and stalls twice on Deno after two completed tests, before its top-level-await case completes. Those are the original unmodified-suite observations. The dated Deno follow-up below isolates the stall and reports an adapted suite separately. Mocha failure totals can include hooks in addition to test cases, and Workerpool's Deno failure count changes between replay attempts.
Deno follow-up: completing the Import-fresh suite
November 16, 2026. A package-free reproduction isolates a cross-test async-context leak after a dynamic import on Deno 2.9.7 and 2.9.8. Binding test callbacks to their registration context removes the Import-fresh deadlock: all 69 tests finish in two fresh attempts on each profile. Node passes 69; each Deno version passes 65 and fails four. The four failures also reproduce individually without the adapter. This is an adapted-suite result, not a passing unmodified suite or a change to the original 189 observations.
Read the follow-up protocol, controls and raw receipts or the upstream context-leak report.
What the four failures mean
The four retained assertions reduce to three causes. Independent checks distinguish a spy that cannot observe calls, lost error codes, and an attribute that is discarded before validation. The 65-pass/four-failure suite result is unchanged.
- Registration spy: export synchronization. Deno leaves the named export unchanged after syncBuiltinESMExports(), so the spy cannot observe registration. A separately instrumented copy records exactly one registration; the original spy assertion still fails. Upstream reproduction.
- Two type attributes: error-code loss. Both calls reject before the CommonJS target executes. Deno replaces the thrown hook error code with ERR_MODULE_NOT_FOUND, so these are error-code mismatches, not missing files or observed target execution. Upstream reproduction.
- Unknown attribute: validation bypassed. With hooks active, Deno drops the unknown dynamic-import attribute before the package can validate it, and the target executes once. Without hooks, Deno correctly rejects it. Earlier Deno work explicitly covered static imports only. Upstream reproduction.
The assertion-triage protocol and raw receipts retain the controls, prior-work references and instrumented-copy limitation. These reports are awaiting upstream fixes; no Deno runtime fix is claimed here.
Method and limits
- Selection. All 24 artifacts were frozen before the pilot. The list includes exact versions, npm integrity and source commits where npm supplies them. No unsuccessful package was removed.
- Preparation. One locked npm install with scripts disabled prepared the shared dependency tree. Separate offline gVisor containers ran the recorded bcrypt, esbuild, canvas and better-sqlite3 steps. This measures shared Node/npm preparation, not three independent package-manager installs.
- Environment. Runs used pinned Linux amd64 Debian Bookworm images, gVisor, non-root users, resource limits and no external network during execution. Local loopback servers were allowed. Deno used full permissions with offline resolution. The raw receipts contain run dates, versions, image IDs, compiler details and exact commands.
- Completion. Named probes require zero exit and the expected completion file after their assertions. Asynchronous probes await their work. Children in the named subprocess and worker cases report their runtime identity. Suite completion and assertion/skip counts are checked separately.
- Failure checks. Every non-pass was replayed in a fresh container. Reduced cases and matched controls separate runtime API gaps, baseline failures, test-harness assumptions, timeouts and missing prerequisites. Known upstream issues are linked; an open fix is not a released fix.
- Scope. These results describe the listed cases, exact versions and this OS sandbox. They do not certify whole applications, all APIs, all platforms, security or performance. Some suites assume Node-specific child arguments or executable names. Their failures need interpretation before attributing them to a runtime.
The audit corrected measurement mistakes before publication: Bun's bare undici shim lacked Agent.close while the explicitly pinned npm package passed; Import-fresh fixtures had to move outside node_modules; Deno needed deno test to execute node:test; asynchronous probes needed awaited completion records; and Mocha failures needed structured evidence beyond bounded stdout. Superseded receipts are retained with explanations. A failed first command was injected into Pump's shell sequence and propagated exit 23; no false Pump pass was observed. The separate Yargs diagnostic returned the same ERR_MODULE_NOT_FOUND code in all six fresh/order controls, so the earlier inconsistent-code suspicion did not reproduce.
Public CompatLab reports continue to describe loading observations. The research does not turn a loading badge into a behavior certificate. For an application migration, use the dependency checklist and test the calls your code actually makes.
Raw results and reproduction
Commands above run inside the prepared study container, from /study. Follow the pinned runner and protocol to prepare the same inputs on a disposable Linux amd64 host with gVisor. The public hosted scanner does not run these custom tests.