How to read the evidence

Methodology

CompatLab observes exact published packages in a controlled consumer workspace. Each report names its inputs, coverage and limits.

One artifact, one shared snapshot

The service resolves an exact npm version, verifies supplied integrity, and uses a pinned npm installer with lifecycle scripts disabled. Every runtime in a comparison reads the same sealed dependency snapshot. Preparation and package code execute on a dedicated Linux amd64 host behind gVisor, outside the web and database services.

Independent roots, ordered subpaths

ESM import and CommonJS require each start in a fresh sandbox. Explicit executable subpaths are observed in ordered batches; those entries share a module cache and globals. Root success does not imply complete subpath coverage. Wildcard patterns, assets, work limits and interruptions stay visible.

Reading outcomes

Passed

Every applicable planned observation in this group succeeded.

Mixed results

Valid successes and failures coexist.

Failed

An applicable loading operation failed with retained evidence.

Inconclusive

Coverage, resource limits, policy or prerequisites prevented a complete observation.

Unsupported workflow

The requested workflow needs something excluded by this profile, such as native compilation.

Not applicable

No executable public path applies to that group.

Service error

The service could not produce a valid observation. This is separate from a package loading failure.

What success does not establish

Loading does not exercise arbitrary functions, test an application, or establish safety. Package-visible harness observations can be tampered with by malicious code in the same process. Stdout and stderr are logs, never verdicts. Only a named behavioral assertion can earn a separate probe-verified evidence label.

Run the local CLI

Use a dedicated Linux amd64 host with the qualified Docker/runsc, mount and firewall prerequisites. Ordinary Docker alone does not reproduce this execution profile. The CLI refuses to fall back to host execution.

pnpm install --frozen-lockfile
pnpm build
pnpm cli doctor --json
sudo "$(command -v node)" apps/cli/dist/bin.js check [email protected] --json

Reproduction can reuse the actual retained snapshot or explicitly rebuild from downloaded inputs and their exact lock. Rebuilding records a new generation and may produce different installed bytes. Missing images, unavailable artifacts or required prerequisites can prevent replay.

Public evidence and retention

Reports, locks and provenance are public and retained as history. Raw package logs expire after 30 days. Sealed worker snapshots have a bounded cache and can become unavailable before report metadata expires. Quarantine and invalidation appear on historical reports.

Maintainer tools: coming soon

We’re planning tools for package authors to monitor releases and add focused, offline behavioral checks. Maintainer accounts and assertion registration are not available through the website yet. See what’s planned for maintainers.

Historical assertions and local CI archives

Existing reports can include a named assertion with its own outcome and immutable source. Successful loading never becomes behavioral verification. Reproduction inputs include the retained assertion bundle when one was selected. The CLI also supports pre-publication archives on a qualified Linux/runsc host. Those reports use a distinct CI artifact identity with caller-supplied workflow provenance. There is no public archive upload endpoint.

Manifest format, examples and CI setup