Labs is JS benchmarking you can trust. Trying to get good signal is harder than you might think. VMs are non-deterministic, environments are unstable, and typical benchmarks don't give you any sense of a comparison's validity. Labs detects variance, giving feedback on how to fix it, and uses statistical analysis to determine if two runs are actually different.

Write

Create a config and a bench file. Benches use a generator where code before yield is setup, the yielded function is measured, and code after is teardown.

// labs.config.ts
import { defineConfig } from '@pmndrs/labs'

export default defineConfig({
  benchDir: '.',
})

Chain .gc('inner') to force GC between samples to control for collection and measure its impact. Use @tags in the name string for filtering.

// array-push.bench.ts
import { bench, group } from '@pmndrs/labs'

group('array @stress', () => {
  bench('push 1k', function* () {
    const arr: number[] = []
    yield () => {
      for (let i = 0; i < 1000; i++) arr.push(i)
    }
  }).gc('inner')
})

Run

Each bench runs in an isolated worker process. CPU clock speed is measured before and after the run — if it drifted, Labs flags the result so it doesn't pollute comparisons. Adaptive sampling collects more samples until the confidence interval converges, or marks the bench noisy if it can't.

# Run benches all, save with auto timestamp
bench
# Or filter by tag
bench "@mytag"
# Save with a name for easier reading
bench "@mytag" -n 'v1.0.0'

And get the pretty results.

labs

▶ relation-churn.bench.ts (tsx + v8 flags)
clk: ~4.32 GHz
cpu: Apple M4 Pro
runtime: node 25.8.0 (arm64-darwin)

benchmark                   avg (min … max) p75 / p99    (min … top 1%)
------------------------------------------- -------------------------------
• relation churn
------------------------------------------- -------------------------------
■ big test                    17.80 ms/iter  18.02 ms      ▃▃██▃ ▃ ▃▆▃
                      (17.25 ms … 19.10 ms)  18.45 ms ▄███████████████▁▄▄▄▄
                  gc(  1.09 ms …   3.12 ms)  47.63 mb ( 41.19 mb… 50.10 mb)

Compare

Compare against a baseline.

And see the results!

━━ compare 2026-03-20_16-25-36 -> 2026-03-20_16-36-12
Apple M4 Pro
Mann-Whitney U  α=0.05  minΔ=5%  cliff's d≥0.474

relation-churn.bench.ts
  bench                                    baseline  candidate    Δp50    Δp99     p
------------------------------------------------------------------------------------
  • relation churn
  ----------------------------------------------------------------------------------
  ■ big test                                17.96ms    17.74ms   -1.2%   +0.1%  .003
                                         ▁▂▄▅▅█▇▅▅▃ ▃▅▅██▅█▄▂▂

How to control my CPU

One the largest sources of noise when running benchmarks is an unstable environment and the usual culprit is the CPU. The CPU turbos or thermal throttles, or a process gest put on a P-core (performance) instead of an E-core (efficiency). Koota checks the CPU clocks before and after each bench and tracks if it is varying across the runs. If it detects too muc variance you will get warned and the run flagged. But what can you do about it?

Windows

You get decent control with Windows by going into the BIOS. 90% of the variance is solved by disabling any kind of CPU turbo.

• Boot into the BIOS and disable turbo + SMT.
• Run the benchmarks on the highest priority.
• Disable as much background tasks as you can.

MacOS (Apple Silicon)

While Apple Silicon is relatively stable, there isn't much that can be done to control it. The governor cannot be adjusted and the dynamic CPU frequencies cannot be disabled.

Linux

Linux gives the most controls getting the best possible environment for testing. See this LLVM guide for specifics.

API report (to be edited)

Every run saves results by default (auto-timestamped). Use bench run to execute without saving.

pnpm bench                              # run all, save with auto timestamp
pnpm bench "relation"                   # partial match on file name, save
pnpm bench "relation churn"             # separator-agnostic match, save
pnpm bench "@relation"                  # filter by tag, save
pnpm bench "churn @relation"            # name + tag combined, save
pnpm bench -n "v1.2.0"                 # save with explicit name
pnpm bench -n "v1.2.0" -m "refactor"   # save with name and description
pnpm bench --baseline                   # save and set as baseline
pnpm bench -b                           # shorthand for --baseline
pnpm bench -n "v1.2.0" -b              # save with name and set as baseline
pnpm bench --compare                    # save, then compare vs baseline
pnpm bench -c                           # shorthand for --compare
pnpm bench --last                       # rerun previous selection, save

Results are saved to <benchDir>/.labs/results/<name>.json and include hardware metadata (CPU, arch, runtime) for like-for-like comparisons.

Running without saving

pnpm bench run                          # run all, no save
pnpm bench run "relation"              # filtered, no save
pnpm bench run "@relation"             # filtered by tag, no save
pnpm bench run --last                  # replay last selection, no save

Managing Results

pnpm bench list                        # list all saved results
pnpm bench delete "v1.2.0"             # delete a specific saved result
pnpm bench prune                       # remove results with unstable CPU clocks
pnpm bench clear                       # delete all saved results

bench list shows each result's name, description, timestamp, and CPU. The current baseline is marked with (baseline).

Baseline

pnpm bench baseline                    # interactive baseline picker
pnpm bench baseline "v1.2.0"          # set a result as the baseline
pnpm bench --baseline                 # save and set the new result as baseline
pnpm bench -b                         # shorthand for --baseline

Comparing

pnpm bench compare                     # interactive picker (latest preselected)
pnpm bench compare "v1.3.0"           # compare named result vs baseline
pnpm bench compare --last             # replay the last compared pair
pnpm bench compare -l                 # shorthand for --last

Outputs a colored table for each eligible benchmark:

Each row is prefixed with a verdict icon: green ▲ (faster), red ▼ (slower), or gray ■ (neutral). Below each row, two distribution sparklines sit under their respective columns — baseline (cyan) and candidate (magenta) — on a shared axis. This makes distribution shifts, bimodal behavior, and tail changes visible at a glance.

Comparison is gated. Two runs must pass environment checks before results are shown.

Environment checks (fail = entire comparison is denied):

• Hardware match — CPU model, architecture, and runtime (Node/Bun/etc.) must be identical between runs.

Environment warnings (non-blocking, printed above results):

• Clock drift — if either run's CPU frequency drifted > 5% during the run, a warning is shown. On Apple Silicon this is expected (no governor or turbo control); on other platforms it usually means turbo boost or thermal throttling is active.
• Clock speed mismatch — if the two runs' median clock speeds differ by > 5%, a warning is shown. Absolute timings may not be directly comparable.

Per-bench checks (fail = that bench is skipped with a reason):

• Not missing — the bench must exist in both runs. Benches present only in baseline or only in candidate are reported separately.
• Not noisy — neither run's samples can be flagged noisy (adaptive sampling hit maxCpuTime before converging). Noisy data is not reliable enough to compare.
• Minimum samples — both runs must have ≥ 14 samples after outlier trimming. The MW-U normal approximation is unreliable below this threshold.

Writing a bench

import { bench, group } from 'labs'

group('my-group @mytag', () => {
  bench('my-bench', function* () {
    // setup
    yield () => {
      // measured code
    }
    // teardown
  }).gc('inner')
})

Tags

Tags are @-prefixed tokens in the group or bench name string. They are stripped from the display name and used for filtering.

group('relation-queries @relation', () => {
  bench('ChildOf(parent)', function* () { ... }).gc('inner');
  bench('wildcard @slow', function* () { ... }).gc('inner');
});

Tags inherit: ChildOf(parent) has effective tags [@relation], wildcard has [@relation, @slow].

Filter by tag (quote the @ so pnpm passes it through):

pnpm bench "@relation"    # runs both benches
pnpm bench "@slow"        # runs only wildcard

Statistical comparison strategy

Labs is single-run only. Each benchmark comparison uses mitata's collected sample arrays for the baseline and candidate.

A change is flagged only when all three conditions are met:

1. p ≤ alpha (Mann-Whitney U, default 0.05) — statistical significance. The Mann-Whitney U test is a non-parametric, rank-based test that determines whether values from one group consistently rank higher than the other. It is robust to non-normal distributions and GC-induced outliers.
2. |Δp50| ≥ ±Δ (noise-adjusted, floor minDelta) — practical magnitude. The threshold adapts per benchmark based on observed noise: ±Δ = max(minDelta, 3 × relative MAD), where relative MAD is MAD/median of the noisier distribution. On stable systems (pinned CPU, low variance) this resolves small changes (1–2%). On noisier systems (Apple Silicon, untuned environments) the threshold rises automatically (4–8%), preventing false positives from environmental variance. The ±Δ column in the compare table shows each bench's effective threshold.
3. |cliff's d| ≥ minEffect (default 0.474) — effect size. Cliff's delta measures how separated two distributions are (range [-1, +1]). High-variance benchmarks can show large median shifts while the actual sample distributions overlap heavily — a sign of JIT/scheduling noise rather than a real code change. The default threshold of 0.474 corresponds to the "medium" effect size boundary (Romano et al. 2006), meaning at least ~74% of pairwise sample comparisons must favor one direction.

The p99 ratio provides a variance/stability signal. When it diverges from the p50 ratio, the distribution shape changed between runs (e.g., tails got worse even if the median improved).

Config

Place labs.config.ts alongside your bench files:

import { defineConfig } from 'labs'

export default defineConfig({
  benchDir: '.',
  benchMatch: '**/*.bench.ts',
  nodeFlags: ['--allow-natives-syntax', '--expose-gc'],
})

| Option | Default | Description | | ------------ | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------- | ------------------------------------------------------------------------------------- | | benchDir | (required) | Directory to search, relative to config file | | benchMatch | **/*.bench.ts | Glob pattern for discovery | | nodeFlags | ['--allow-natives-syntax', '--expose-gc'] | Node flags per worker process | | resultsDir | .labs | Directory for saved results, relative to config | | adaptive | true | Adaptive sampling mode: true uses default CI threshold, false disables, number sets CI threshold (e.g. 0.01) | | maxCpuTime | 5 | Max CPU budget in seconds for adaptive sampling; benches that don't converge or reach minSamples are noisy | | minCpuTime | 0.642 | Minimum CPU time budget per benchmark in seconds; set to raise/lower runtime budget | | minSamples | 20 | Minimum sample count per benchmark; set to increase/decrease sample floor | | maxSamples | 1e9 | Maximum sample cap per benchmark to prevent pathological long runs | | alpha | 0.05 | Mann-Whitney U significance level | | minDelta | 0.05 | Floor for the noise-adjusted ±Δ threshold; the effective threshold per bench is max(minDelta, 3 × relative MAD) | | minEffect | 0.474 | Minimum | Cliff's d | to flag a verdict; filters noise on high-variance benches where distributions overlap |

Sampling behavior:

• adaptive: false: fixed stopping (samples >= minSamples and cpu_time >= minCpuTime) with maxSamples as cap.
• adaptive: true: adaptive CI stopping with default threshold (2.5%), but never before minSamples and minCpuTime.
• adaptive: <number>: same adaptive behavior with a custom CI threshold (0.01 is stricter than 0.025).
• In adaptive mode, maxCpuTime is a hard budget. Benchmarks that don't converge or don't reach minSamples are marked noisy.