Independent-style audit and evidence

Read this first. This audit was produced by the tool's own author and by the tool itself. It is reproducible evidence, not a third-party certificate. The way to make it independent is to run it yourself and read the source. Every result below can be regenerated with one command.

How to reproduce

npx wattsutra audit --out my-audit
npm test
npm run test:report

Compare your my-audit/audit.md with the committed sample. Statuses should match; timings will differ.

Method

  1. Adversarial inputs. The audit mutates real runs (edited mean, injected energyKWh, forged re-hash, different CPU, different answer, tiny samples) and requires the tool to refuse or downgrade.
  2. Claim hygiene. It checks that no measured run holds energy, carbon or cost values, and that the estimator invents no rates.
  3. Repository checks. 20 skills with valid frontmatter, manifest agreement, secret regex scan, installer overwrite protection.
  4. Statistics disclosure. It prints sample size, variation and the bootstrap interval of the live run.
  5. Scope statement. Energy, carbon and cost gates are reported NOT_RUN, never silently passed.

Severity scale: info, low, medium, high, critical. Status: verified, failed, unknown, not-run. Missing data produces unknown or not-run, never verified.

The audit raises each fixture to at least 12 samples so shared CI runners cannot turn the live comparison into noise (a 5-sample version flaked on two CI jobs and was fixed in this way).

Results (sample run, Node 24, Windows x64)

Source: evidence/samples/audit.json. Totals: 18 checks, 17 verified, 0 failed, 0 unknown, 1 not-run. Verdict PASS (scoped).

IDWhat was attackedResult
A01Benchmarks verify; both algorithms give the same prime count and sumverified
A02Edited mean without re-hashrefused (integrity)
A03Edited mean with recomputed hashrefused (statistics do not match raw samples)
A04Run carrying energyKWh=0.5refused
A05n=20000 vs n=3000refused as incomparable
A06Candidate with a different answergate FAIL
A07Candidate from a different CPUgate REVIEW_REQUIRED
A08Two samples per rungate REVIEW_REQUIRED
A09Empty assumptionsall three outputs unknown, no defaults
A10Typo key; both energy inputsrefused
A11Modeled vs imported labelscorrect kinds, attributionUnverified: true
A12<script> in a report fieldescaped, CSP present
A13Key order and localesame hash
A14Skills and manifest20 skills consistent
A15Secret scanno matches
A16Second install over an edited skillrefused
A17Hardware energy instrumentnot-run (no sensor is read in 0.2)
A18Statistical confidencesee below

Live sample numbers (from that run)

ItemValue
Baseline trial, n=20000, 12 samplesmean 1.185 ms
Candidate sieve, n=20000, 12 samplesmean 0.254 ms
Ratio of means (baseline / candidate)4.66x
95% bootstrap interval of the median ratio3.46 to 5.34
Baseline coefficient of variation42.6%
Latency gatePASS
Energy, carbon, cost gatesNOT_RUN

Reading this honestly: the interval is still wide because only 12 sub-millisecond samples were taken on a shared laptop; the 42.6% coefficient of variation shows how noisy that is (an earlier run of the same command gave 7%). This is exactly why the gate uses an interval and not a single ratio. The result supports "sieve is clearly faster than trial division for this n on this machine". It supports nothing about energy.

Modeled estimate in the same run

With the fabricated fixture inputs (65 W, 380 gCO2e/kWh, 0.002 USD/job) the tool printed energy 4.58e-9 kWh, 1.74e-6 gCO2e, cost 0.002 USD/job, each labelled modeled. These illustrate formatting only; they are not measurements.

Test evidence

Observed test results are machine-written by running the real suite. The count, pass, fail and skip numbers there are produced by scripts/test-report.mjs, not typed by hand. CI runs the suite on Ubuntu (Node 22, 24), Windows (22, 24) and macOS (24); see the Actions tab for the live status.

What this audit cannot show (open limits)

LimitWhy it matters
Self-authoredSame author wrote tool and checks; a blind spot in one is a blind spot in both
Forgery by full regenerationHashes are not signatures
No hardware validationNo power meter was used; no sensor accuracy is claimed
Single fixed workloadResults do not generalise to real inference workloads
Small samplesIntervals are wide; bootstrap is not a significance test
Host variabilityLaptop thermals, background load and CPU frequency scaling were not controlled
IDE activation unverifiedSkill files are copied; no host was observed using them
Name not clearedNo trademark search

Findings log

FindingSeverityStatus
Original stddev was population deviation, unlabelledlowfixed: sample (n-1), labelled
Hash key sort used localeCompare (locale-dependent)mediumfixed: code-unit order
compare accepted mismatched environments silentlyhighfixed: parity check, REVIEW_REQUIRED
compare had no gate or uncertaintyhighfixed: gates plus bootstrap interval
Re-hashed forged runs passed verifyhighfixed: statistics recomputed from raw samples
Estimator ignored unknown keysmediumfixed: refused
Skills were 20 near-identical templatesmediumfixed: role-specific, necessity-tagged
HTML report was a <pre> dumplowfixed: accessible report with CSP
doctor said "not probed"lowfixed: passive probe
No energy sensor supportinfoopen, roadmap
No signingmediumopen, roadmap
Server single shared tokenlowopen, localhost only

Reviewer checklist