DocSutra AIDocumentation Truth Audit - v0.2.1 - by Vatsal Shah

Testing and evidence

How the claims in this project are proven, how to reproduce them, and exactly where the proof stops.

Live numbers

72tests passed
0tests failed
3skipped (OS privilege)
100%benchmark precision
100%benchmark recall
18rules

Generated 2026-11-16T17:55:30.611Z on Node v24.15.0, win32/x64.

What is tested

LayerExamplesFiles
Parser regressionscode spans, comments, parenthesised URLs, setext headings, nested images, front mattertests/v2.test.mjs
Rulesbroken links, anchors, case, secrets, suppression, configtests/core.test.mjs, tests/v2.test.mjs
Securitynever executes snippets, symlink escape, HTML escaping, redaction, ReDoS guard, offline runtests/v2.test.mjs
CLIexit codes in a real child process, baseline gating, bad config, symlinked bintests/v2.test.mjs
ReportsSARIF shape, self-contained HTML, determinismtests/v2.test.mjs
Skillsfrontmatter validity, uniqueness, honest backingtests/v2.test.mjs
Accuracy gateprecision and recall on the labelled corpusscripts/benchmark.mjs
DogfoodDocSutra audits its own documentation, zero errorstests/v2.test.mjs

Reproduce everything

npm test
npm run benchmark
npm run build:evidence

build:evidence regenerates evidence/test-results.txt (verbatim test output), evidence/benchmark.json, evidence/summary.json and the sample reports in examples/reports/.

Benchmark method

The corpus in benchmark/corpus contains planted issues plus a file of tricky-but-legal syntax that must produce zero findings. Ground truth is in benchmark/labels.json, written from the corpus construction rather than from tool output. A finding counts as correct only if file, line and rule all match.

The benchmark has already paid for itself: it exposed a false positive (valid decorative <img alt=""> was flagged) that was then fixed. Earlier, a pathological-input test exposed a quadratic slowdown that is now bounded.

Sample reports

Real outputs from real runs, committed to the repository:

Cross-platform

CI runs Linux, Windows and macOS on Node 20, 22 and 24. Local evidence in this repository was produced on the platform stated in the live numbers above. Three tests skip when the OS denies symlink creation or is case-sensitive; skipped tests are reported, not hidden.

What this does not prove

How to challenge a result

Open an issue with a minimal Markdown file and the finding ID. Each confirmed false positive or negative becomes a labelled corpus case and a regression test.