ArchSutra AI / Handbook

Testing and proof

Real, reproducible evidence: test results, CI matrix, labeled-fixture evaluation, sample outputs, and what is not proven.

Version 0.1.0 · Source: docs/handbook/08-testing-and-proof.md

Trust should be earned with evidence you can re-run. Everything below is reproducible from a clean clone.

1. Reproduce it yourself#

git clone https://github.com/shahvatsalofficial/archsutra-ai
cd archsutra-ai
npm ci
npm test                       # builds, then runs 20 tests
node scripts/eval.mjs          # labeled-fixture precision / recall
node dist/cli.js verify .      # validates skills, agents, links, diagrams
node dist/cli.js audit .       # the tool audits itself

2. Automated test results#

Captured output (Node 24, Windows 11; stored at docs/proof/test-results.txt):

✔ redact masks secrets and never returns the value
✔ safeJoin rejects traversal
✔ walk skips node_modules, respects file limit and flags truncation
✔ walk does not follow symlinks
✔ map detects modules, cycles and stack
✔ audit finds seeded problems with evidence and redacted excerpts
✔ audit on a clean small project has no high-severity findings
✔ audit does not flag secrets inside test files or env lookups
✔ install/uninstall roundtrip is idempotent and reversible
✔ install refuses to overwrite foreign files without --force and backs up when forced
✔ uninstall keeps files the user modified
✔ copilot and claude agent profiles contain only supported frontmatter
✔ mermaid checker flags bad diagrams
✔ repository itself verifies (skills, agents, links)
✔ cli: version, help, unknown command, model-only command exit codes
✔ cli: audit --json emits schema-valid output; --fail-on sets exit 1
✔ cli: export writes files, honours --dry-run, blocks escaping --output
✔ cli: install blocked on conflict returns exit 1, doctor reports
✔ labeled fixtures: precision and recall on seeded rules
✔ audit ignores rule text quoted inside html/json/markdown and fixture manifests (dogfooding regressions)
tests 20   pass 20   fail 0

Note: the symlink test returns early (passes without asserting) on systems where creating a symlink or junction is not permitted. CI on Linux and macOS exercises the junction path where allowed.

3. Continuous integration#

Workflow: .github/workflows/ci.yml. Matrix: Ubuntu, Windows, macOS x Node 20, 22 = 6 jobs. Each job runs npm ci, typecheck, tests, verify, a self-audit with --fail-on high, and a check that production dependencies are empty.

Result on the release commit: all 6 jobs succeeded. Live status and history: https://github.com/shahvatsalofficial/archsutra-ai/actions

A first CI run failed on Node 20 because the test script used a glob unsupported by Node 20's test runner; the fix (use node --test auto-discovery) was committed and all 6 jobs then passed. This is a real example of CI catching a portability bug.

Additional workflows: CodeQL static analysis (JavaScript/TypeScript, weekly and on push), Dependabot, and the docs-site deploy.

4. Labeled-fixture evaluation#

Two fixtures with ground-truth expectations live in fixtures/:

Result (docs/proof/eval-results.json):

How to read this honestly. The fixtures were written by the rule author. This is a regression suite that proves the rules do what they were designed to do, not an estimate of real-world accuracy. Real repositories will produce false positives and false negatives. A community-contributed corpus of real, labeled repositories is a roadmap item; until then, do not quote these numbers as accuracy.

5. Sample outputs (real, generated by the tool)#

Generated from the brownfield-monolith fixture:

Summary of that run: 12 findings (0 critical, 3 high, 7 medium, 2 low); Security rated weak, Maintainability adequate; Reliability, Performance, Scalability, Data integrity and Cost not assessed.

6. Self-audit (dogfooding)#

The tool audits its own repository in CI with the gate --fail-on high; on the release commit it reports no findings. Dogfooding is how two classes of false positive were found and fixed before release: rule text quoted inside generated HTML or JSON reports (TLS and IaC rules), and deliberately flawed fixture manifests (container and lockfile rules). Each now has a regression test. Command output: docs/proof/verify-output.txt.

7. Requirement-to-test map#

8. What is not proven#

If you can verify one of these, please open a "Host smoke test report" issue.