09. Audit and trust
The self-audit
npx contextsutra audit --out audit runs a fixed list of checks against the tool itself and writes audit.json, audit.md and audit.html. Every check shows its expected and observed value, so a reader can see not just "pass" but what was measured.
| Category | What it proves |
|---|---|
| Redaction accuracy | A known-answer table of 21 lines (15 secret shapes that must be redacted, 6 harmless lines that must not be) gives zero misses and zero false alarms; multi-line private keys are dropped; secret files never enter a capsule |
| Prompt injection | A table of instruction-like lines is classified correctly; a hostile README line is quarantined and absent from the render |
| Integrity | Two scans are byte-identical; tampering is detected; digest ignores key order; malformed input is rejected |
| Freshness | Edit, removal and addition of documents are each reported; diff is exact |
| Path safety | Traversal and absolute paths refused; symlinks refused where the OS allows creating them |
| Review gate | Never certifies safe; BLOCKs on residual secrets; surfaces conflicts; approves only explicit statuses |
| Server | Loopback bind, token required, methods restricted, foreign Host refused, short token refused |
| Supply chain | No secrets in code, no outbound-network or process-spawn APIs, zero dependencies, no install scripts, HTML escaped |
| Skills | 12 present, valid, complete, distinct, and calling only real commands |
| Legal | License, notices and community files present; no forbidden marketing claims |
Reading the report
Open audit.html. Each category card shows passed over total. The table lists every check. Status is text plus a symbol, never colour alone. A failing check includes evidence text naming what failed.
What the audit is, and is not
- It is a regression and integrity check you can run in seconds and re-run in CI.
- It is not an independent third-party attestation: it runs in the same process as the code it checks.
- It does not cover live IDEs, live models, or your own repository's contents.
- The known-answer tables are finite. They prove the listed behaviours.
How an independent auditor can extend it
- Add rows to
KNOWNorINJECTinpackages/audit/audit.mjswith a ground-truth expectation. - Add a fixture repository with a known fact (a secret, a planted instruction, a drifting ADR).
- Run
npm run audit. A new failing row is a finding. - Publish the diff. Do not edit expectations to make a failure disappear.
Building trust over time
| Practice | Status |
|---|---|
| Public source, Apache-2.0 | Yes |
| Public CI on three operating systems | Yes (see Actions tab) |
| Reproducible evidence generated by script | Yes |
| Zero dependencies | Yes |
| Compatibility matrix with explicit "untested" | Yes |
| Independent third-party review | Not yet; contributions welcome |
| Signed releases and provenance | Planned |