ContextSutra AI

09. Audit and trust

The self-audit

npx contextsutra audit --out audit runs a fixed list of checks against the tool itself and writes audit.json, audit.md and audit.html. Every check shows its expected and observed value, so a reader can see not just "pass" but what was measured.

CategoryWhat it proves
Redaction accuracyA known-answer table of 21 lines (15 secret shapes that must be redacted, 6 harmless lines that must not be) gives zero misses and zero false alarms; multi-line private keys are dropped; secret files never enter a capsule
Prompt injectionA table of instruction-like lines is classified correctly; a hostile README line is quarantined and absent from the render
IntegrityTwo scans are byte-identical; tampering is detected; digest ignores key order; malformed input is rejected
FreshnessEdit, removal and addition of documents are each reported; diff is exact
Path safetyTraversal and absolute paths refused; symlinks refused where the OS allows creating them
Review gateNever certifies safe; BLOCKs on residual secrets; surfaces conflicts; approves only explicit statuses
ServerLoopback bind, token required, methods restricted, foreign Host refused, short token refused
Supply chainNo secrets in code, no outbound-network or process-spawn APIs, zero dependencies, no install scripts, HTML escaped
Skills12 present, valid, complete, distinct, and calling only real commands
LegalLicense, notices and community files present; no forbidden marketing claims

Reading the report

Open audit.html. Each category card shows passed over total. The table lists every check. Status is text plus a symbol, never colour alone. A failing check includes evidence text naming what failed.

What the audit is, and is not

How an independent auditor can extend it

  1. Add rows to KNOWN or INJECT in packages/audit/audit.mjs with a ground-truth expectation.
  2. Add a fixture repository with a known fact (a secret, a planted instruction, a drifting ADR).
  3. Run npm run audit. A new failing row is a finding.
  4. Publish the diff. Do not edit expectations to make a failure disappear.

Building trust over time

PracticeStatus
Public source, Apache-2.0Yes
Public CI on three operating systemsYes (see Actions tab)
Reproducible evidence generated by scriptYes
Zero dependenciesYes
Compatibility matrix with explicit "untested"Yes
Independent third-party reviewNot yet; contributions welcome
Signed releases and provenancePlanned