Independent-style audit and evidence

Reviewer stance. This is a skeptical review of the v0.1.0 starter and the v0.2.0 result, written from the evidence in the repository. It was prepared by the project's own tooling and author session, so it is not independent in the third-party sense and is not a certification. Where it says "verified", a command in this repository reproduces the result.

1. Method

  1. Baseline the inherited starter: run its tests and demo, read all source.
  2. Compare behaviour with the written specification (master prompt, tool catalogue, skill file).
  3. Probe edge cases directly (timestamps, secrets, hostile text, tampering, local API).
  4. Fix defects, add regression tests, add an automated audit that re-checks them on every run.
  5. Publish what is not covered.

2. Defects found in the starter and their resolution

#FindingEvidenceSeverityResolution
F1Date.parse accepted impossible dates: 2026-02-30T00:00:00Z parsed as 2026-03-02; T24:00:00Z rolled to next daydirect Node probeHigh (timeline integrity)Explicit calendar validation; 13 rejection tests
F2Millisecond-only time; no precision field; nanosecond order lost; OTLP timeUnixNano of 0 treated as a real timecode reviewMediumtimestampNs BigInt string, timePrecision, observed-time fallback with warning
F3CI ran node --test tests/*.test.mjs; Windows shells do not expand the glob on Node 20CI failure reportHigh (broken CI)node --test
F4Symlink tests threw EPERM on hosts that forbid symlinkslocal runMediumhelper returns early when creation is denied
F5All 20 skills were one template with a changed sentencefile comparisonMedium (misleading)Role-specific skills; uniqueness test; necessity catalogue
F6Redaction covered only bearer, a few key=value, email, AWS keycode reviewHigh (privacy)12 rules, 14-class corpus, counts, known gaps published
F7Authorization: Bearer x left a dangling Bearer after maskingprobeLowsingle combined rule; test
F8One unparseable file aborted the whole runcode reviewMediumreported as ingestion error; others continue
F9Nested symlinks skipped silentlycode reviewLow (hidden gap)listed in skippedInputs and a gap
F10Hypotheses ignored errors preceding a change and unaffected servicesfixture reviewHigh (causal safety)contradictions recorded per hypothesis
F11Only temporal and generic hypotheses; no dependency, saturation, datastore, latent familiesspec comparisonMediumfamilies added with counter-evidence
F12Reports: HTML plus a Markdown file mislabelled postmortem-draft.md; no MD/HTML audit reportfeedbackMediumincident-report.md, real postmortem template, audit writes MD, HTML and JSON
F13Local API accepted any Host headercode reviewMedium (DNS rebinding)Host allowlist, timeouts
F14Package marked private, no files, no repository metadatapackage.jsonMedium (not publishable)publish-ready metadata, provenance
F15Equal timestamps ordered by random-looking IDcode reviewLowsort by ns, file, record

3. Automated audit (20 checks)

Run npx incidentsutra audit --out audit. The committed run is in evidence/audit/audit-report.html (also .md, .json). Latest committed verdict: PASS_WITH_NOTES (19 pass, 1 skipped, 0 fail). The skipped check (A13, symlinks) is skipped on Windows hosts that forbid symlink creation; Linux and macOS CI exercise it.

AreaChecks
agentsA01 manifest, skills and role files agree; A02 guardrail wording in all 20
contractsA03 report, events and agent contract validate against schemas
determinismA04 identical bytes for identical input
acceptanceA05 time zones, correlation-only, unknown metrics, claims cite sources
privacyA06 14 secret classes never reach any output
securityA07 hostile markup, A08 injection corpus, A09 static scan, A11 local API
integrityA10 six tamper mutations detected
ingestA12 limits and strict timestamps, A13 symlinks
algorithmA14 26-case benchmark
legalA15 licence and notices, A16 overclaim scan
secretsA17 repository secret scan
supply chainA18 zero dependencies
safetyA19 read-only policy, A20 redaction helper

4. Benchmark

26 synthetic incident families, 36 expectations, all pass on the committed run (evidence/benchmark.json). Caveat: labels are written by the author and not independently validated. The result shows internal consistency, not accuracy on real incidents. No precision or recall figure is claimed.

5. Tests

See test-results.md: machine-written results from npm run test:report, including Node version, platform and commit. CI additionally runs the suite on Ubuntu and Windows (Node 20, 22, 24) and macOS (Node 24).

6. Sample proof

evidence/samples/incident-report.html is a full report from the synthetic fixture. It shows a deployment 2.7.0, an error before deployment 2.7.1, an alert, a dependency warning that appears after onset, and a metric rising from 0.1% to 8% errors. Hypotheses remain unconfirmed; contradictions are listed.

7. What was not tested (read this)

8. Residual risks

RiskEffectMitigation
Redaction misses a secretLeak in a shared reportsanitize upstream; review before sharing
Keyword classification misses unusual wordingMissing hypothesis familythe unknown family is always present; read the gaps
Over-reading hypothesesWrong fixwording ("may be related"), legend, human review status
Digest forgedUndetected editdo not treat as a signature; compare source hashes
Local token disclosureLocal read of reportloopback only, short-lived token

9. Verdict

For its stated scope (offline, read-only evidence assistant), the implementation meets its own acceptance criteria and has no known open defect after the fixes above. Suitability for production incident work depends on your data hygiene and process. A third-party review is recommended before any networked or multi-user extension.