Skip to content

Runtime evidence suite

scripts/runtime-evidence.sh collects evidence from a real SSH lab host and packages it for review, handover or release validation.

The suite runs the extended SSH smoke, exports the latest run as Markdown and JSON, creates a portable evidence ZIP, and writes checksums for the produced files. It is intentionally skipped when no runtime target is configured.

Required environment

AUTOMAX_SSH_HOST=192.0.2.10
AUTOMAX_SSH_USER=deploy

Recommended secure defaults:

AUTOMAX_SSH_KEY_FILE=~/.ssh/id_ed25519
AUTOMAX_SSH_HOST_KEY_POLICY=reject
AUTOMAX_SSH_KNOWN_HOSTS=~/.ssh/known_hosts

Useful output controls:

AUTOMAX_STATE_DIR=/tmp/automax-runtime-evidence-runs
AUTOMAX_RUNTIME_EVIDENCE_DIR=dist/runtime-evidence
AUTOMAX_RUNTIME_EVIDENCE_PREFIX=automax-runtime-evidence

Run the suite:

scripts/runtime-evidence.sh

A successful run produces files similar to:

dist/runtime-evidence/
  automax-runtime-evidence-<run-id>.md
  automax-runtime-evidence-<run-id>.json
  automax-runtime-evidence-<run-id>.zip
  automax-runtime-evidence-<run-id>-summary.txt
  SHA256SUMS

The script uses the same public run evidence commands that operators can run manually:

automax runs export <run-id> --output evidence.md
automax runs export <run-id> --format json --output evidence.json
automax runs events <run-id> --output events.jsonl
automax runs bundle <run-id> --output automax-evidence.zip

The ZIP bundle contains evidence.json, evidence.md, events/events.jsonl, events/events.json, an artifact manifest and captured artifact files.

Optional package smoke evidence

Set AUTOMAX_RUNTIME_EVIDENCE_PACKAGE=1 to run package build/install validation as part of the same evidence collection pass:

AUTOMAX_RUNTIME_EVIDENCE_PACKAGE=1 scripts/runtime-evidence.sh

The package smoke log is stored in the evidence directory and included in SHA256SUMS.

Index and retention

The runtime evidence script updates dist/runtime-evidence-index.json automatically after producing evidence. Set AUTOMAX_RUNTIME_EVIDENCE_RETAIN to keep only the latest N SSH evidence runs while refreshing the index:

AUTOMAX_RUNTIME_EVIDENCE_RETAIN=10 scripts/runtime-evidence.sh

Use scripts/runtime-evidence-index.sh --verify to validate existing checksums and rebuild the index without running a new SSH smoke.

CI behavior

The script exits with code 77 when AUTOMAX_SSH_HOST or AUTOMAX_SSH_USER is missing. CI jobs can therefore include it as an optional runtime validation step without requiring a persistent lab host.