Hermit
Isolated suite execution is implemented. CI currently validates and dry-runs the plan; automatic full-plan execution is pending.
Hermit creates an isolated Compose project, databases, network, fixtures, and evidence for every suite. It never reuses the normal kesita dev database and never calls paid providers.
./bin/hermit validate-plan tests/plans/platform/integration.yaml./bin/hermit run tests/plans/platform/integration.yaml# Focused AIX plan (includes assignments and runtime)./bin/hermit run tests/plans/platform/aix.yamlflowchart LR
Plan[Test plan] --> Hermit[Hermit]
subgraph Run[One isolated Compose project per suite]
Fixture[Migrators + synthetic fixtures]
Data[(PostgreSQL + Azurite)]
SUT[Selected system under test]
Sim[Simulation Platform]
Runner[Test runner]
Fixture --> Data
SUT --> Data
SUT -->|provider and fake Legal API calls| Sim
Runner -->|assertions| SUT
end
Hermit -->|create and configure| Run
Run --> Evidence[Results + redacted logs<br/>+ simulation journal]
Evidence --> Cleanup[Remove containers, volumes<br/>and generated images]
The Simulation Platform is a protocol-level test double for Anthropic, OpenAI, Voyage, Stripe, Resend, Document Intelligence and, when selected by a suite, the Legal API. Scenarios are explicit and unmatched requests fail. PostgreSQL, migrations, workload authentication and selected application binaries remain real.
The platform plan also starts the Kesita app for a real dependency check through Legal Core, and starts the public legal gateway for authenticated search and source requests through the running Legal Core API. CI validates and dry-runs the Hermit plan; it does not execute its suites automatically.
service selects the system under test. Depending on the suite, that is the Kesita app, Legal Core API, or real Go workflow binaries executed by the isolated runner. A suite can choose a real Legal Core API or the simulated Legal API; paid external providers always remain inside the simulator.
Plans live in tests/plans/, suites in tests/suites/, and synthetic data in tests/fixtures/. The suite catalog at tools/hermit/catalog.yaml defines available dependencies and scenarios.
Results are written below artifacts/hermit/. Check both test execution and cleanup before treating a run as successful. Failed artifacts may contain operational detail, so do not publish them as documentation.
Add a suite
Section titled “Add a suite”- Copy the closest existing suite and keep its command, dependencies, timeout, and fixture contract together.
- Use only synthetic, deterministic fixtures.
- Validate and run the suite locally.
- Add it to the relevant plan; a YAML file alone is not a gate. CI validates the plan without running its suites.
scripts/check-integration-coverage.mjs checks that app .integration.ts files and top-level Go tests in _integration_test.go files are selected by the platform plan. The script owns exclusions for helper entry points invoked by parent tests.
flowchart LR
TestCode[Test implementation] --> Suite[Suite definition]
Suite --> Plan[Plan selection]
Plan --> Validation[CI validation and dry run]
Plan --> Execution[Explicit suite execution]
Unit tests do not prove row-level security, database roles, service authentication, or recovery behavior. Use the integration plan for those boundaries. Browser suites select the Playwright runner in their Hermit definition; browsers and dependencies are installed in the runner image before its offline test phase.
Worker coordination has a dedicated cold PostgreSQL suite: ./bin/hermit test tests/suites/data-foundation/worker-coordination.yaml. It uses real worker subprocesses, including SIGKILL and SIGSTOP/SIGCONT, rather than mocking lease expiry alone. See worker recovery for the architecture, timing budgets, fencing guarantees, and failure cases. The suite is included in both the AIX and platform integration plans.
The suite YAML owns its test selection, required test count and timeouts. scripts/check-integration-coverage.mjs owns discovery/exclusions, and tests/assert-go-tests.sh rejects missing or skipped required tests. Consult those sources rather than copying counts or timing values into guides.