175 lines
8.2 KiB
Markdown
175 lines
8.2 KiB
Markdown
# Council Review Skill Test Plan
|
|
|
|
## Purpose
|
|
|
|
This directory tracks human-readable test plans for the `skills/council-review/` Codex skill bundle.
|
|
|
|
These documents are not command-contract specs for the `orch council` CLI itself.
|
|
That coverage already lives under [../orch/](../orch/).
|
|
|
|
This directory exists to describe a different test surface:
|
|
|
|
- whether a leader agent can actually use the packaged `council-review` skill
|
|
- whether the bundled `./assets/orch` CLI works inside real skill-guided council workflows
|
|
- whether a council run driven by the skill reaches the expected reviewer, grouping, tally, and report state
|
|
|
|
## Test Model
|
|
|
|
- `README.md` is the index for this directory
|
|
- each skill test case lives in its own Markdown file
|
|
- use stable case slugs in filenames
|
|
|
|
## Shared Execution Contract
|
|
|
|
Use these defaults unless a case file explicitly overrides them:
|
|
|
|
- run the scenario with real subagents, not simulated transcripts
|
|
- inject `skills/council-review/` into the leader agent
|
|
- inject `skills/inbox/` into reviewer agents whenever reviewer task completion is required
|
|
- initialize the shared SQLite DB before launching role agents with `INBOX_SKILL_PATH/assets/inbox --db TMPDIR/coord.db --json init`
|
|
- require the leader to coordinate through the bundled `./assets/orch` CLI from the council-review skill instead of ordinary chat
|
|
- require reviewer agents to coordinate through the bundled `./assets/inbox` CLI from their skill instead of ordinary chat
|
|
- validate final council run, reviewer task state, and report state independently from the main thread after the agents stop
|
|
- create any required repo fixture before launching agents for mixed or repo-target cases
|
|
|
|
## How An Agent Runs These Cases
|
|
|
|
Use one test-runner agent to execute each case.
|
|
|
|
The test-runner agent is responsible for:
|
|
|
|
- reading this `README.md` first, then one specific case file
|
|
- creating an isolated temporary directory and DB path for that run
|
|
- initializing the DB once through the bundled inbox CLI before launching role agents
|
|
- creating any required temporary Git repo fixture before launching role agents
|
|
- launching the role agents described in `Agent Topology`
|
|
- injecting `skills/council-review/` into the leader and `skills/inbox/` into reviewers
|
|
- passing each role agent the prompt text from the case file with concrete values substituted for `COUNCIL_SKILL_PATH`, `INBOX_SKILL_PATH`, `TMPDIR`, `RUN_ID`, `THREAD_ID`, and `REPORT_PATH` when needed
|
|
- coordinating launch order or parallel start according to the case file
|
|
- collecting agent final summaries as evidence
|
|
- resolving final run ids, thread ids, and report artifact paths from agent outputs
|
|
- running the `Validation Commands` from the main thread after the role agents stop
|
|
- comparing the observed results against `Expected Outcomes` and `Assertions`
|
|
- returning a final pass/fail judgment with concrete evidence
|
|
|
|
The role agents are responsible for:
|
|
|
|
- acting only within the role assigned in the case file
|
|
- using the injected skill bundle rather than ad hoc repository discovery
|
|
- coordinating through the bundled CLI and shared DB
|
|
- reporting concrete run ids, thread ids, report artifact paths, and key command outcomes back to the test-runner agent
|
|
|
|
The test-runner agent should treat a case as passed only when:
|
|
|
|
- all role agents reach a final state without violating the case contract
|
|
- the independent validation commands succeed
|
|
- the final council, orch, and inbox state matches the assertions in the case file
|
|
|
|
The test-runner agent should treat a case as failed when:
|
|
|
|
- any required agent times out or stalls
|
|
- a required council, orch, or inbox action is skipped
|
|
- the leader falls back to ordinary chat for workflow control that should go through the bundled council-review skill
|
|
- reviewer agents fall back to ordinary chat instead of returning results through inbox
|
|
- the final council grouping, summary, or report state conflicts with the documented assertions
|
|
|
|
The test-runner agent should report results in this shape:
|
|
|
|
- `case`
|
|
- `db_path`
|
|
- `run_id`
|
|
- `thread_ids`
|
|
- `report_paths`
|
|
- `result`: `pass` or `fail`
|
|
- `agent_summaries`
|
|
- `validation_evidence`
|
|
- `assertion_checklist`
|
|
- `notes`
|
|
|
|
## Default Timeouts
|
|
|
|
Use these defaults unless a case file explicitly overrides them:
|
|
|
|
- per-agent timeout: `4m`
|
|
- overall scenario timeout: `6m`
|
|
- async wait margin for the main thread: `45s`
|
|
|
|
## Default Failure Conditions
|
|
|
|
Treat the test as failed if any of the following happens:
|
|
|
|
- any required agent does not reach a final state before timeout
|
|
- any required council, orch, or inbox command returns a non-success result unless the case expects that failure
|
|
- the final `council report --json` output does not match the expected grouped recommendations
|
|
- the final `orch status` output does not match the expected reviewer task state
|
|
- a required markdown report artifact is missing when the case expects one
|
|
- the agents fall back to ordinary chat for critical coordination instead of the bundled CLIs
|
|
|
|
## Evidence Capture
|
|
|
|
Collect at least the following artifacts for every run:
|
|
|
|
- agent final summaries
|
|
- final `council report --json` output when the case reaches report stage
|
|
- final `orch status --run RUN_ID --json` output
|
|
- final `inbox show --thread THREAD_ID --json` output for every relevant reviewer thread when reviewers participated
|
|
- any `council wait` or `council tally` output relevant to the case
|
|
- the temporary DB path, resolved run id, resolved thread ids, and any report artifact paths
|
|
|
|
## Cleanup Policy
|
|
|
|
Use these defaults unless a case file explicitly overrides them:
|
|
|
|
- keep the temporary DB, repo fixture, and working directory on failure for debugging
|
|
- cleanup the temporary working directory on success only if the caller does not need replay artifacts
|
|
|
|
## Per-Case Template
|
|
|
|
Each case file should use this structure:
|
|
|
|
- `Test Type`
|
|
- `Purpose`
|
|
- `Preconditions`
|
|
- `Agent Topology`
|
|
- `Inputs`
|
|
- `Execution Parameters`
|
|
- `Execution Steps`
|
|
- `Validation Commands`
|
|
- `Expected Outcomes`
|
|
- `Assertions`
|
|
- `Cleanup`
|
|
- `Recorded Example Run` when a real run has already been captured
|
|
|
|
## Case Files
|
|
|
|
| Case Slug | File | Coverage Note |
|
|
| --- | --- | --- |
|
|
| `council-brainstorm-end-to-end-through-bundled-cli` | [council-brainstorm-end-to-end-through-bundled-cli.md](./council-brainstorm-end-to-end-through-bundled-cli.md) | validates that the council-review skill can drive `start -> wait -> tally -> report` with three real reviewer agents |
|
|
| `council-unanimous-only-default-report-through-bundled-cli` | [council-unanimous-only-default-report-through-bundled-cli.md](./council-unanimous-only-default-report-through-bundled-cli.md) | validates that unanimous-only runs default to `consensus` output while preserving the underlying summary counts |
|
|
| `council-wait-timeout-through-bundled-cli` | [council-wait-timeout-through-bundled-cli.md](./council-wait-timeout-through-bundled-cli.md) | validates that the leader sees the expected timeout contract when reviewer tasks do not complete |
|
|
| `council-report-rejects-before-tally-through-bundled-cli` | [council-report-rejects-before-tally-through-bundled-cli.md](./council-report-rejects-before-tally-through-bundled-cli.md) | validates that the skill surfaces the stable invalid-state error when report is attempted before tally |
|
|
|
|
## Scope
|
|
|
|
In scope:
|
|
|
|
- explicit `$council-review` skill invocation
|
|
- bundled `./assets/orch` CLI usage for `orch council ...`
|
|
- end-to-end council start, wait, tally, and report flows
|
|
- interaction between a leader using `skills/council-review/` and reviewers using `skills/inbox/`
|
|
- default report policy, unanimous-only behavior, and timeout/error-path validation
|
|
|
|
Out of scope:
|
|
|
|
- per-command flag and JSON contract coverage for `orch council`
|
|
- generic leader orchestration flows that already belong under [../orch-skill/](../orch-skill/)
|
|
- worker-only skill behavior that belongs under [../inbox-skill/](../inbox-skill/)
|
|
- implicit skill triggering without `$council-review`
|
|
|
|
## Relationship To Other Test Docs
|
|
|
|
- [../orch/](../orch/) covers CLI command behavior
|
|
- [../orch-skill/](../orch-skill/) covers generic leader-side orchestration behavior on top of `orch`
|
|
- [../inbox-skill/](../inbox-skill/) covers worker-side skill-guided behavior on top of inbox
|
|
- this directory covers the separate user-facing `council-review` skill on top of `orch council`
|