283 lines
14 KiB
Markdown
283 lines
14 KiB
Markdown
# Orch Test Documentation Roadmap
|
|
|
|
## Purpose
|
|
|
|
This roadmap tracks the human-readable Markdown test plan for `orch`.
|
|
|
|
It exists so a new agent can immediately answer four questions without re-reading the whole codebase:
|
|
|
|
- which test-plan documents already exist
|
|
- which cases have already been written down
|
|
- which cases are still missing
|
|
- what file should be updated next
|
|
|
|
This roadmap is for the Markdown test-plan set under `docs/tests/orch/`.
|
|
It is not a replacement for automated Go tests.
|
|
|
|
## Current Snapshot
|
|
|
|
Snapshot date:
|
|
|
|
- `2026-03-19`
|
|
|
|
Current state:
|
|
|
|
- `orch` CLI is implemented for the current scheduler, strict worktree, wait, and council review surfaces
|
|
- automated Go integration tests already cover the main scheduler lifecycle, dependency gating, blocked-answer flow, worktree dispatch behavior, waits, retries, reassignments, cleanup, and council start/wait/tally/report flows
|
|
- this roadmap now exists under `docs/tests/orch/ROADMAP.md`
|
|
- the initial `docs/tests/orch/` skeleton now exists with shared, workflow, and per-command index documents
|
|
- command-case Markdown files have not been authored yet; the next work is filling the backlog in a stable order
|
|
|
|
Progress summary for planned test-plan documents, excluding `ROADMAP.md`:
|
|
|
|
- planned document files: `22`
|
|
- authored document files: `22`
|
|
- planned case slugs in this roadmap: `35`
|
|
- authored case slugs in this roadmap: `0`
|
|
|
|
## Scope
|
|
|
|
In scope:
|
|
|
|
- `orch run init`
|
|
- `orch run show`
|
|
- `orch task add`
|
|
- `orch dep add`
|
|
- `orch ready`
|
|
- `orch dispatch`
|
|
- `orch reconcile`
|
|
- `orch wait`
|
|
- `orch blocked`
|
|
- `orch answer`
|
|
- `orch retry`
|
|
- `orch reassign`
|
|
- `orch cancel`
|
|
- `orch cleanup`
|
|
- `orch status`
|
|
- `orch council start`
|
|
- `orch council wait`
|
|
- `orch council tally`
|
|
- `orch council report`
|
|
- cross-command workflows
|
|
- shared test conventions for JSON output, exit codes, temp databases, repo fixtures, and worktree assertions
|
|
|
|
Out of scope:
|
|
|
|
- `inbox` command behavior except as supporting setup for `orch` scenarios
|
|
- implementation details that are not visible through the `orch` CLI contract
|
|
- future `orch` commands that are not currently implemented
|
|
|
|
## Tracking Rules
|
|
|
|
Directory model:
|
|
|
|
- one folder per leaf command or shared area
|
|
- each folder keeps a `README.md` entrypoint
|
|
- command folders use `README.md` as an index only
|
|
- each command case lives in its own Markdown file named after the case slug
|
|
- cross-command workflow cases remain grouped in `docs/tests/orch/workflows/README.md`
|
|
|
|
Case identity:
|
|
|
|
- do not use numeric IDs
|
|
- identify each command case by its concrete file path
|
|
- identify each workflow case by `path + case slug`
|
|
- command case file naming pattern:
|
|
|
|
```text
|
|
<case-slug>.md
|
|
```
|
|
|
|
- workflow case heading pattern:
|
|
|
|
```md
|
|
## case: run-dispatch-reconcile-status-happy-path
|
|
```
|
|
|
|
Per-case structure inside the case document:
|
|
|
|
- `用例意义`
|
|
- `前置条件`
|
|
- `输入`
|
|
- `预期输出`
|
|
- `断言结论`
|
|
|
|
How to update this roadmap when a new case is written:
|
|
|
|
1. if it is a command case, create or update the target `<case-slug>.md` file under the relevant leaf-command folder
|
|
2. if it is a command case, add or update the entry in that folder `README.md` index
|
|
3. if it is a workflow case, add or update the case inside `docs/tests/orch/workflows/README.md`
|
|
4. move the case slug from `Pending Case Backlog` to `Authored Case Register`
|
|
5. update the authored counts in `Current Snapshot`
|
|
6. if a new Markdown file is created, update `Document Progress`
|
|
|
|
Allowed status values in this roadmap:
|
|
|
|
- `pending`
|
|
- `in_progress`
|
|
- `done`
|
|
- `deferred`
|
|
|
|
## Existing Automated Coverage Reference
|
|
|
|
The Markdown test-plan set starts at zero, but these automated tests already exist and should be used as source material when writing the docs:
|
|
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L12) `TestOrchRunDispatchReconcileLifecycle`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L184) `TestOrchDependencyBlockedAndAnswerFlow`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L458) `TestOrchDispatchRejectsNonReadyTask`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L512) `TestOrchDispatchCreatesStrictWorktree`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L631) `TestOrchStrictWorktreeRejectsDirtyRepoWithoutBaseRef`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L680) `TestOrchStrictWorktreeAllowsExplicitBaseRefOnDirtyRepo`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L733) `TestOrchDispatchAutoEnablesWorktreeForCodeLikeTask`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L782) `TestOrchDispatchDoesNotAutoEnableWorktreeForNonCodeTask`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L829) `TestOrchWaitWakesOnBlockedEvent`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L944) `TestOrchWaitTimesOutWithoutMatchingEvent`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L981) `TestOrchRetryCreatesNewAttempt`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1079) `TestOrchReassignCancelsOldThreadAndDispatchesNewAttempt`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1186) `TestOrchCancelTaskAndRun`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1315) `TestOrchCleanupRemovesCompletedWorktree`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1402) `TestOrchCouncilStartDispatchesThreeReviewers`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1545) `TestOrchCouncilWaitWakesWhenAllReviewersComplete`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1678) `TestOrchCouncilWaitTimesOutWhenReviewersIncomplete`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1715) `TestOrchCouncilTallyGroupsReviewerFindingsNormal`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1813) `TestOrchCouncilTallyStrictKeepsDistinctProposals`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1873) `TestOrchCouncilReportDefaultShowsConsensusAndMajority`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1950) `TestOrchCouncilReportShowAllIncludesMinority`
|
|
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1979) `TestOrchCouncilReportJSONShape`
|
|
|
|
These tests do not remove the need for the Markdown plan. They only reduce discovery work.
|
|
|
|
## Planned Directory Tree
|
|
|
|
```text
|
|
docs/tests/orch/
|
|
ROADMAP.md
|
|
README.md
|
|
_shared/
|
|
README.md
|
|
workflows/
|
|
README.md
|
|
run-init/
|
|
README.md
|
|
run-show/
|
|
README.md
|
|
task-add/
|
|
README.md
|
|
dep-add/
|
|
README.md
|
|
ready/
|
|
README.md
|
|
dispatch/
|
|
README.md
|
|
reconcile/
|
|
README.md
|
|
wait/
|
|
README.md
|
|
blocked/
|
|
README.md
|
|
answer/
|
|
README.md
|
|
retry/
|
|
README.md
|
|
reassign/
|
|
README.md
|
|
cancel/
|
|
README.md
|
|
cleanup/
|
|
README.md
|
|
status/
|
|
README.md
|
|
council-start/
|
|
README.md
|
|
council-wait/
|
|
README.md
|
|
council-tally/
|
|
README.md
|
|
council-report/
|
|
README.md
|
|
```
|
|
|
|
## Document Progress
|
|
|
|
| Path | Purpose | Planned Cases | Authored Cases | Status |
|
|
| --- | --- | ---: | ---: | --- |
|
|
| `docs/tests/orch/README.md` | Global testing conventions and glossary | 0 | 0 | done |
|
|
| `docs/tests/orch/_shared/README.md` | Shared fixtures, JSON assertions, exit-code rules, and worktree conventions | 0 | 0 | done |
|
|
| `docs/tests/orch/workflows/README.md` | Cross-command scenarios | 4 | 0 | done |
|
|
| `docs/tests/orch/run-init/README.md` | `run init` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/run-show/README.md` | `run show` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/task-add/README.md` | `task add` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/dep-add/README.md` | `dep add` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/ready/README.md` | `ready` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/dispatch/README.md` | `dispatch` command case index | 6 | 0 | done |
|
|
| `docs/tests/orch/reconcile/README.md` | `reconcile` command case index | 2 | 0 | done |
|
|
| `docs/tests/orch/wait/README.md` | `wait` command case index | 2 | 0 | done |
|
|
| `docs/tests/orch/blocked/README.md` | `blocked` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/answer/README.md` | `answer` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/retry/README.md` | `retry` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/reassign/README.md` | `reassign` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/cancel/README.md` | `cancel` command case index | 2 | 0 | done |
|
|
| `docs/tests/orch/cleanup/README.md` | `cleanup` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/status/README.md` | `status` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/council-start/README.md` | `council start` command case index | 1 | 0 | done |
|
|
| `docs/tests/orch/council-wait/README.md` | `council wait` command case index | 2 | 0 | done |
|
|
| `docs/tests/orch/council-tally/README.md` | `council tally` command case index | 2 | 0 | done |
|
|
| `docs/tests/orch/council-report/README.md` | `council report` command case index | 3 | 0 | done |
|
|
|
|
## Authoring Order
|
|
|
|
1. global conventions in `docs/tests/orch/README.md`
|
|
2. shared fixtures and assertion helpers in `docs/tests/orch/_shared/README.md`
|
|
3. workflow cases in `docs/tests/orch/workflows/README.md`
|
|
4. core scheduler command docs: `run-init`, `task-add`, `dep-add`, `ready`, `dispatch`, `reconcile`, `status`
|
|
5. interactive leader command docs: `wait`, `blocked`, `answer`, `retry`, `reassign`, `cancel`, `cleanup`
|
|
6. council workflow docs: `council-start`, `council-wait`, `council-tally`, `council-report`
|
|
|
|
## Pending Case Backlog
|
|
|
|
### Workflows
|
|
|
|
- `docs/tests/orch/workflows/README.md :: run-dispatch-reconcile-status-happy-path` - end-to-end happy path from run creation through final `status`
|
|
- `docs/tests/orch/workflows/README.md :: dependency-blocked-answer-resume-flow` - dependency gating plus blocked question and answer flow
|
|
- `docs/tests/orch/workflows/README.md :: strict-worktree-dispatch-to-cleanup` - worktree creation through cleanup of a completed attempt
|
|
- `docs/tests/orch/workflows/README.md :: council-review-end-to-end` - council start, wait, tally, and report sequence
|
|
|
|
### Command Cases
|
|
|
|
- `docs/tests/orch/run-init/run-init-creates-new-run.md` - create a run with goal and optional summary
|
|
- `docs/tests/orch/run-show/run-show-returns-run-summary-and-task-counts.md` - show aggregate run metadata after activity
|
|
- `docs/tests/orch/task-add/task-add-creates-ready-root-task.md` - adding a dependency-free task becomes `ready`
|
|
- `docs/tests/orch/dep-add/dep-add-blocks-dependent-task-until-prerequisite-completes.md` - dependency edge prevents immediate readiness
|
|
- `docs/tests/orch/ready/ready-lists-only-eligible-tasks.md` - ready list excludes blocked by dependency tasks
|
|
- `docs/tests/orch/dispatch/dispatch-creates-attempt-and-thread-for-ready-task.md` - happy-path dispatch with new attempt metadata
|
|
- `docs/tests/orch/dispatch/dispatch-rejects-non-ready-task.md` - invalid-state rejection when dispatching a gated task
|
|
- `docs/tests/orch/dispatch/dispatch-creates-strict-worktree.md` - explicit strict worktree dispatch persists workspace metadata
|
|
- `docs/tests/orch/dispatch/dispatch-rejects-dirty-repo-without-base-ref.md` - strict worktree guard on dirty repository state
|
|
- `docs/tests/orch/dispatch/dispatch-allows-explicit-base-ref-on-dirty-repo.md` - explicit base ref bypasses dirty working tree rejection
|
|
- `docs/tests/orch/dispatch/dispatch-auto-enables-worktree-for-code-like-task.md` - inferred worktree mode for code-like tasks
|
|
- `docs/tests/orch/dispatch/dispatch-skips-auto-worktree-for-non-code-task.md` - non-code tasks stay on the normal dispatch path
|
|
- `docs/tests/orch/reconcile/reconcile-maps-claimed-or-in-progress-thread-to-running.md` - reconcile turns claimed/in-progress inbox state into running task state
|
|
- `docs/tests/orch/reconcile/reconcile-maps-done-or-failed-thread-to-terminal-task-state.md` - reconcile turns terminal inbox state into task terminal state
|
|
- `docs/tests/orch/wait/wait-wakes-on-matching-run-event.md` - wait wakes on a later matching task event
|
|
- `docs/tests/orch/wait/wait-times-out-without-matching-event.md` - wait returns a non-woken timeout result
|
|
- `docs/tests/orch/blocked/blocked-lists-latest-question-for-blocked-task.md` - blocked list includes the active question payload
|
|
- `docs/tests/orch/answer/answer-appends-answer-to-active-thread.md` - answer writes an inbox answer message for the blocked attempt
|
|
- `docs/tests/orch/retry/retry-creates-new-attempt-for-failed-task.md` - retry creates a successor attempt after failure
|
|
- `docs/tests/orch/reassign/reassign-cancels-old-thread-and-dispatches-new-attempt.md` - reassign moves work to a new worker and attempt
|
|
- `docs/tests/orch/cancel/cancel-cancels-single-task.md` - cancel transitions one task to `cancelled`
|
|
- `docs/tests/orch/cancel/cancel-cancels-entire-run.md` - cancel cascades terminal state across the run
|
|
- `docs/tests/orch/cleanup/cleanup-removes-completed-worktree.md` - cleanup removes a completed attempt worktree
|
|
- `docs/tests/orch/status/status-returns-run-summary-and-task-list.md` - status returns run aggregate state and per-task statuses
|
|
- `docs/tests/orch/council-start/council-start-dispatches-three-reviewers.md` - council start creates reviewer tasks and metadata
|
|
- `docs/tests/orch/council-wait/council-wait-wakes-when-all-reviewers-complete.md` - council wait completes successfully when all reviewers finish
|
|
- `docs/tests/orch/council-wait/council-wait-times-out-when-reviewers-incomplete.md` - council wait timeout behavior
|
|
- `docs/tests/orch/council-tally/council-tally-groups-reviewer-findings-in-normal-mode.md` - normal similarity grouping behavior
|
|
- `docs/tests/orch/council-tally/council-tally-keeps-distinct-proposals-in-strict-mode.md` - strict similarity grouping behavior
|
|
- `docs/tests/orch/council-report/council-report-defaults-to-consensus-and-majority.md` - default report buckets and markdown artifact behavior
|
|
- `docs/tests/orch/council-report/council-report-show-all-includes-minority.md` - `--show all` includes minority output
|
|
- `docs/tests/orch/council-report/council-report-json-shape-is-stable.md` - JSON output shape and artifact metadata contract
|
|
|
|
## Authored Case Register
|
|
|
|
- none yet
|