28 KiB
28 KiB
Orch Test Documentation Roadmap
Purpose
This roadmap tracks the human-readable Markdown test plan for orch.
It exists so a new agent can immediately answer four questions without re-reading the whole codebase:
- which test-plan documents already exist
- which cases have already been written down
- which cases are still missing
- what file should be updated next
This roadmap is for the Markdown test-plan set under docs/tests/orch/.
It is not a replacement for automated Go tests.
Current Snapshot
Snapshot date:
2026-03-19
Current state:
orchCLI is implemented for the current scheduler, strict worktree, wait, and council review surfaces- automated Go tests now cover every currently documented
orchcommand case and workflow case, combining the original integration suite with focused contract tests for run/task/ready/dispatch/blocked/answer/cleanup/status/reconcile/workflow/council-report edges - this roadmap now exists under
docs/tests/orch/ROADMAP.md - all planned global, shared, workflow, command-index, and command-case Markdown documents in the current
orchtest-plan set have been authored - every implemented
orchleaf-command folder now usesREADME.mdas an index plus one Markdown file per planned case - workflow cases now exist in
docs/tests/orch/workflows/README.md, and the automated suite now explicitly covers both command-level contracts and the remaining end-to-end workflow gaps
Progress summary for planned test-plan documents, excluding ROADMAP.md:
- planned document files:
64 - authored document files:
64 - planned case slugs in this roadmap:
46 - authored case slugs in this roadmap:
46
Scope
In scope:
orch run initorch run showorch task addorch dep addorch readyorch dispatchorch reconcileorch waitorch blockedorch answerorch retryorch reassignorch cancelorch cleanuporch statusorch council startorch council waitorch council tallyorch council report- cross-command workflows
- shared test conventions for JSON output, exit codes, temp databases, repo fixtures, and worktree assertions
Out of scope:
inboxcommand behavior except as supporting setup fororchscenarios- implementation details that are not visible through the
orchCLI contract - future
orchcommands that are not currently implemented
Tracking Rules
Directory model:
- one folder per leaf command or shared area
- each folder keeps a
README.mdentrypoint - command folders use
README.mdas an index only - each command case lives in its own Markdown file named after the case slug
- cross-command workflow cases remain grouped in
docs/tests/orch/workflows/README.md
Case identity:
- do not use numeric IDs
- identify each command case by its concrete file path
- identify each workflow case by
path + case slug - command case file naming pattern:
<case-slug>.md
- workflow case heading pattern:
## case: run-dispatch-reconcile-status-happy-path
Per-case structure inside the case document:
用例意义前置条件输入预期输出断言结论
How to update this roadmap when a new case is written:
- if it is a command case, create or update the target
<case-slug>.mdfile under the relevant leaf-command folder - if it is a command case, add or update the entry in that folder
README.mdindex - if it is a workflow case, add or update the case inside
docs/tests/orch/workflows/README.md - move the case slug from
Pending Case BacklogtoAuthored Case Register - update the authored counts in
Current Snapshot - if a new Markdown file is created, update
Document Progress
Allowed status values in this roadmap:
pendingin_progressdonedeferred
Existing Automated Coverage Reference
The Markdown test-plan set starts at zero, but these automated tests already exist and should be used as source material when writing the docs:
- integration_test.go
TestOrchRunDispatchReconcileLifecycle - integration_test.go
TestOrchDependencyBlockedAndAnswerFlow - integration_test.go
TestOrchDispatchRejectsNonReadyTask - integration_test.go
TestOrchDispatchCreatesStrictWorktree - integration_test.go
TestOrchStrictWorktreeRejectsDirtyRepoWithoutBaseRef - integration_test.go
TestOrchStrictWorktreeAllowsExplicitBaseRefOnDirtyRepo - integration_test.go
TestOrchDispatchAutoEnablesWorktreeForCodeLikeTask - integration_test.go
TestOrchDispatchDoesNotAutoEnableWorktreeForNonCodeTask - integration_test.go
TestOrchWaitWakesOnBlockedEvent - integration_test.go
TestOrchWaitTimesOutWithoutMatchingEvent - integration_test.go
TestOrchRetryCreatesNewAttempt - integration_test.go
TestOrchReassignCancelsOldThreadAndDispatchesNewAttempt - integration_test.go
TestOrchCancelTaskAndRun - integration_test.go
TestOrchCleanupRemovesCompletedWorktree - integration_test.go
TestOrchCouncilStartDispatchesThreeReviewers - integration_test.go
TestOrchCouncilWaitWakesWhenAllReviewersComplete - integration_test.go
TestOrchCouncilWaitTimesOutWhenReviewersIncomplete - integration_test.go
TestOrchCouncilTallyGroupsReviewerFindingsNormal - integration_test.go
TestOrchCouncilTallyStrictKeepsDistinctProposals - integration_test.go
TestOrchCouncilReportDefaultShowsConsensusAndMajority - integration_test.go
TestOrchCouncilReportShowAllIncludesMinority - integration_test.go
TestOrchCouncilReportJSONShape - command_contracts_core_test.go
TestOrchRunShowReturnsRunSummaryAndTaskCounts - command_contracts_core_test.go
TestOrchRunShowRejectsMissingRun - command_contracts_core_test.go
TestOrchTaskAddRejectsInvalidAcceptanceJSON - command_contracts_core_test.go
TestOrchTaskAddRejectsInvalidPriority - command_contracts_core_test.go
TestOrchReadyOrdersByPriorityAndRespectsLimit - command_contracts_edges_test.go
TestOrchAnswerAcceptsPayloadJSONWithoutBody - command_contracts_edges_test.go
TestOrchAnswerRejectsEmptyBodyAndPayload - command_contracts_edges_test.go
TestOrchCleanupRejectsAttemptWithoutTask - command_contracts_edges_test.go
TestOrchCleanupReturnsNoMatchingWorkWhenFiltersMiss - command_contracts_remaining_test.go
TestOrchRunInitCreatesNewRun - command_contracts_remaining_test.go
TestOrchDispatchCreatesAttemptAndThreadForReadyTask - command_contracts_remaining_test.go
TestOrchBlockedListsLatestQuestionForBlockedTask - command_contracts_remaining_test.go
TestOrchStatusReturnsRunSummaryAndTaskList - command_contracts_remaining_test.go
TestOrchReconcileMapsFailedThreadToTerminalTaskState - command_contracts_remaining_test.go
TestOrchWorkflowStrictWorktreeDispatchToCleanup - command_contracts_remaining_test.go
TestOrchWorkflowCouncilReviewEndToEnd - council_report_contracts_test.go
TestOrchCouncilReportRejectsBeforeTally - council_report_contracts_test.go
TestOrchCouncilReportRejectsInvalidShow - council_report_contracts_test.go
TestOrchCouncilReportDefaultsToConsensusForOnlyUnanimousRun
These tests do not remove the need for the Markdown plan. They only reduce discovery work.
Planned Directory Tree
docs/tests/orch/
ROADMAP.md
README.md
_shared/
README.md
workflows/
README.md
run-init/
README.md
run-show/
README.md
task-add/
README.md
dep-add/
README.md
ready/
README.md
dispatch/
README.md
reconcile/
README.md
wait/
README.md
blocked/
README.md
answer/
README.md
retry/
README.md
reassign/
README.md
cancel/
README.md
cleanup/
README.md
status/
README.md
council-start/
README.md
council-wait/
README.md
council-tally/
README.md
council-report/
README.md
Document Progress
| Path | Purpose | Planned Cases | Authored Cases | Status |
|---|---|---|---|---|
docs/tests/orch/README.md |
Global testing conventions and glossary | 0 | 0 | done |
docs/tests/orch/_shared/README.md |
Shared fixtures, JSON assertions, exit-code rules, and worktree conventions | 0 | 0 | done |
docs/tests/orch/workflows/README.md |
Cross-command scenarios | 4 | 4 | done |
docs/tests/orch/run-init/README.md |
run init command case index |
0 | 0 | done |
docs/tests/orch/run-init/run-init-creates-new-run.md |
run init command case |
1 | 1 | done |
docs/tests/orch/run-show/README.md |
run show command case index |
0 | 0 | done |
docs/tests/orch/run-show/run-show-returns-run-summary-and-task-counts.md |
run show command case |
1 | 1 | done |
docs/tests/orch/task-add/README.md |
task add command case index |
0 | 0 | done |
docs/tests/orch/task-add/task-add-creates-ready-root-task.md |
task add command case |
1 | 1 | done |
docs/tests/orch/task-add/task-add-rejects-invalid-acceptance-json.md |
task add command case |
1 | 1 | done |
docs/tests/orch/task-add/task-add-rejects-invalid-priority.md |
task add command case |
1 | 1 | done |
docs/tests/orch/dep-add/README.md |
dep add command case index |
0 | 0 | done |
docs/tests/orch/dep-add/dep-add-blocks-dependent-task-until-prerequisite-completes.md |
dep add command case |
1 | 1 | done |
docs/tests/orch/ready/README.md |
ready command case index |
0 | 0 | done |
docs/tests/orch/ready/ready-lists-only-eligible-tasks.md |
ready command case |
1 | 1 | done |
docs/tests/orch/ready/ready-orders-by-priority-and-respects-limit.md |
ready command case |
1 | 1 | done |
docs/tests/orch/dispatch/README.md |
dispatch command case index |
0 | 0 | done |
docs/tests/orch/dispatch/dispatch-creates-attempt-and-thread-for-ready-task.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-rejects-non-ready-task.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-creates-strict-worktree.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-rejects-dirty-repo-without-base-ref.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-allows-explicit-base-ref-on-dirty-repo.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-auto-enables-worktree-for-code-like-task.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/dispatch/dispatch-skips-auto-worktree-for-non-code-task.md |
dispatch command case |
1 | 1 | done |
docs/tests/orch/reconcile/README.md |
reconcile command case index |
0 | 0 | done |
docs/tests/orch/reconcile/reconcile-maps-claimed-or-in-progress-thread-to-running.md |
reconcile command case |
1 | 1 | done |
docs/tests/orch/reconcile/reconcile-maps-done-or-failed-thread-to-terminal-task-state.md |
reconcile command case |
1 | 1 | done |
docs/tests/orch/wait/README.md |
wait command case index |
0 | 0 | done |
docs/tests/orch/wait/wait-wakes-on-matching-run-event.md |
wait command case |
1 | 1 | done |
docs/tests/orch/wait/wait-times-out-without-matching-event.md |
wait command case |
1 | 1 | done |
docs/tests/orch/blocked/README.md |
blocked command case index |
0 | 0 | done |
docs/tests/orch/blocked/blocked-lists-latest-question-for-blocked-task.md |
blocked command case |
1 | 1 | done |
docs/tests/orch/answer/README.md |
answer command case index |
0 | 0 | done |
docs/tests/orch/answer/answer-appends-answer-to-active-thread.md |
answer command case |
1 | 1 | done |
docs/tests/orch/answer/answer-accepts-payload-json-without-body.md |
answer command case |
1 | 1 | done |
docs/tests/orch/answer/answer-rejects-empty-body-and-payload.md |
answer command case |
1 | 1 | done |
docs/tests/orch/retry/README.md |
retry command case index |
0 | 0 | done |
docs/tests/orch/retry/retry-creates-new-attempt-for-failed-task.md |
retry command case |
1 | 1 | done |
docs/tests/orch/reassign/README.md |
reassign command case index |
0 | 0 | done |
docs/tests/orch/reassign/reassign-cancels-old-thread-and-dispatches-new-attempt.md |
reassign command case |
1 | 1 | done |
docs/tests/orch/cancel/README.md |
cancel command case index |
0 | 0 | done |
docs/tests/orch/cancel/cancel-cancels-single-task.md |
cancel command case |
1 | 1 | done |
docs/tests/orch/cancel/cancel-cancels-entire-run.md |
cancel command case |
1 | 1 | done |
docs/tests/orch/cleanup/README.md |
cleanup command case index |
0 | 0 | done |
docs/tests/orch/cleanup/cleanup-removes-completed-worktree.md |
cleanup command case |
1 | 1 | done |
docs/tests/orch/cleanup/cleanup-rejects-attempt-without-task.md |
cleanup command case |
1 | 1 | done |
docs/tests/orch/cleanup/cleanup-returns-no-matching-work-when-filters-miss.md |
cleanup command case |
1 | 1 | done |
docs/tests/orch/status/README.md |
status command case index |
0 | 0 | done |
docs/tests/orch/status/status-returns-run-summary-and-task-list.md |
status command case |
1 | 1 | done |
docs/tests/orch/council-start/README.md |
council start command case index |
0 | 0 | done |
docs/tests/orch/council-start/council-start-dispatches-three-reviewers.md |
council start command case |
1 | 1 | done |
docs/tests/orch/council-wait/README.md |
council wait command case index |
0 | 0 | done |
docs/tests/orch/council-wait/council-wait-wakes-when-all-reviewers-complete.md |
council wait command case |
1 | 1 | done |
docs/tests/orch/council-wait/council-wait-times-out-when-reviewers-incomplete.md |
council wait command case |
1 | 1 | done |
docs/tests/orch/council-tally/README.md |
council tally command case index |
0 | 0 | done |
docs/tests/orch/council-tally/council-tally-groups-reviewer-findings-in-normal-mode.md |
council tally command case |
1 | 1 | done |
docs/tests/orch/council-tally/council-tally-keeps-distinct-proposals-in-strict-mode.md |
council tally command case |
1 | 1 | done |
docs/tests/orch/council-report/README.md |
council report command case index |
0 | 0 | done |
docs/tests/orch/council-report/council-report-defaults-to-consensus-and-majority.md |
council report command case |
1 | 1 | done |
docs/tests/orch/council-report/council-report-show-all-includes-minority.md |
council report command case |
1 | 1 | done |
docs/tests/orch/council-report/council-report-json-shape-is-stable.md |
council report command case |
1 | 1 | done |
docs/tests/orch/council-report/council-report-rejects-before-tally.md |
council report command case |
1 | 1 | done |
docs/tests/orch/council-report/council-report-rejects-invalid-show.md |
council report command case |
1 | 1 | done |
docs/tests/orch/council-report/council-report-defaults-to-consensus-when-run-is-only-unanimous.md |
council report command case |
1 | 1 | done |
Authoring Order
- global conventions in
docs/tests/orch/README.md - shared fixtures and assertion helpers in
docs/tests/orch/_shared/README.md - workflow cases in
docs/tests/orch/workflows/README.md - core scheduler command docs:
run-init,task-add,dep-add,ready,dispatch,reconcile,status - interactive leader command docs:
wait,blocked,answer,retry,reassign,cancel,cleanup - council workflow docs:
council-start,council-wait,council-tally,council-report
Authored Case Register
| Path | Case Slug | Coverage Note | Status |
|---|---|---|---|
docs/tests/orch/workflows/README.md |
run-dispatch-reconcile-status-happy-path |
end-to-end happy path from run creation through final status | done |
docs/tests/orch/workflows/README.md |
dependency-blocked-answer-resume-flow |
dependency gating plus blocked question and answer recovery | done |
docs/tests/orch/workflows/README.md |
strict-worktree-dispatch-to-cleanup |
worktree-backed code task flows from dispatch through cleanup | done |
docs/tests/orch/workflows/README.md |
council-review-end-to-end |
council workflow runs from reviewer dispatch through final report | done |
docs/tests/orch/run-init/run-init-creates-new-run.md |
run-init-creates-new-run |
creates a run with goal and optional summary | done |
docs/tests/orch/run-show/run-show-returns-run-summary-and-task-counts.md |
run-show-returns-run-summary-and-task-counts |
shows aggregate run metadata after activity | done |
docs/tests/orch/task-add/task-add-creates-ready-root-task.md |
task-add-creates-ready-root-task |
dependency-free task becomes ready immediately | done |
docs/tests/orch/task-add/task-add-rejects-invalid-acceptance-json.md |
task-add-rejects-invalid-acceptance-json |
malformed --acceptance-json returns stable invalid_input |
done |
docs/tests/orch/task-add/task-add-rejects-invalid-priority.md |
task-add-rejects-invalid-priority |
unsupported priorities are rejected with invalid_input | done |
docs/tests/orch/dep-add/dep-add-blocks-dependent-task-until-prerequisite-completes.md |
dep-add-blocks-dependent-task-until-prerequisite-completes |
dependency edge prevents immediate readiness | done |
docs/tests/orch/ready/ready-lists-only-eligible-tasks.md |
ready-lists-only-eligible-tasks |
ready list excludes dependency-gated tasks | done |
docs/tests/orch/ready/ready-orders-by-priority-and-respects-limit.md |
ready-orders-by-priority-and-respects-limit |
ready output orders by priority and applies explicit limit truncation | done |
docs/tests/orch/dispatch/dispatch-creates-attempt-and-thread-for-ready-task.md |
dispatch-creates-attempt-and-thread-for-ready-task |
ready task dispatch creates attempt, thread, and task message | done |
docs/tests/orch/dispatch/dispatch-rejects-non-ready-task.md |
dispatch-rejects-non-ready-task |
dispatch on gated task returns invalid_state | done |
docs/tests/orch/dispatch/dispatch-creates-strict-worktree.md |
dispatch-creates-strict-worktree |
explicit strict worktree dispatch provisions isolated workspace metadata | done |
docs/tests/orch/dispatch/dispatch-rejects-dirty-repo-without-base-ref.md |
dispatch-rejects-dirty-repo-without-base-ref |
dirty repository without explicit base ref is rejected in strict mode | done |
docs/tests/orch/dispatch/dispatch-allows-explicit-base-ref-on-dirty-repo.md |
dispatch-allows-explicit-base-ref-on-dirty-repo |
explicit base ref allows dispatch from a dirty repository | done |
docs/tests/orch/dispatch/dispatch-auto-enables-worktree-for-code-like-task.md |
dispatch-auto-enables-worktree-for-code-like-task |
code-like task metadata auto-enables worktree mode | done |
docs/tests/orch/dispatch/dispatch-skips-auto-worktree-for-non-code-task.md |
dispatch-skips-auto-worktree-for-non-code-task |
clearly non-code tasks stay on normal dispatch path | done |
docs/tests/orch/reconcile/reconcile-maps-claimed-or-in-progress-thread-to-running.md |
reconcile-maps-claimed-or-in-progress-thread-to-running |
reconcile maps active inbox execution to running task state | done |
docs/tests/orch/reconcile/reconcile-maps-done-or-failed-thread-to-terminal-task-state.md |
reconcile-maps-done-or-failed-thread-to-terminal-task-state |
reconcile maps terminal inbox states to terminal task states | done |
docs/tests/orch/wait/wait-wakes-on-matching-run-event.md |
wait-wakes-on-matching-run-event |
wait wakes on a later matching run-scoped event | done |
docs/tests/orch/wait/wait-times-out-without-matching-event.md |
wait-times-out-without-matching-event |
wait timeout returns a normal non-woken result | done |
docs/tests/orch/blocked/blocked-lists-latest-question-for-blocked-task.md |
blocked-lists-latest-question-for-blocked-task |
blocked view includes latest question payload for the task | done |
docs/tests/orch/answer/answer-appends-answer-to-active-thread.md |
answer-appends-answer-to-active-thread |
answer appends an inbox answer message to the blocked attempt thread | done |
docs/tests/orch/answer/answer-accepts-payload-json-without-body.md |
answer-accepts-payload-json-without-body |
payload-only answers stay valid and machine-readable | done |
docs/tests/orch/answer/answer-rejects-empty-body-and-payload.md |
answer-rejects-empty-body-and-payload |
empty answer requests fail with invalid_input | done |
docs/tests/orch/retry/retry-creates-new-attempt-for-failed-task.md |
retry-creates-new-attempt-for-failed-task |
retry dispatches a successor attempt after failure | done |
docs/tests/orch/reassign/reassign-cancels-old-thread-and-dispatches-new-attempt.md |
reassign-cancels-old-thread-and-dispatches-new-attempt |
reassign cancels old execution and opens a new attempt | done |
docs/tests/orch/cancel/cancel-cancels-single-task.md |
cancel-cancels-single-task |
single-task cancel moves only the targeted task to cancelled | done |
docs/tests/orch/cancel/cancel-cancels-entire-run.md |
cancel-cancels-entire-run |
run cancel cascades terminal state across the run | done |
docs/tests/orch/cleanup/cleanup-removes-completed-worktree.md |
cleanup-removes-completed-worktree |
cleanup removes completed attempt worktree artifacts | done |
docs/tests/orch/cleanup/cleanup-rejects-attempt-without-task.md |
cleanup-rejects-attempt-without-task |
cleanup enforces --task when --attempt is specified |
done |
docs/tests/orch/cleanup/cleanup-returns-no-matching-work-when-filters-miss.md |
cleanup-returns-no-matching-work-when-filters-miss |
cleanup returns no_matching_work when selectors find no candidates | done |
docs/tests/orch/status/status-returns-run-summary-and-task-list.md |
status-returns-run-summary-and-task-list |
status reports aggregate run state and per-task statuses | done |
docs/tests/orch/council-start/council-start-dispatches-three-reviewers.md |
council-start-dispatches-three-reviewers |
council start creates and dispatches three fixed reviewer tasks | done |
docs/tests/orch/council-wait/council-wait-wakes-when-all-reviewers-complete.md |
council-wait-wakes-when-all-reviewers-complete |
council wait wakes when all reviewer tasks complete | done |
docs/tests/orch/council-wait/council-wait-times-out-when-reviewers-incomplete.md |
council-wait-times-out-when-reviewers-incomplete |
council wait timeout stays machine-readable | done |
docs/tests/orch/council-tally/council-tally-groups-reviewer-findings-in-normal-mode.md |
council-tally-groups-reviewer-findings-in-normal-mode |
normal similarity groups semantically aligned reviewer findings | done |
docs/tests/orch/council-tally/council-tally-keeps-distinct-proposals-in-strict-mode.md |
council-tally-keeps-distinct-proposals-in-strict-mode |
strict similarity preserves wording-level proposal separation | done |
docs/tests/orch/council-report/council-report-defaults-to-consensus-and-majority.md |
council-report-defaults-to-consensus-and-majority |
default report keeps main output on consensus and majority buckets | done |
docs/tests/orch/council-report/council-report-show-all-includes-minority.md |
council-report-show-all-includes-minority |
--show all includes minority recommendations in final report |
done |
docs/tests/orch/council-report/council-report-json-shape-is-stable.md |
council-report-json-shape-is-stable |
JSON response shape and report artifact metadata remain stable | done |
docs/tests/orch/council-report/council-report-rejects-before-tally.md |
council-report-rejects-before-tally |
report generation before tally fails with invalid_state | done |
docs/tests/orch/council-report/council-report-rejects-invalid-show.md |
council-report-rejects-invalid-show |
unsupported --show values return invalid_input |
done |
docs/tests/orch/council-report/council-report-defaults-to-consensus-when-run-is-only-unanimous.md |
council-report-defaults-to-consensus-when-run-is-only-unanimous |
omitted --show collapses to consensus for only-unanimous runs |
done |
Pending Case Backlog
No pending case slugs remain in the current plan.
When a new orch CLI contract or workflow needs coverage:
- if it is a command case, create a new
<case-slug>.mdfile under the relevant leaf-command folder and add it to that folderREADME.mdindex - if it is a workflow case, add it to
docs/tests/orch/workflows/README.md - add the new slug to
Authored Case Register - update
Current SnapshotandDocument Progress
Definition Of Done
This roadmap is complete only when all of the following are true:
- every implemented
orchleaf command has a corresponding document folder - each planned command index and case document exists
- each pending case slug has been either authored or explicitly deferred
- the authored-case register matches the actual Markdown files on disk
- a new agent can pick any future case and know exactly where it should be written