From dd8da4aa58c786f7fec348f8e9b92ab6364a99bd Mon Sep 17 00:00:00 2001 From: kurihada Date: Thu, 19 Mar 2026 16:00:11 +0800 Subject: [PATCH] Add orch Markdown test-plan skeleton --- docs/implementation-roadmap.md | 36 ++- .../orch-markdown-test-plan-skeleton.md | 60 ++++ docs/tests/orch/README.md | 74 +++++ docs/tests/orch/ROADMAP.md | 282 ++++++++++++++++++ docs/tests/orch/_shared/README.md | 121 ++++++++ docs/tests/orch/answer/README.md | 7 + docs/tests/orch/blocked/README.md | 7 + docs/tests/orch/cancel/README.md | 7 + docs/tests/orch/cleanup/README.md | 7 + docs/tests/orch/council-report/README.md | 7 + docs/tests/orch/council-start/README.md | 7 + docs/tests/orch/council-tally/README.md | 7 + docs/tests/orch/council-wait/README.md | 7 + docs/tests/orch/dep-add/README.md | 7 + docs/tests/orch/dispatch/README.md | 7 + docs/tests/orch/ready/README.md | 7 + docs/tests/orch/reassign/README.md | 7 + docs/tests/orch/reconcile/README.md | 7 + docs/tests/orch/retry/README.md | 7 + docs/tests/orch/run-init/README.md | 7 + docs/tests/orch/run-show/README.md | 7 + docs/tests/orch/status/README.md | 7 + docs/tests/orch/task-add/README.md | 7 + docs/tests/orch/wait/README.md | 7 + docs/tests/orch/workflows/README.md | 22 ++ 25 files changed, 724 insertions(+), 4 deletions(-) create mode 100644 docs/roadmaps/archive/orch-markdown-test-plan-skeleton.md create mode 100644 docs/tests/orch/README.md create mode 100644 docs/tests/orch/ROADMAP.md create mode 100644 docs/tests/orch/_shared/README.md create mode 100644 docs/tests/orch/answer/README.md create mode 100644 docs/tests/orch/blocked/README.md create mode 100644 docs/tests/orch/cancel/README.md create mode 100644 docs/tests/orch/cleanup/README.md create mode 100644 docs/tests/orch/council-report/README.md create mode 100644 docs/tests/orch/council-start/README.md create mode 100644 docs/tests/orch/council-tally/README.md create mode 100644 docs/tests/orch/council-wait/README.md create mode 100644 docs/tests/orch/dep-add/README.md create mode 100644 docs/tests/orch/dispatch/README.md create mode 100644 docs/tests/orch/ready/README.md create mode 100644 docs/tests/orch/reassign/README.md create mode 100644 docs/tests/orch/reconcile/README.md create mode 100644 docs/tests/orch/retry/README.md create mode 100644 docs/tests/orch/run-init/README.md create mode 100644 docs/tests/orch/run-show/README.md create mode 100644 docs/tests/orch/status/README.md create mode 100644 docs/tests/orch/task-add/README.md create mode 100644 docs/tests/orch/wait/README.md create mode 100644 docs/tests/orch/workflows/README.md diff --git a/docs/implementation-roadmap.md b/docs/implementation-roadmap.md index 506e3a1..53eab8c 100644 --- a/docs/implementation-roadmap.md +++ b/docs/implementation-roadmap.md @@ -21,6 +21,7 @@ As of now: - `inbox` supports blocking waits, lease renewal, unread fetches backed by per-agent read cursors, `--body-file`, artifact attachments, and structured JSON errors with stable exit codes - integration tests now cover each implemented inbox command, plus the main inbox workflows, wait/watch flows, artifact persistence, unread behavior, and JSON error contracts - a human-readable inbox command test-plan set has been authored under `docs/tests/inbox/` +- an inbox-style human-readable `orch` test-plan skeleton now exists under `docs/tests/orch/`, with a `ROADMAP.md`, shared conventions, workflow entrypoint, and one index `README.md` per implemented leaf command - a reusable Codex skill package for `inbox` now exists under `skills/inbox/`, with a formal `SKILL.md`, `agents/openai.yaml`, and a bundled CLI binary asset - an inbox skill forward-test plan directory now exists under `docs/tests/inbox-skill/`, with a shared execution template and multiple scenario cases - an execution-roadmap workflow now exists under `docs/roadmaps/active/` and `docs/roadmaps/archive/` for agent-level work traces and completion archives @@ -475,15 +476,42 @@ Definition of done: - shared assumptions are centralized instead of copied into each command file - a new agent can pick any case and implement it as an automated test with minimal additional discovery +## Orch Test Documentation Roadmap + +Status: + +- initial skeleton created under `docs/tests/orch/` +- global conventions, shared fixtures, workflow entrypoint, and per-command index documents now exist +- command-case Markdown files are still pending and should be authored against the backlog in `docs/tests/orch/ROADMAP.md` + +Goal: + +- make `orch` behavior easy for a new agent to understand and convert into automated tests or manual validation steps without re-reading all scheduler code paths + +Directory layout: + +- `docs/tests/orch/README.md` +- `docs/tests/orch/ROADMAP.md` +- `docs/tests/orch/_shared/README.md` +- `docs/tests/orch/workflows/README.md` +- `docs/tests/orch//README.md` + +Current document model: + +- one folder per implemented leaf command +- each command folder uses `README.md` as an index only +- workflow cases live in `docs/tests/orch/workflows/README.md` +- detailed case backlog is tracked centrally in `docs/tests/orch/ROADMAP.md` + +Next documentation step: + +- start authoring the highest-value scheduler and workflow cases from `docs/tests/orch/ROADMAP.md`, beginning with the happy-path lifecycle, dependency-blocked-answer flow, and strict worktree dispatch cases + ## Out Of Scope For First Pass Do not block v1 on these: -- distributed execution - advanced auth or permissions -- patch-producing council mode -- configurable reviewer counts beyond three -- rich similarity engines for proposal grouping - background daemons beyond blocking CLI commands ## Handoff Notes For Future Agents diff --git a/docs/roadmaps/archive/orch-markdown-test-plan-skeleton.md b/docs/roadmaps/archive/orch-markdown-test-plan-skeleton.md new file mode 100644 index 0000000..f4ee5e9 --- /dev/null +++ b/docs/roadmaps/archive/orch-markdown-test-plan-skeleton.md @@ -0,0 +1,60 @@ +# Orch Markdown Test Plan Skeleton + +## Status + +- `completed` + +## Owner + +- codex + +## Started At + +- `2026-03-19` + +## Goal + +- create an inbox-style human-readable Markdown test-plan skeleton for `orch` so future agents can add command and workflow cases without rediscovering the documentation structure + +## Scope + +- add `docs/tests/orch/README.md` and `docs/tests/orch/ROADMAP.md` +- add shared and workflow entrypoint documents under `docs/tests/orch/` +- add one directory per `orch` leaf command with an index `README.md` +- update the implementation roadmap and archive this execution roadmap when complete + +## Checklist + +- [x] read the existing inbox Markdown test-plan roadmap and directory conventions +- [x] define the `orch` leaf-command directory model +- [x] add the `docs/tests/orch/` skeleton files and per-command indexes +- [x] update `docs/implementation-roadmap.md` +- [x] archive this roadmap with a completion summary + +## Files + +- `docs/roadmaps/archive/orch-markdown-test-plan-skeleton.md` +- `docs/implementation-roadmap.md` +- `docs/tests/orch/README.md` +- `docs/tests/orch/ROADMAP.md` +- `docs/tests/orch/_shared/README.md` +- `docs/tests/orch/workflows/README.md` + +## Decisions + +- mirror the inbox test-plan structure at the directory and index level, but model `orch` by leaf commands rather than command groups +- seed the new roadmap with a concrete pending-case backlog derived from existing `internal/cli/orch/integration_test.go` coverage so later authoring can proceed without rediscovery + +## Blockers + +- none + +## Next Step + +- start writing the highest-value `orch` case documents from the new backlog, beginning with the happy-path scheduler workflow and strict worktree dispatch cases + +## Completion Summary + +- created `docs/tests/orch/` with a top-level `README.md`, `ROADMAP.md`, shared conventions, workflow entrypoint, and one index `README.md` per implemented leaf command +- documented the initial `orch` Markdown test-plan model as leaf-command directories instead of command groups +- updated the project implementation roadmap so future agents can discover the new `orch` test-plan skeleton from the main handoff document diff --git a/docs/tests/orch/README.md b/docs/tests/orch/README.md new file mode 100644 index 0000000..6ee48e2 --- /dev/null +++ b/docs/tests/orch/README.md @@ -0,0 +1,74 @@ +# Orch Markdown Test Plan + +## Purpose + +This directory contains the human-readable Markdown test plan for the `orch` CLI. + +It complements automated Go tests. The goal is to preserve the user-visible scheduler contract in a form that can be reviewed, extended, and executed manually without re-deriving command behavior from implementation code. + +## Directory Rules + +- one folder per `orch` leaf command or shared area +- each folder keeps a `README.md` entrypoint +- command folders use `README.md` as an index only +- each command test case lives in its own Markdown file named after the case slug +- no numeric test IDs +- each command case is identified by its concrete file path + +Case file naming pattern: + +```text +.md +``` + +## Authoring Principles + +- focus on externally visible CLI behavior rather than store internals +- prefer stable command sequences that a new agent can replay against a temp database +- document both success contracts and failure boundaries +- reuse scenarios from automated `orch` integration tests before inventing new cases +- keep terminology consistent with the scheduler concepts exposed by `orch`: run, task, dependency, attempt, blocked task, worktree, and council review + +## Common Execution Model + +Most cases in this directory assume the same baseline: + +1. create an isolated temporary directory +2. choose a database path such as `TMPDIR/coord.db` +3. run the target `orch` command sequence with `--db TMPDIR/coord.db --json` +4. when a case needs worker-side state transitions, drive them through `inbox` against the same database + +Unless a case says otherwise: + +- commands should use `--json` +- assertions should check both exit code and JSON payload +- `orch` may be pointed at an empty database path; schema bootstrapping happens automatically on open + +## Folder Map + +- `README.md`: global conventions and glossary +- `ROADMAP.md`: document progress, planned case backlog, and authored-case register +- `_shared/README.md`: reusable fixtures, JSON assertions, exit-code rules, and worktree conventions +- `workflows/README.md`: cross-command end-to-end scenarios +- per-command folders: one leaf-command directory per implemented `orch` command surface + +## Glossary + +- `run`: one coordinated execution for a user request +- `task`: one schedulable unit of work inside a run +- `dependency`: an edge that gates one task on another +- `attempt`: one execution try for a task +- `dispatch`: the act of materializing a task into an inbox thread +- `workspace`: the branch and worktree assigned to a code-writing attempt +- `blocked task`: a task whose active attempt requires clarification or another external decision +- `council review`: a higher-level workflow built on top of `orch` that dispatches fixed reviewer roles and tallies recommendations + +## Relationship To Automated Tests + +The current best executable reference is [internal/cli/orch/integration_test.go](../../../internal/cli/orch/integration_test.go). + +When this Markdown plan expands: + +- prefer matching an existing automated scenario first +- record any additional manual-only contract coverage explicitly in the relevant command case file +- keep [ROADMAP.md](./ROADMAP.md) synchronized with authored files and case slugs diff --git a/docs/tests/orch/ROADMAP.md b/docs/tests/orch/ROADMAP.md new file mode 100644 index 0000000..fa06470 --- /dev/null +++ b/docs/tests/orch/ROADMAP.md @@ -0,0 +1,282 @@ +# Orch Test Documentation Roadmap + +## Purpose + +This roadmap tracks the human-readable Markdown test plan for `orch`. + +It exists so a new agent can immediately answer four questions without re-reading the whole codebase: + +- which test-plan documents already exist +- which cases have already been written down +- which cases are still missing +- what file should be updated next + +This roadmap is for the Markdown test-plan set under `docs/tests/orch/`. +It is not a replacement for automated Go tests. + +## Current Snapshot + +Snapshot date: + +- `2026-03-19` + +Current state: + +- `orch` CLI is implemented for the current scheduler, strict worktree, wait, and council review surfaces +- automated Go integration tests already cover the main scheduler lifecycle, dependency gating, blocked-answer flow, worktree dispatch behavior, waits, retries, reassignments, cleanup, and council start/wait/tally/report flows +- this roadmap now exists under `docs/tests/orch/ROADMAP.md` +- the initial `docs/tests/orch/` skeleton now exists with shared, workflow, and per-command index documents +- command-case Markdown files have not been authored yet; the next work is filling the backlog in a stable order + +Progress summary for planned test-plan documents, excluding `ROADMAP.md`: + +- planned document files: `22` +- authored document files: `22` +- planned case slugs in this roadmap: `35` +- authored case slugs in this roadmap: `0` + +## Scope + +In scope: + +- `orch run init` +- `orch run show` +- `orch task add` +- `orch dep add` +- `orch ready` +- `orch dispatch` +- `orch reconcile` +- `orch wait` +- `orch blocked` +- `orch answer` +- `orch retry` +- `orch reassign` +- `orch cancel` +- `orch cleanup` +- `orch status` +- `orch council start` +- `orch council wait` +- `orch council tally` +- `orch council report` +- cross-command workflows +- shared test conventions for JSON output, exit codes, temp databases, repo fixtures, and worktree assertions + +Out of scope: + +- `inbox` command behavior except as supporting setup for `orch` scenarios +- implementation details that are not visible through the `orch` CLI contract +- future `orch` commands that are not currently implemented + +## Tracking Rules + +Directory model: + +- one folder per leaf command or shared area +- each folder keeps a `README.md` entrypoint +- command folders use `README.md` as an index only +- each command case lives in its own Markdown file named after the case slug +- cross-command workflow cases remain grouped in `docs/tests/orch/workflows/README.md` + +Case identity: + +- do not use numeric IDs +- identify each command case by its concrete file path +- identify each workflow case by `path + case slug` +- command case file naming pattern: + +```text +.md +``` + +- workflow case heading pattern: + +```md +## case: run-dispatch-reconcile-status-happy-path +``` + +Per-case structure inside the case document: + +- `用例意义` +- `前置条件` +- `输入` +- `预期输出` +- `断言结论` + +How to update this roadmap when a new case is written: + +1. if it is a command case, create or update the target `.md` file under the relevant leaf-command folder +2. if it is a command case, add or update the entry in that folder `README.md` index +3. if it is a workflow case, add or update the case inside `docs/tests/orch/workflows/README.md` +4. move the case slug from `Pending Case Backlog` to `Authored Case Register` +5. update the authored counts in `Current Snapshot` +6. if a new Markdown file is created, update `Document Progress` + +Allowed status values in this roadmap: + +- `pending` +- `in_progress` +- `done` +- `deferred` + +## Existing Automated Coverage Reference + +The Markdown test-plan set starts at zero, but these automated tests already exist and should be used as source material when writing the docs: + +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L12) `TestOrchRunDispatchReconcileLifecycle` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L184) `TestOrchDependencyBlockedAndAnswerFlow` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L458) `TestOrchDispatchRejectsNonReadyTask` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L512) `TestOrchDispatchCreatesStrictWorktree` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L631) `TestOrchStrictWorktreeRejectsDirtyRepoWithoutBaseRef` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L680) `TestOrchStrictWorktreeAllowsExplicitBaseRefOnDirtyRepo` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L733) `TestOrchDispatchAutoEnablesWorktreeForCodeLikeTask` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L782) `TestOrchDispatchDoesNotAutoEnableWorktreeForNonCodeTask` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L829) `TestOrchWaitWakesOnBlockedEvent` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L944) `TestOrchWaitTimesOutWithoutMatchingEvent` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L981) `TestOrchRetryCreatesNewAttempt` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1079) `TestOrchReassignCancelsOldThreadAndDispatchesNewAttempt` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1186) `TestOrchCancelTaskAndRun` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1315) `TestOrchCleanupRemovesCompletedWorktree` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1402) `TestOrchCouncilStartDispatchesThreeReviewers` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1545) `TestOrchCouncilWaitWakesWhenAllReviewersComplete` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1678) `TestOrchCouncilWaitTimesOutWhenReviewersIncomplete` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1715) `TestOrchCouncilTallyGroupsReviewerFindingsNormal` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1813) `TestOrchCouncilTallyStrictKeepsDistinctProposals` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1873) `TestOrchCouncilReportDefaultShowsConsensusAndMajority` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1950) `TestOrchCouncilReportShowAllIncludesMinority` +- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1979) `TestOrchCouncilReportJSONShape` + +These tests do not remove the need for the Markdown plan. They only reduce discovery work. + +## Planned Directory Tree + +```text +docs/tests/orch/ + ROADMAP.md + README.md + _shared/ + README.md + workflows/ + README.md + run-init/ + README.md + run-show/ + README.md + task-add/ + README.md + dep-add/ + README.md + ready/ + README.md + dispatch/ + README.md + reconcile/ + README.md + wait/ + README.md + blocked/ + README.md + answer/ + README.md + retry/ + README.md + reassign/ + README.md + cancel/ + README.md + cleanup/ + README.md + status/ + README.md + council-start/ + README.md + council-wait/ + README.md + council-tally/ + README.md + council-report/ + README.md +``` + +## Document Progress + +| Path | Purpose | Planned Cases | Authored Cases | Status | +| --- | --- | ---: | ---: | --- | +| `docs/tests/orch/README.md` | Global testing conventions and glossary | 0 | 0 | done | +| `docs/tests/orch/_shared/README.md` | Shared fixtures, JSON assertions, exit-code rules, and worktree conventions | 0 | 0 | done | +| `docs/tests/orch/workflows/README.md` | Cross-command scenarios | 4 | 0 | done | +| `docs/tests/orch/run-init/README.md` | `run init` command case index | 1 | 0 | done | +| `docs/tests/orch/run-show/README.md` | `run show` command case index | 1 | 0 | done | +| `docs/tests/orch/task-add/README.md` | `task add` command case index | 1 | 0 | done | +| `docs/tests/orch/dep-add/README.md` | `dep add` command case index | 1 | 0 | done | +| `docs/tests/orch/ready/README.md` | `ready` command case index | 1 | 0 | done | +| `docs/tests/orch/dispatch/README.md` | `dispatch` command case index | 6 | 0 | done | +| `docs/tests/orch/reconcile/README.md` | `reconcile` command case index | 2 | 0 | done | +| `docs/tests/orch/wait/README.md` | `wait` command case index | 2 | 0 | done | +| `docs/tests/orch/blocked/README.md` | `blocked` command case index | 1 | 0 | done | +| `docs/tests/orch/answer/README.md` | `answer` command case index | 1 | 0 | done | +| `docs/tests/orch/retry/README.md` | `retry` command case index | 1 | 0 | done | +| `docs/tests/orch/reassign/README.md` | `reassign` command case index | 1 | 0 | done | +| `docs/tests/orch/cancel/README.md` | `cancel` command case index | 2 | 0 | done | +| `docs/tests/orch/cleanup/README.md` | `cleanup` command case index | 1 | 0 | done | +| `docs/tests/orch/status/README.md` | `status` command case index | 1 | 0 | done | +| `docs/tests/orch/council-start/README.md` | `council start` command case index | 1 | 0 | done | +| `docs/tests/orch/council-wait/README.md` | `council wait` command case index | 2 | 0 | done | +| `docs/tests/orch/council-tally/README.md` | `council tally` command case index | 2 | 0 | done | +| `docs/tests/orch/council-report/README.md` | `council report` command case index | 3 | 0 | done | + +## Authoring Order + +1. global conventions in `docs/tests/orch/README.md` +2. shared fixtures and assertion helpers in `docs/tests/orch/_shared/README.md` +3. workflow cases in `docs/tests/orch/workflows/README.md` +4. core scheduler command docs: `run-init`, `task-add`, `dep-add`, `ready`, `dispatch`, `reconcile`, `status` +5. interactive leader command docs: `wait`, `blocked`, `answer`, `retry`, `reassign`, `cancel`, `cleanup` +6. council workflow docs: `council-start`, `council-wait`, `council-tally`, `council-report` + +## Pending Case Backlog + +### Workflows + +- `docs/tests/orch/workflows/README.md :: run-dispatch-reconcile-status-happy-path` - end-to-end happy path from run creation through final `status` +- `docs/tests/orch/workflows/README.md :: dependency-blocked-answer-resume-flow` - dependency gating plus blocked question and answer flow +- `docs/tests/orch/workflows/README.md :: strict-worktree-dispatch-to-cleanup` - worktree creation through cleanup of a completed attempt +- `docs/tests/orch/workflows/README.md :: council-review-end-to-end` - council start, wait, tally, and report sequence + +### Command Cases + +- `docs/tests/orch/run-init/run-init-creates-new-run.md` - create a run with goal and optional summary +- `docs/tests/orch/run-show/run-show-returns-run-summary-and-task-counts.md` - show aggregate run metadata after activity +- `docs/tests/orch/task-add/task-add-creates-ready-root-task.md` - adding a dependency-free task becomes `ready` +- `docs/tests/orch/dep-add/dep-add-blocks-dependent-task-until-prerequisite-completes.md` - dependency edge prevents immediate readiness +- `docs/tests/orch/ready/ready-lists-only-eligible-tasks.md` - ready list excludes blocked by dependency tasks +- `docs/tests/orch/dispatch/dispatch-creates-attempt-and-thread-for-ready-task.md` - happy-path dispatch with new attempt metadata +- `docs/tests/orch/dispatch/dispatch-rejects-non-ready-task.md` - invalid-state rejection when dispatching a gated task +- `docs/tests/orch/dispatch/dispatch-creates-strict-worktree.md` - explicit strict worktree dispatch persists workspace metadata +- `docs/tests/orch/dispatch/dispatch-rejects-dirty-repo-without-base-ref.md` - strict worktree guard on dirty repository state +- `docs/tests/orch/dispatch/dispatch-allows-explicit-base-ref-on-dirty-repo.md` - explicit base ref bypasses dirty working tree rejection +- `docs/tests/orch/dispatch/dispatch-auto-enables-worktree-for-code-like-task.md` - inferred worktree mode for code-like tasks +- `docs/tests/orch/dispatch/dispatch-skips-auto-worktree-for-non-code-task.md` - non-code tasks stay on the normal dispatch path +- `docs/tests/orch/reconcile/reconcile-maps-claimed-or-in-progress-thread-to-running.md` - reconcile turns claimed/in-progress inbox state into running task state +- `docs/tests/orch/reconcile/reconcile-maps-done-or-failed-thread-to-terminal-task-state.md` - reconcile turns terminal inbox state into task terminal state +- `docs/tests/orch/wait/wait-wakes-on-matching-run-event.md` - wait wakes on a later matching task event +- `docs/tests/orch/wait/wait-times-out-without-matching-event.md` - wait returns a non-woken timeout result +- `docs/tests/orch/blocked/blocked-lists-latest-question-for-blocked-task.md` - blocked list includes the active question payload +- `docs/tests/orch/answer/answer-appends-answer-to-active-thread.md` - answer writes an inbox answer message for the blocked attempt +- `docs/tests/orch/retry/retry-creates-new-attempt-for-failed-task.md` - retry creates a successor attempt after failure +- `docs/tests/orch/reassign/reassign-cancels-old-thread-and-dispatches-new-attempt.md` - reassign moves work to a new worker and attempt +- `docs/tests/orch/cancel/cancel-cancels-single-task.md` - cancel transitions one task to `cancelled` +- `docs/tests/orch/cancel/cancel-cancels-entire-run.md` - cancel cascades terminal state across the run +- `docs/tests/orch/cleanup/cleanup-removes-completed-worktree.md` - cleanup removes a completed attempt worktree +- `docs/tests/orch/status/status-returns-run-summary-and-task-list.md` - status returns run aggregate state and per-task statuses +- `docs/tests/orch/council-start/council-start-dispatches-three-reviewers.md` - council start creates reviewer tasks and metadata +- `docs/tests/orch/council-wait/council-wait-wakes-when-all-reviewers-complete.md` - council wait completes successfully when all reviewers finish +- `docs/tests/orch/council-wait/council-wait-times-out-when-reviewers-incomplete.md` - council wait timeout behavior +- `docs/tests/orch/council-tally/council-tally-groups-reviewer-findings-in-normal-mode.md` - normal similarity grouping behavior +- `docs/tests/orch/council-tally/council-tally-keeps-distinct-proposals-in-strict-mode.md` - strict similarity grouping behavior +- `docs/tests/orch/council-report/council-report-defaults-to-consensus-and-majority.md` - default report buckets and markdown artifact behavior +- `docs/tests/orch/council-report/council-report-show-all-includes-minority.md` - `--show all` includes minority output +- `docs/tests/orch/council-report/council-report-json-shape-is-stable.md` - JSON output shape and artifact metadata contract + +## Authored Case Register + +- none yet diff --git a/docs/tests/orch/_shared/README.md b/docs/tests/orch/_shared/README.md new file mode 100644 index 0000000..d8170b1 --- /dev/null +++ b/docs/tests/orch/_shared/README.md @@ -0,0 +1,121 @@ +# Orch Shared Test Conventions + +## Purpose + +This document captures shared assumptions used by multiple `orch` test-plan documents so command and workflow files can stay focused on behavior instead of repeating setup boilerplate. + +## Recommended Fixture Shape + +Use an isolated temp workspace per case: + +- database path: `TMPDIR/coord.db` +- optional repository path: `TMPDIR/repo` +- optional workspace root: `TMPDIR/worktrees` or `REPO/.orch/worktrees` +- optional body file: `TMPDIR/body.md` + +Recommended minimal bootstrap command: + +```bash +orch --db TMPDIR/coord.db --json run init --run run_demo_001 --goal "Demo run" +``` + +Some cases need additional bootstrap: + +- `task add` for task-oriented flows +- `dep add` for dependency-gating flows +- `inbox` commands when simulating worker claim, progress, blocked, done, or fail transitions + +## Global Flags + +Root-level flags apply to every `orch` subcommand: + +- `--db`: SQLite database path, default `.agents/coord.db` +- `--json`: emit machine-readable JSON + +The current root command does not define a global `--agent`; worker-side activity should be modeled through `inbox` using the same database. + +## Success JSON Contract + +Successful JSON output uses this shape: + +```json +{ + "ok": true, + "command": "dispatch", + "data": {} +} +``` + +Shared assertion points: + +- `ok` is `true` +- `command` matches the invoked leaf command +- `data` contains the command-specific payload + +## Error JSON Contract + +Failure JSON output uses this shape: + +```json +{ + "ok": false, + "error": { + "code": "invalid_input", + "message": "..." + } +} +``` + +Shared assertion points: + +- `ok` is `false` +- `error.code` matches the stable contract +- `error.message` is present and human-readable + +## Exit Code Contract + +The current `orch` CLI contract primarily uses these exit codes: + +| Exit Code | Meaning | Typical Error Code | +| --- | --- | --- | +| `0` | success | none | +| `30` | invalid input, invalid state, or usage-style error | `invalid_input` or `invalid_state` | +| `40` | referenced run, task, or thread missing | `not_found` | +| `50` | unexpected internal failure | `internal_error` | + +When a case expects failure, assert both the exit code and the JSON error code. + +## Body Input Rules + +Commands that support `--body` and `--body-file` should be documented with these shared rules: + +- `--body` and `--body-file` are mutually exclusive +- `--body-file` content is read verbatim into the dispatched or answer body +- unreadable `--body-file` should be treated as `invalid_input` + +Relevant commands: + +- `dispatch` +- `answer` +- `retry` + +## Worktree And Repo Rules + +Cases covering worktree behavior should state: + +- whether the source repository is clean or dirty +- whether `--strict-worktree` is enabled explicitly or inferred automatically +- whether `--base-ref` is omitted or explicitly provided +- where the expected worktree path should be created + +When worktree behavior is under test, assert at least: + +- attempt `base_ref` +- attempt `base_commit` +- attempt `branch_name` +- attempt `worktree_path` +- attempt `workspace_status` + +## Workflow Authoring Rule + +If a case spans multiple `orch` commands, place the end-to-end narrative in `workflows/README.md` first, then add narrower command-level cases only when they are easier to reason about in isolation. diff --git a/docs/tests/orch/answer/README.md b/docs/tests/orch/answer/README.md new file mode 100644 index 0000000..39d3f06 --- /dev/null +++ b/docs/tests/orch/answer/README.md @@ -0,0 +1,7 @@ +# Orch `answer` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/blocked/README.md b/docs/tests/orch/blocked/README.md new file mode 100644 index 0000000..a698ed7 --- /dev/null +++ b/docs/tests/orch/blocked/README.md @@ -0,0 +1,7 @@ +# Orch `blocked` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/cancel/README.md b/docs/tests/orch/cancel/README.md new file mode 100644 index 0000000..853020c --- /dev/null +++ b/docs/tests/orch/cancel/README.md @@ -0,0 +1,7 @@ +# Orch `cancel` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/cleanup/README.md b/docs/tests/orch/cleanup/README.md new file mode 100644 index 0000000..8e13b99 --- /dev/null +++ b/docs/tests/orch/cleanup/README.md @@ -0,0 +1,7 @@ +# Orch `cleanup` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/council-report/README.md b/docs/tests/orch/council-report/README.md new file mode 100644 index 0000000..cb0b11c --- /dev/null +++ b/docs/tests/orch/council-report/README.md @@ -0,0 +1,7 @@ +# Orch `council report` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/council-start/README.md b/docs/tests/orch/council-start/README.md new file mode 100644 index 0000000..c6aa1a7 --- /dev/null +++ b/docs/tests/orch/council-start/README.md @@ -0,0 +1,7 @@ +# Orch `council start` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/council-tally/README.md b/docs/tests/orch/council-tally/README.md new file mode 100644 index 0000000..74a8ddd --- /dev/null +++ b/docs/tests/orch/council-tally/README.md @@ -0,0 +1,7 @@ +# Orch `council tally` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/council-wait/README.md b/docs/tests/orch/council-wait/README.md new file mode 100644 index 0000000..e76363e --- /dev/null +++ b/docs/tests/orch/council-wait/README.md @@ -0,0 +1,7 @@ +# Orch `council wait` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/dep-add/README.md b/docs/tests/orch/dep-add/README.md new file mode 100644 index 0000000..849602e --- /dev/null +++ b/docs/tests/orch/dep-add/README.md @@ -0,0 +1,7 @@ +# Orch `dep add` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/dispatch/README.md b/docs/tests/orch/dispatch/README.md new file mode 100644 index 0000000..c28697c --- /dev/null +++ b/docs/tests/orch/dispatch/README.md @@ -0,0 +1,7 @@ +# Orch `dispatch` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/ready/README.md b/docs/tests/orch/ready/README.md new file mode 100644 index 0000000..2069960 --- /dev/null +++ b/docs/tests/orch/ready/README.md @@ -0,0 +1,7 @@ +# Orch `ready` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/reassign/README.md b/docs/tests/orch/reassign/README.md new file mode 100644 index 0000000..79c52ff --- /dev/null +++ b/docs/tests/orch/reassign/README.md @@ -0,0 +1,7 @@ +# Orch `reassign` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/reconcile/README.md b/docs/tests/orch/reconcile/README.md new file mode 100644 index 0000000..417a0a0 --- /dev/null +++ b/docs/tests/orch/reconcile/README.md @@ -0,0 +1,7 @@ +# Orch `reconcile` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/retry/README.md b/docs/tests/orch/retry/README.md new file mode 100644 index 0000000..de4a2c0 --- /dev/null +++ b/docs/tests/orch/retry/README.md @@ -0,0 +1,7 @@ +# Orch `retry` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/run-init/README.md b/docs/tests/orch/run-init/README.md new file mode 100644 index 0000000..a87ba3e --- /dev/null +++ b/docs/tests/orch/run-init/README.md @@ -0,0 +1,7 @@ +# Orch `run init` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/run-show/README.md b/docs/tests/orch/run-show/README.md new file mode 100644 index 0000000..dbb785f --- /dev/null +++ b/docs/tests/orch/run-show/README.md @@ -0,0 +1,7 @@ +# Orch `run show` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/status/README.md b/docs/tests/orch/status/README.md new file mode 100644 index 0000000..e961fcc --- /dev/null +++ b/docs/tests/orch/status/README.md @@ -0,0 +1,7 @@ +# Orch `status` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/task-add/README.md b/docs/tests/orch/task-add/README.md new file mode 100644 index 0000000..30a9eba --- /dev/null +++ b/docs/tests/orch/task-add/README.md @@ -0,0 +1,7 @@ +# Orch `task add` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/wait/README.md b/docs/tests/orch/wait/README.md new file mode 100644 index 0000000..65b44f2 --- /dev/null +++ b/docs/tests/orch/wait/README.md @@ -0,0 +1,7 @@ +# Orch `wait` Test Plan Index + +## Status + +No command case files are authored yet. + +Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress. diff --git a/docs/tests/orch/workflows/README.md b/docs/tests/orch/workflows/README.md new file mode 100644 index 0000000..d8b59c7 --- /dev/null +++ b/docs/tests/orch/workflows/README.md @@ -0,0 +1,22 @@ +# Orch Workflow Test Plan + +## Scope + +This document tracks cross-command scenarios where the main value is the interaction between multiple `orch` commands, and often between `orch` and `inbox`. + +All examples assume: + +- isolated temp database +- `orch --db TMPDIR/coord.db --json` is used consistently +- assertions follow the shared rules in [../_shared/README.md](../_shared/README.md) + +## Current Status + +No workflow case documents are authored yet. + +Planned first workflow cases live in [../ROADMAP.md](../ROADMAP.md), starting with: + +- `run-dispatch-reconcile-status-happy-path` +- `dependency-blocked-answer-resume-flow` +- `strict-worktree-dispatch-to-cleanup` +- `council-review-end-to-end`