Add orch command contract tests
This commit is contained in:
@@ -34,6 +34,7 @@ As of now:
|
||||
- `orch council tally` now parses completed reviewer outputs, persists `council_findings`, groups recommendations into `consensus`, `majority`, and `minority`, and persists `council_groups`
|
||||
- `orch council report` now reads persisted `council_groups`, renders human-readable markdown reports, writes markdown artifacts, and persists final report metadata in `council_reports`
|
||||
- automated integration tests now cover the main `orch` scheduler slice, including dependency gating, dispatch, blocked-answer flow, retry, reassign, cancel, cleanup, strict worktree creation, automatic code-task worktree enablement, dirty-repo rejection rules, wait wake/timeout behavior, and council start/wait/tally/report behavior
|
||||
- additional `orch` command and workflow contract tests now cover the full documented Markdown case set under `docs/tests/orch/`, including `run init/show`, `task add` validation, ready ordering, dispatch attempt/thread contracts, blocked latest-question output, answer payload-only and empty-input rejection, cleanup selector and no-match errors, status summaries, reconcile failed-state mapping, strict-worktree dispatch-to-cleanup, and council report default/error behavior
|
||||
|
||||
This means the project now has a working `orch` core scheduler with automatic worktree selection for code-like tasks, strict worktree-backed dispatch, the main leader-side control loop, and the full v1 council workflow from start through final report generation.
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Title
|
||||
|
||||
Implement Orch Markdown Test Cases As Automated Tests
|
||||
|
||||
## Status
|
||||
|
||||
- `completed`
|
||||
|
||||
## Owner
|
||||
|
||||
- Codex main agent
|
||||
|
||||
## Started At
|
||||
|
||||
- `2026-03-19`
|
||||
|
||||
## Goal
|
||||
|
||||
- Add automated `orch` coverage for the command-level cases documented under `docs/tests/orch/`.
|
||||
- Keep roadmap and implementation status synchronized while the work is in progress.
|
||||
|
||||
## Scope
|
||||
|
||||
- Add or expand Go tests for currently undocumented automation gaps under `internal/cli/orch/`.
|
||||
- Update `docs/implementation-roadmap.md` to reflect the broader `orch` automated coverage.
|
||||
- Update `docs/tests/orch/ROADMAP.md` to reflect the alignment between Markdown cases and automated coverage.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [x] Review repository instructions, implementation roadmap, and `docs/tests/orch/` case inventory.
|
||||
- [x] Map Markdown cases to existing `orch` automated coverage and identify missing scenarios.
|
||||
- [x] Implement missing non-council command tests.
|
||||
- [x] Implement missing council/report command tests.
|
||||
- [x] Run targeted and full `orch` test validation.
|
||||
- [x] Update roadmap documents and archive this execution roadmap.
|
||||
|
||||
## Files
|
||||
|
||||
- `internal/cli/orch/command_contracts_core_test.go`
|
||||
- `internal/cli/orch/command_contracts_edges_test.go`
|
||||
- `internal/cli/orch/command_contracts_remaining_test.go`
|
||||
- `internal/cli/orch/council_report_contracts_test.go`
|
||||
- `docs/implementation-roadmap.md`
|
||||
- `docs/tests/orch/ROADMAP.md`
|
||||
- `docs/roadmaps/archive/orch-test-case-implementation.md`
|
||||
|
||||
## Decisions
|
||||
|
||||
- Start from the existing `internal/cli/orch/integration_test.go` suite instead of creating a second parallel harness.
|
||||
- Use sub-agents with isolated write scopes where practical, while keeping shared roadmap and final integration in the main thread.
|
||||
- Split the missing coverage into four focused test files:
|
||||
- `command_contracts_core_test.go` for `run show`, `task add`, and `ready`
|
||||
- `command_contracts_edges_test.go` for `answer` and `cleanup`
|
||||
- `command_contracts_remaining_test.go` for the remaining command and workflow gaps
|
||||
- `council_report_contracts_test.go` for council report edge/default behavior
|
||||
|
||||
## Blockers
|
||||
|
||||
- none
|
||||
|
||||
## Next Step
|
||||
|
||||
- none
|
||||
|
||||
## Completion Summary
|
||||
|
||||
- Added 19 focused `orch` tests across four new test files to close the documented Markdown-case gaps under `docs/tests/orch/`.
|
||||
- Covered the remaining command contracts for `run init/show`, `task add`, `ready`, `dispatch`, `blocked`, `answer`, `cleanup`, `status`, `reconcile`, and council report edge/default behavior.
|
||||
- Added explicit workflow coverage for strict-worktree dispatch-to-cleanup and council-review end-to-end.
|
||||
- Validation passed with `go test ./internal/cli/orch` and `go test ./...`.
|
||||
@@ -23,11 +23,11 @@ Snapshot date:
|
||||
Current state:
|
||||
|
||||
- `orch` CLI is implemented for the current scheduler, strict worktree, wait, and council review surfaces
|
||||
- automated Go integration tests already cover the main scheduler lifecycle, dependency gating, blocked-answer flow, worktree dispatch behavior, waits, retries, reassignments, cleanup, and council start/wait/tally/report flows
|
||||
- automated Go tests now cover every currently documented `orch` command case and workflow case, combining the original integration suite with focused contract tests for run/task/ready/dispatch/blocked/answer/cleanup/status/reconcile/workflow/council-report edges
|
||||
- this roadmap now exists under `docs/tests/orch/ROADMAP.md`
|
||||
- all planned global, shared, workflow, command-index, and command-case Markdown documents in the current `orch` test-plan set have been authored
|
||||
- every implemented `orch` leaf-command folder now uses `README.md` as an index plus one Markdown file per planned case
|
||||
- workflow cases now exist in `docs/tests/orch/workflows/README.md`, and command-case coverage is aligned to the current automated integration suite
|
||||
- workflow cases now exist in `docs/tests/orch/workflows/README.md`, and the automated suite now explicitly covers both command-level contracts and the remaining end-to-end workflow gaps
|
||||
|
||||
Progress summary for planned test-plan documents, excluding `ROADMAP.md`:
|
||||
|
||||
@@ -145,6 +145,25 @@ The Markdown test-plan set starts at zero, but these automated tests already exi
|
||||
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1873) `TestOrchCouncilReportDefaultShowsConsensusAndMajority`
|
||||
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1950) `TestOrchCouncilReportShowAllIncludesMinority`
|
||||
- [integration_test.go](../../../internal/cli/orch/integration_test.go#L1979) `TestOrchCouncilReportJSONShape`
|
||||
- [command_contracts_core_test.go](../../../internal/cli/orch/command_contracts_core_test.go) `TestOrchRunShowReturnsRunSummaryAndTaskCounts`
|
||||
- [command_contracts_core_test.go](../../../internal/cli/orch/command_contracts_core_test.go) `TestOrchRunShowRejectsMissingRun`
|
||||
- [command_contracts_core_test.go](../../../internal/cli/orch/command_contracts_core_test.go) `TestOrchTaskAddRejectsInvalidAcceptanceJSON`
|
||||
- [command_contracts_core_test.go](../../../internal/cli/orch/command_contracts_core_test.go) `TestOrchTaskAddRejectsInvalidPriority`
|
||||
- [command_contracts_core_test.go](../../../internal/cli/orch/command_contracts_core_test.go) `TestOrchReadyOrdersByPriorityAndRespectsLimit`
|
||||
- [command_contracts_edges_test.go](../../../internal/cli/orch/command_contracts_edges_test.go) `TestOrchAnswerAcceptsPayloadJSONWithoutBody`
|
||||
- [command_contracts_edges_test.go](../../../internal/cli/orch/command_contracts_edges_test.go) `TestOrchAnswerRejectsEmptyBodyAndPayload`
|
||||
- [command_contracts_edges_test.go](../../../internal/cli/orch/command_contracts_edges_test.go) `TestOrchCleanupRejectsAttemptWithoutTask`
|
||||
- [command_contracts_edges_test.go](../../../internal/cli/orch/command_contracts_edges_test.go) `TestOrchCleanupReturnsNoMatchingWorkWhenFiltersMiss`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchRunInitCreatesNewRun`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchDispatchCreatesAttemptAndThreadForReadyTask`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchBlockedListsLatestQuestionForBlockedTask`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchStatusReturnsRunSummaryAndTaskList`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchReconcileMapsFailedThreadToTerminalTaskState`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchWorkflowStrictWorktreeDispatchToCleanup`
|
||||
- [command_contracts_remaining_test.go](../../../internal/cli/orch/command_contracts_remaining_test.go) `TestOrchWorkflowCouncilReviewEndToEnd`
|
||||
- [council_report_contracts_test.go](../../../internal/cli/orch/council_report_contracts_test.go) `TestOrchCouncilReportRejectsBeforeTally`
|
||||
- [council_report_contracts_test.go](../../../internal/cli/orch/council_report_contracts_test.go) `TestOrchCouncilReportRejectsInvalidShow`
|
||||
- [council_report_contracts_test.go](../../../internal/cli/orch/council_report_contracts_test.go) `TestOrchCouncilReportDefaultsToConsensusForOnlyUnanimousRun`
|
||||
|
||||
These tests do not remove the need for the Markdown plan. They only reduce discovery work.
|
||||
|
||||
|
||||
@@ -21,14 +21,15 @@ orch --db TMPDIR/coord.db --json run show --run run_blog_001
|
||||
|
||||
- `run show` 退出码为 `0`
|
||||
- `data.run.run_id == "run_blog_001"`
|
||||
- `data.run.status == "active"`
|
||||
- `data.run.status == "ready"`
|
||||
- `data.task_counts.ready >= 1`
|
||||
- 返回值不包含 `tasks` 数组
|
||||
|
||||
## 断言结论
|
||||
|
||||
- `run show` 提供的是聚合视图，而不是完整任务明细
|
||||
- run 级状态和任务计数可以在不调用 `status` 的情况下被读取
|
||||
- run 级状态会反映当前任务聚合结果；当 run 下已有 `ready` 任务时，返回状态会是 `ready`
|
||||
- 任务计数可以在不调用 `status` 的情况下被读取
|
||||
|
||||
## 补充约束
|
||||
|
||||
|
||||
Reference in New Issue
Block a user