kurihada/ai-workflow-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Author orch Markdown test plan

Browse Source
This commit is contained in:
kurihada
2026-03-19 16:27:28 +08:00
parent b448d98e71
commit a20bec1cac
68 changed files with 2225 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tests/orch/council-report/README.md
+9 -4
View File
@@ -1,7 +1,12 @@
# Orch `council report` Test Plan Index
## Status
## Case Files
No command case files are authored yet.
Use [../ROADMAP.md](../ROADMAP.md) for planned case slugs and document progress.
| Case Slug | File | Coverage Note |
| --- | --- | --- |
| `council-report-defaults-to-consensus-and-majority` | [council-report-defaults-to-consensus-and-majority.md](./council-report-defaults-to-consensus-and-majority.md) | renders a markdown report that omits minority recommendations by default and writes a markdown artifact |
| `council-report-show-all-includes-minority` | [council-report-show-all-includes-minority.md](./council-report-show-all-includes-minority.md) | includes minority recommendations when `--show all` is requested |
| `council-report-json-shape-is-stable` | [council-report-json-shape-is-stable.md](./council-report-json-shape-is-stable.md) | returns the stable JSON report contract with summary, filtered groups, and artifact metadata |
| `council-report-rejects-before-tally` | [council-report-rejects-before-tally.md](./council-report-rejects-before-tally.md) | rejects report generation with `invalid_state` when grouped recommendations have not been tallied yet |
| `council-report-rejects-invalid-show` | [council-report-rejects-invalid-show.md](./council-report-rejects-invalid-show.md) | rejects unsupported `--show` bucket values with `invalid_input` |
| `council-report-defaults-to-consensus-when-run-is-only-unanimous` | [council-report-defaults-to-consensus-when-run-is-only-unanimous.md](./council-report-defaults-to-consensus-when-run-is-only-unanimous.md) | defaults omitted `--show` to `consensus` when the run was started with `--only-unanimous` |
docs/tests/orch/council-report/council-report-defaults-to-consensus-and-majority.md
+70
View File
@@ -0,0 +1,70 @@
# Case: `council-report-defaults-to-consensus-and-majority`
## 用例意义
验证 `council report` 默认只展示 `consensus` 与 `majority` bucket，同时生成 markdown artifact。
## 前置条件
- 使用隔离的临时目录 `TMPDIR`
- 已准备好能产出 `consensus`、`majority`、`minority` 三类 recommendation 的 reviewer 输出 JSON
- 本地可使用 `sqlite3` 从 `task_attempts` 中读取 reviewer thread ID
## 输入
```bash
cat <<'EOF' > TMPDIR/architecture-review.json
{"reviewer_role":"architecture-reviewer","findings":[{"title":"Split contracts","summary":"Transport contracts are mixed into UI code.","proposal":"Move API contract definitions into a dedicated module.","rationale":"This lowers coupling.","confidence":"high","tags":["architecture"],"target_refs":{"repo_path":"."}},{"title":"Share helpers","summary":"Council report rendering paths are repeated.","proposal":"Introduce shared council coordinator helpers for report rendering.","rationale":"This keeps report assembly consistent.","confidence":"medium","tags":["reporting"],"target_refs":{"repo_path":"."}}]}
EOF
cat <<'EOF' > TMPDIR/implementation-review.json
{"reviewer_role":"implementation-reviewer","findings":[{"title":"Extract contracts","summary":"Shared transport shapes are duplicated.","proposal":"Move API contract definitions into dedicated module","rationale":"This reduces duplication.","confidence":"high","tags":["maintainability"],"target_refs":{"repo_path":"."}},{"title":"Reuse report helpers","summary":"Formatting logic should stay shared.","proposal":"Introduce shared council coordinator helpers for report rendering","rationale":"This avoids formatter drift.","confidence":"medium","tags":["reporting"],"target_refs":{"repo_path":"."}}]}
EOF
cat <<'EOF' > TMPDIR/risk-review.json
{"reviewer_role":"risk-reviewer","findings":[{"title":"Lock contracts","summary":"Contract drift becomes risky over time.","proposal":"Move API contract definitions into a dedicated module.","rationale":"This reduces integration regressions.","confidence":"high","tags":["risk"],"target_refs":{"repo_path":"."}},{"title":"Cover JSON output","summary":"The council report response should stay stable.","proposal":"Add regression tests for council report JSON output.","rationale":"This catches contract regressions earlier.","confidence":"high","tags":["testing"],"target_refs":{"repo_path":"."}}]}
EOF
orch --db TMPDIR/coord.db --json council start \
--run council_blog_report_001 \
--target "Review the council reporting flow."
THREAD_ID_CR1=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_001' AND task_id = 'CR1' AND attempt_no = 1;")
THREAD_ID_CR2=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_001' AND task_id = 'CR2' AND attempt_no = 1;")
THREAD_ID_CR3=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_001' AND task_id = 'CR3' AND attempt_no = 1;")
inbox --db TMPDIR/coord.db --json claim --agent architecture-reviewer --thread "$THREAD_ID_CR1"
inbox --db TMPDIR/coord.db --json done --agent architecture-reviewer --thread "$THREAD_ID_CR1" --summary "Review complete" --body-file TMPDIR/architecture-review.json
inbox --db TMPDIR/coord.db --json claim --agent implementation-reviewer --thread "$THREAD_ID_CR2"
inbox --db TMPDIR/coord.db --json done --agent implementation-reviewer --thread "$THREAD_ID_CR2" --summary "Review complete" --body-file TMPDIR/implementation-review.json
inbox --db TMPDIR/coord.db --json claim --agent risk-reviewer --thread "$THREAD_ID_CR3"
inbox --db TMPDIR/coord.db --json done --agent risk-reviewer --thread "$THREAD_ID_CR3" --summary "Review complete" --body-file TMPDIR/risk-review.json
orch --db TMPDIR/coord.db --json council tally \
--run council_blog_report_001 \
--similarity normal
orch --db TMPDIR/coord.db council report \
--run council_blog_report_001
```
## 预期输出
- `council report` 退出码为 `0`
- stdout 是 markdown，而不是 JSON
- 报告正文包含 `# Council Review Report`
- 报告正文包含 `## Consensus`
- 报告正文包含 `## Majority`
- 报告正文不包含 `## Minority`
- `TMPDIR/.orch/reports/council_blog_report_001.md` 被创建，且内容与 stdout 一致
## 断言结论
- `council report` 的默认呈现策略是“主报告展示 consensus + majority，隐藏 minority”
- 该命令既是渲染命令，也是 artifact 产出命令
## 补充约束
- 默认 bucket 行为受 `council run` 的 `only_unanimous` 配置影响；当前常规路径默认仍是 `consensus,majority`
docs/tests/orch/council-report/council-report-defaults-to-consensus-when-run-is-only-unanimous.md
+73
View File
@@ -0,0 +1,73 @@
# Case: `council-report-defaults-to-consensus-when-run-is-only-unanimous`
## 用例意义
验证当 council run 以 `--only-unanimous` 启动时，省略 `--show` 的 `council report --json` 默认只返回 `consensus` bucket。
## 前置条件
- 使用隔离的临时目录 `TMPDIR`
- 已准备好与 `council-report-defaults-to-consensus-and-majority` 相同的 3 份 reviewer 输出 JSON
- 本地可使用 `sqlite3` 从 `task_attempts` 中读取 reviewer thread ID
## 输入
```bash
cat <<'EOF' > TMPDIR/architecture-review.json
{"reviewer_role":"architecture-reviewer","findings":[{"title":"Split contracts","summary":"Transport contracts are mixed into UI code.","proposal":"Move API contract definitions into a dedicated module.","rationale":"This lowers coupling.","confidence":"high","tags":["architecture"],"target_refs":{"repo_path":"."}},{"title":"Share helpers","summary":"Council report rendering paths are repeated.","proposal":"Introduce shared council coordinator helpers for report rendering.","rationale":"This keeps report assembly consistent.","confidence":"medium","tags":["reporting"],"target_refs":{"repo_path":"."}}]}
EOF
cat <<'EOF' > TMPDIR/implementation-review.json
{"reviewer_role":"implementation-reviewer","findings":[{"title":"Extract contracts","summary":"Shared transport shapes are duplicated.","proposal":"Move API contract definitions into dedicated module","rationale":"This reduces duplication.","confidence":"high","tags":["maintainability"],"target_refs":{"repo_path":"."}},{"title":"Reuse report helpers","summary":"Formatting logic should stay shared.","proposal":"Introduce shared council coordinator helpers for report rendering","rationale":"This avoids formatter drift.","confidence":"medium","tags":["reporting"],"target_refs":{"repo_path":"."}}]}
EOF
cat <<'EOF' > TMPDIR/risk-review.json
{"reviewer_role":"risk-reviewer","findings":[{"title":"Lock contracts","summary":"Contract drift becomes risky over time.","proposal":"Move API contract definitions into a dedicated module.","rationale":"This reduces integration regressions.","confidence":"high","tags":["risk"],"target_refs":{"repo_path":"."}},{"title":"Cover JSON output","summary":"The council report response should stay stable.","proposal":"Add regression tests for council report JSON output.","rationale":"This catches contract regressions earlier.","confidence":"high","tags":["testing"],"target_refs":{"repo_path":"."}}]}
EOF
orch --db TMPDIR/coord.db --json council start \
--run council_blog_report_011 \
--target "Review the council reporting flow." \
--only-unanimous
THREAD_ID_CR1=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_011' AND task_id = 'CR1' AND attempt_no = 1;")
THREAD_ID_CR2=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_011' AND task_id = 'CR2' AND attempt_no = 1;")
THREAD_ID_CR3=$(sqlite3 TMPDIR/coord.db "SELECT thread_id FROM task_attempts WHERE run_id = 'council_blog_report_011' AND task_id = 'CR3' AND attempt_no = 1;")
inbox --db TMPDIR/coord.db --json claim --agent architecture-reviewer --thread "$THREAD_ID_CR1"
inbox --db TMPDIR/coord.db --json done --agent architecture-reviewer --thread "$THREAD_ID_CR1" --summary "Review complete" --body-file TMPDIR/architecture-review.json
inbox --db TMPDIR/coord.db --json claim --agent implementation-reviewer --thread "$THREAD_ID_CR2"
inbox --db TMPDIR/coord.db --json done --agent implementation-reviewer --thread "$THREAD_ID_CR2" --summary "Review complete" --body-file TMPDIR/implementation-review.json
inbox --db TMPDIR/coord.db --json claim --agent risk-reviewer --thread "$THREAD_ID_CR3"
inbox --db TMPDIR/coord.db --json done --agent risk-reviewer --thread "$THREAD_ID_CR3" --summary "Review complete" --body-file TMPDIR/risk-review.json
orch --db TMPDIR/coord.db --json council tally \
--run council_blog_report_011 \
--similarity normal
orch --db TMPDIR/coord.db --json council report \
--run council_blog_report_011
```
## 预期输出
- 最后一条 `council report` 命令退出码为 `0`
- `ok == true`
- `data.run_id == "council_blog_report_011"`
- `data.show == ["consensus"]`
- `data.summary.consensus == 1`
- `data.summary.majority == 1`
- `data.summary.minority == 1`
- `data.grouped_recommendations` 长度为 `1`
- 唯一返回的 recommendation 的 `bucket == "consensus"`
## 断言结论
- `--only-unanimous` 不会删除持久化的 `majority` 或 `minority` 数据，但会改变省略 `--show` 时的默认输出策略
- leader 若希望在 unanimous-only run 中仍查看 `majority`，必须显式传入 `--show`
## 补充约束
- 即使这里使用 `--json` 断言 `show` 默认值，命令仍会写出 markdown artifact
docs/tests/orch/council-report/council-report-json-shape-is-stable.md
+41
View File
@@ -0,0 +1,41 @@
# Case: `council-report-json-shape-is-stable`
## 用例意义
验证 `council report --json` 返回稳定 JSON 契约，包含 `show`、`summary`、过滤后的 grouped recommendations，以及 report artifact 元数据。
## 前置条件
- 已按 `council-report-defaults-to-consensus-and-majority` 的前置流程完成 reviewer 输出与 `council tally`
- 运行 ID 为 `council_blog_report_003`
## 输入
```bash
orch --db TMPDIR/coord.db --json council report \
--run council_blog_report_003
```
## 预期输出
- 退出码为 `0`
- `ok == true`
- `command == "council report"`
- `data.run_id == "council_blog_report_003"`
- `data.show == ["consensus","majority"]`
- `data.summary.consensus == 1`
- `data.summary.majority == 1`
- `data.summary.minority == 1`
- `data.report_artifacts` 长度为 `1`
- 首个 artifact 的 `kind == "markdown"`
- `data.grouped_recommendations` 长度为 `2`
- 第一组 recommendation 的 `bucket == "consensus"`
## 断言结论
- `--json` 模式返回的是 leader 可继续消费的稳定 machine-readable contract
- 默认 JSON 输出只返回被当前 `show` 过滤后的 recommendation，而 summary 仍保留全量 bucket 统计
## 补充约束
- 即使 `--json` 模式返回 artifact path，markdown artifact 仍应实际落盘
docs/tests/orch/council-report/council-report-rejects-before-tally.md
+32
View File
@@ -0,0 +1,32 @@
# Case: `council-report-rejects-before-tally`
## 用例意义
验证 `council report` 在还没有持久化 grouped recommendations 时会返回稳定的 `invalid_state` 契约，而不是生成空报告。
## 前置条件
- 使用隔离的临时目录 `TMPDIR`
- 当前数据库中尚未对该 council run 执行 `council tally`
## 输入
```bash
orch --db TMPDIR/coord.db --json council start \
--run council_blog_report_010 \
--target "Review the council reporting flow."
orch --db TMPDIR/coord.db --json council report \
--run council_blog_report_010
```
## 预期输出
- 第二条 `council report` 命令退出码为 `30`
- JSON 错误码为 `invalid_state`
- 错误消息指出 grouped recommendations 尚不可用，需先执行 `council tally`
## 断言结论
- `council report` 不是“边读 reviewer 输出边临时汇总”的命令
- report 阶段依赖已持久化的 `council_groups`，因此 `tally -> report` 的顺序是稳定 CLI 契约
docs/tests/orch/council-report/council-report-rejects-invalid-show.md
+29
View File
@@ -0,0 +1,29 @@
# Case: `council-report-rejects-invalid-show`
## 用例意义
验证 `council report --show` 对非法 bucket 值返回稳定的 `invalid_input`，避免 leader 误以为未知 bucket 会被静默忽略。
## 前置条件
- 已按 `council-report-defaults-to-consensus-and-majority` 的前置流程完成 reviewer 输出与 `council tally`
- 运行 ID 为 `council_blog_report_001`
## 输入
```bash
orch --db TMPDIR/coord.db --json council report \
--run council_blog_report_001 \
--show consensus,invalid
```
## 预期输出
- 退出码为 `30`
- JSON 错误码为 `invalid_input`
- 错误消息说明 `--show` 只接受 `consensus`、`majority`、`minority` 或 `all`
## 断言结论
- `--show` 不是宽松过滤参数；未知 bucket 会触发显式输入错误
- leader 侧脚本可以依赖这一点来尽早发现错误配置，而不是事后对空报告排障
docs/tests/orch/council-report/council-report-show-all-includes-minority.md
+29
View File
@@ -0,0 +1,29 @@
# Case: `council-report-show-all-includes-minority`
## 用例意义
验证 `council report --show all` 会把默认被省略的 `minority` recommendation 一并展示出来。
## 前置条件
- 已按 `council-report-defaults-to-consensus-and-majority` 的前置流程完成 reviewer 输出与 `council tally`
- 运行 ID 为 `council_blog_report_002`
## 输入
```bash
orch --db TMPDIR/coord.db council report \
--run council_blog_report_002 \
--show all
```
## 预期输出
- `council report` 退出码为 `0`
- stdout markdown 同时包含 `## Consensus`、`## Majority`、`## Minority`
- markdown 中出现 minority proposal，例如 `Add regression tests for council report JSON output.`
## 断言结论
- `--show all` 会覆盖默认的 bucket 过滤策略
- `minority` recommendation 会保留在持久化数据里，只是默认不进入主报告