kurihada/cadence-ui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(harness): document control plane workflow

Browse Source 
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This commit is contained in:
kurihada
2026-03-24 18:34:56 +08:00
parent 704cacfd8d
commit 800db360b3
4 changed files with 168 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/exec-plans/2026-03-24-harness-control-plane-hardening.md
+83
View File
@@ -0,0 +1,83 @@
# Harness Control Plane Hardening
- Status: `in-progress`
- Owner: `sisyphus`
- Date: `2026-03-24`
## Goal
Finish the next harness-engineering hardening slice by making changed-suite selection authoritative in
CI, promoting harness artifacts to first-class debugging outputs, centralizing browser and a11y
coverage under one explicit contract, and turning orchestration from a thin wrapper into a safer
repo capability with its own tests and docs.
## Scope
- In scope:
- broaden harness selection so high-risk control-plane and release changes escalate to the right
validation gate
- make CI run the selected harness suites instead of always forcing the PR gate
- upload harness, a11y, and browser test artifacts from CI
- replace duplicated story coverage lists with one shared harness coverage contract
- improve orchestration binary discovery, preflight checks, and plan-linked dispatch support
- add tests for harness control-plane helpers and update docs to match the implementation
- Out of scope:
- redesigning the component library or Storybook content beyond harness metadata needs
- replacing the external orchestration engine itself
- introducing visual snapshot infrastructure or hosted browser grids
## Constraints
- Preserve the existing `pnpm harness:*` commands as the main public interface.
- Keep package-first release validation as the canonical release path.
- Reuse execution plans as the source of orchestration intent instead of creating a parallel plan
format.
- Keep new harness checks targeted so CI cost stays proportional to risk.
## Affected Surfaces
- `scripts/harness/*`
- `tests/e2e/*`
- `.github/workflows/harness-validate.yml`
- `package.json`
- `vitest.config.ts`
- `docs/harness-engineering.md`
- `docs/orchestration.md`
- `README.md`
- `CONTRIBUTING.md`
## Plan
1. Refactor harness suite selection so release-risk and harness-risk changes escalate beyond
static checks and let `changed` execution become the CI entrypoint.
2. Promote `.artifacts/*` outputs into uploaded CI artifacts and make changed-suite execution log
its selected suites clearly.
3. Introduce one shared harness coverage contract for Storybook browser smoke and a11y checks,
then move both runners onto it.
4. Add orchestration preflight and plan-linked dispatch helpers so execution plans directly support
orchestration runs.
5. Add control-plane tests plus doc updates so harness behavior, workflow guidance, and public
commands stay aligned.
## Validation
- `pnpm exec vitest run scripts/harness/**/*.test.ts`
- `pnpm harness:suites`
- `pnpm harness:select -- --json --changed-file scripts/harness/core.mjs`
- `pnpm harness:validate:changed -- --dry-run --changed-file scripts/harness/core.mjs`
- `pnpm test:e2e:a11y`
- `pnpm test:e2e:smoke`
- `pnpm harness:validate:pr`
## Orchestration Task Sketch
- `T1`: harden suite selection and CI execution flow
- `T2`: centralize browser and a11y coverage contract
- `T3`: improve orchestration preflight and plan-linked dispatch
- `T4`: add harness tests and reconcile docs
## Status Log
- `2026-03-24 18:40` reviewed harness docs, scripts, CI wiring, and recent release hardening work;
confirmed the main gaps are authoritative suite selection, CI artifact visibility, explicit
browser coverage ownership, orchestration preflight, and missing control-plane tests
docs/exec-plans/TEMPLATE.md
+3
View File
@@ -40,6 +40,9 @@ Describe the outcome in one short paragraph.
## Orchestration Task Sketch
- Optional task IDs and dependencies when the work should be dispatched through `pnpm harness:orch`
- Suggested format:
- `T1`: isolated first slice
- `T2 -> T1`: follow-up slice that depends on `T1`
## Status Log
docs/harness-engineering.md
+43 -3
View File
@@ -63,6 +63,8 @@ Primary suites:
- lint, typecheck, and unit coverage for normal component work
- `pnpm harness:validate:docs`
- Storybook build for docs-surface changes
- `pnpm harness:validate:a11y`
- curated Storybook accessibility validation from the shared browser coverage contract
- `pnpm harness:validate:docs-smoke`
- Playwright smoke coverage for high-value Storybook flows
- `pnpm harness:validate:consumers`
@@ -76,6 +78,9 @@ Primary suites:
Each run writes a JSON report to `.artifacts/harness/<suite>.json`.
GitHub Actions uploads the generated harness, a11y, and browser test artifacts from `.artifacts/`
so CI failures stay debuggable after the run finishes.
## Working Loop
Recommended change loop:
@@ -98,6 +103,10 @@ Harness selection is exposed through `pnpm harness:select` and `pnpm harness:val
- `pnpm harness:validate:changed`
- runs the selected suites with a JSON report
The CI workflow uses `pnpm harness:validate:changed` as its execution entrypoint. Selection is no
longer advisory-only: harness-risk and release-risk surfaces escalate directly to the broader `pr`
or `release` gates instead of falling back to `static`.
Selection intentionally maps repo surfaces to validation surfaces:
- package source changes select `component`, `docs`, `docs-smoke`, and `consumers`
@@ -105,6 +114,25 @@ Selection intentionally maps repo surfaces to validation surfaces:
- registry/consumer changes select `static` and `consumers`
- doc-only or metadata-only changes may select no suites
High-risk control-plane changes override those narrow mappings:
- `scripts/harness/*` and other harness-control files escalate to `pr`
- release workflows, release docs, registry docs, and consumer/release scripts escalate to `release`
## Browser Coverage Contract
Curated browser coverage lives in `tests/e2e/support/story-harness-contract.json`.
That contract is the shared source of truth for:
- `pnpm harness:validate:a11y`
- `pnpm harness:validate:docs-smoke`
Each entry records the story id, why it is covered, which suites own it, and any required
interaction scenario keys. When a new high-value review surface should participate in browser or
accessibility validation, update this contract instead of adding one-off story lists in multiple
scripts.
## Worktree Orchestration
Cadence UI also exposes an orchestration wrapper:
@@ -114,23 +142,35 @@ Cadence UI also exposes an orchestration wrapper:
The wrapper keeps orchestration state under `.artifacts/orch/` and applies worktree defaults for
dispatch. Details live in [docs/orchestration.md](/Users/xd/project/cadence-ui/docs/orchestration.md).
Use `pnpm harness:orch -- doctor` to verify binary discovery and plan parsing on a new machine, and
prefer `dispatch --plan-file <plan>` so execution plans remain the source of truth for generated
task bodies.
## Current Rollout Scope
This repository is adding harness engineering in phases.
Phase 1 establishes:
Phase 1 established:
- a documented harness workflow
- execution-plan conventions
- shared validation suites
- a pull request workflow that runs the PR suite
- a pull request workflow that runs the repository harness gate
Phase 2 adds:
Phase 2 established:
- change-aware suite selection from git diff
- a stabilized Storybook smoke harness
- worktree-oriented orchestration defaults
Phase 3 hardens:
- authoritative changed-suite execution in CI
- uploaded CI artifacts for harness, a11y, and browser test output
- a shared browser coverage contract
- orchestration preflight and plan-linked dispatch helpers
- harness self-tests for selection, artifact, and orchestration helpers
Future phases can layer on:
- richer browser/app harnesses for interactive review
docs/orchestration.md
+39 -5
View File
@@ -12,6 +12,14 @@ Use the repository wrapper instead of calling `orch` directly:
pnpm harness:orch -- <orch command> [flags]
```
The wrapper also exposes repo-local helper commands:
```bash
pnpm harness:orch -- doctor --plan-file docs/exec-plans/my-plan.md
pnpm harness:orch -- plan inspect --plan-file docs/exec-plans/my-plan.md
pnpm harness:orch -- plan body --run cadence_ui_demo --task T1 --plan-file docs/exec-plans/my-plan.md
```
The wrapper applies these defaults:
- orchestration database: `.artifacts/orch/coord.db`
@@ -20,13 +28,24 @@ The wrapper applies these defaults:
- strict worktree mode for dispatch
- `--base-ref` defaults to the current branch if not provided
Binary resolution order:
1. `CADENCE_UI_ORCH_BIN`
2. `orch` on `PATH`
3. the legacy `~/.codex/skills/orch/assets/orch` location
If no binary is found, `pnpm harness:orch -- doctor` prints the checked locations and exits
non-zero so the failure is explicit instead of looking like a broken wrapper.
## Suggested Workflow
1. Write or update an execution plan in `docs/exec-plans/`.
2. Create an orchestration run.
3. Add tasks and dependencies that map to the execution plan.
4. Dispatch ready tasks into isolated worktrees.
5. Reconcile worker state and run the relevant harness suites before merge.
2. Inspect the plan task sketch with `pnpm harness:orch -- plan inspect --plan-file <plan>`.
3. Create an orchestration run.
4. Add tasks and dependencies that map to the execution plan.
5. Dispatch ready tasks into isolated worktrees, ideally with `--plan-file` so the wrapper can
generate the task body from the plan automatically.
6. Reconcile worker state and run the relevant harness suites before merge.
## Example
@@ -67,9 +86,13 @@ pnpm harness:orch -- dispatch \
--run cadence_ui_harness_001 \
--task T1 \
--to default-worker \
--body-file docs/exec-plans/task-t1.md
--plan-file docs/exec-plans/2026-03-24-harness-control-plane-hardening.md
```
When `--plan-file` is provided without `--body-file`, the wrapper parses the plan's
`## Orchestration Task Sketch`, finds the matching task id, and writes a generated task body under
`.artifacts/orch/task-bodies/<run>/<task>.md` before dispatching.
Inspect status:
```bash
@@ -89,9 +112,20 @@ the plan:
- task bodies should point back to the execution plan and the relevant harness suites
- workers should validate the narrowest useful suites first, then report what remains
Suggested task sketch format:
```md
- `T1`: stabilize docs smoke coverage
- `T2 -> T1`: reconcile a11y coverage and docs after the smoke slice lands
```
`T2 -> T1` means task `T2` depends on `T1`.
## Safety Notes
- dispatch from a committed or otherwise reviewable base when possible
- keep shared integration files on the leader side when multiple workers are active
- prefer one task per isolated write scope
- use `status`, `blocked`, `answer`, and `reconcile` instead of ad hoc coordination
- run `pnpm harness:orch -- doctor` before the first dispatch on a new machine so binary discovery,
defaults, and plan parsing all fail fast