diff --git a/docs/exec-plans/2026-03-24-harness-control-plane-hardening.md b/docs/exec-plans/2026-03-24-harness-control-plane-hardening.md new file mode 100644 index 0000000..0e61e21 --- /dev/null +++ b/docs/exec-plans/2026-03-24-harness-control-plane-hardening.md @@ -0,0 +1,83 @@ +# Harness Control Plane Hardening + +- Status: `in-progress` +- Owner: `sisyphus` +- Date: `2026-03-24` + +## Goal + +Finish the next harness-engineering hardening slice by making changed-suite selection authoritative in +CI, promoting harness artifacts to first-class debugging outputs, centralizing browser and a11y +coverage under one explicit contract, and turning orchestration from a thin wrapper into a safer +repo capability with its own tests and docs. + +## Scope + +- In scope: + - broaden harness selection so high-risk control-plane and release changes escalate to the right + validation gate + - make CI run the selected harness suites instead of always forcing the PR gate + - upload harness, a11y, and browser test artifacts from CI + - replace duplicated story coverage lists with one shared harness coverage contract + - improve orchestration binary discovery, preflight checks, and plan-linked dispatch support + - add tests for harness control-plane helpers and update docs to match the implementation +- Out of scope: + - redesigning the component library or Storybook content beyond harness metadata needs + - replacing the external orchestration engine itself + - introducing visual snapshot infrastructure or hosted browser grids + +## Constraints + +- Preserve the existing `pnpm harness:*` commands as the main public interface. +- Keep package-first release validation as the canonical release path. +- Reuse execution plans as the source of orchestration intent instead of creating a parallel plan + format. +- Keep new harness checks targeted so CI cost stays proportional to risk. + +## Affected Surfaces + +- `scripts/harness/*` +- `tests/e2e/*` +- `.github/workflows/harness-validate.yml` +- `package.json` +- `vitest.config.ts` +- `docs/harness-engineering.md` +- `docs/orchestration.md` +- `README.md` +- `CONTRIBUTING.md` + +## Plan + +1. Refactor harness suite selection so release-risk and harness-risk changes escalate beyond + static checks and let `changed` execution become the CI entrypoint. +2. Promote `.artifacts/*` outputs into uploaded CI artifacts and make changed-suite execution log + its selected suites clearly. +3. Introduce one shared harness coverage contract for Storybook browser smoke and a11y checks, + then move both runners onto it. +4. Add orchestration preflight and plan-linked dispatch helpers so execution plans directly support + orchestration runs. +5. Add control-plane tests plus doc updates so harness behavior, workflow guidance, and public + commands stay aligned. + +## Validation + +- `pnpm exec vitest run scripts/harness/**/*.test.ts` +- `pnpm harness:suites` +- `pnpm harness:select -- --json --changed-file scripts/harness/core.mjs` +- `pnpm harness:validate:changed -- --dry-run --changed-file scripts/harness/core.mjs` +- `pnpm test:e2e:a11y` +- `pnpm test:e2e:smoke` +- `pnpm harness:validate:pr` + +## Orchestration Task Sketch + +- `T1`: harden suite selection and CI execution flow +- `T2`: centralize browser and a11y coverage contract +- `T3`: improve orchestration preflight and plan-linked dispatch +- `T4`: add harness tests and reconcile docs + +## Status Log + +- `2026-03-24 18:40` reviewed harness docs, scripts, CI wiring, and recent release hardening work; + confirmed the main gaps are authoritative suite selection, CI artifact visibility, explicit + browser coverage ownership, orchestration preflight, and missing control-plane tests diff --git a/docs/exec-plans/TEMPLATE.md b/docs/exec-plans/TEMPLATE.md index 991f215..15cafd0 100644 --- a/docs/exec-plans/TEMPLATE.md +++ b/docs/exec-plans/TEMPLATE.md @@ -40,6 +40,9 @@ Describe the outcome in one short paragraph. ## Orchestration Task Sketch - Optional task IDs and dependencies when the work should be dispatched through `pnpm harness:orch` +- Suggested format: + - `T1`: isolated first slice + - `T2 -> T1`: follow-up slice that depends on `T1` ## Status Log diff --git a/docs/harness-engineering.md b/docs/harness-engineering.md index 1f8cf66..a66b856 100644 --- a/docs/harness-engineering.md +++ b/docs/harness-engineering.md @@ -63,6 +63,8 @@ Primary suites: - lint, typecheck, and unit coverage for normal component work - `pnpm harness:validate:docs` - Storybook build for docs-surface changes +- `pnpm harness:validate:a11y` + - curated Storybook accessibility validation from the shared browser coverage contract - `pnpm harness:validate:docs-smoke` - Playwright smoke coverage for high-value Storybook flows - `pnpm harness:validate:consumers` @@ -76,6 +78,9 @@ Primary suites: Each run writes a JSON report to `.artifacts/harness/.json`. +GitHub Actions uploads the generated harness, a11y, and browser test artifacts from `.artifacts/` +so CI failures stay debuggable after the run finishes. + ## Working Loop Recommended change loop: @@ -98,6 +103,10 @@ Harness selection is exposed through `pnpm harness:select` and `pnpm harness:val - `pnpm harness:validate:changed` - runs the selected suites with a JSON report +The CI workflow uses `pnpm harness:validate:changed` as its execution entrypoint. Selection is no +longer advisory-only: harness-risk and release-risk surfaces escalate directly to the broader `pr` +or `release` gates instead of falling back to `static`. + Selection intentionally maps repo surfaces to validation surfaces: - package source changes select `component`, `docs`, `docs-smoke`, and `consumers` @@ -105,6 +114,25 @@ Selection intentionally maps repo surfaces to validation surfaces: - registry/consumer changes select `static` and `consumers` - doc-only or metadata-only changes may select no suites +High-risk control-plane changes override those narrow mappings: + +- `scripts/harness/*` and other harness-control files escalate to `pr` +- release workflows, release docs, registry docs, and consumer/release scripts escalate to `release` + +## Browser Coverage Contract + +Curated browser coverage lives in `tests/e2e/support/story-harness-contract.json`. + +That contract is the shared source of truth for: + +- `pnpm harness:validate:a11y` +- `pnpm harness:validate:docs-smoke` + +Each entry records the story id, why it is covered, which suites own it, and any required +interaction scenario keys. When a new high-value review surface should participate in browser or +accessibility validation, update this contract instead of adding one-off story lists in multiple +scripts. + ## Worktree Orchestration Cadence UI also exposes an orchestration wrapper: @@ -114,23 +142,35 @@ Cadence UI also exposes an orchestration wrapper: The wrapper keeps orchestration state under `.artifacts/orch/` and applies worktree defaults for dispatch. Details live in [docs/orchestration.md](/Users/xd/project/cadence-ui/docs/orchestration.md). +Use `pnpm harness:orch -- doctor` to verify binary discovery and plan parsing on a new machine, and +prefer `dispatch --plan-file ` so execution plans remain the source of truth for generated +task bodies. + ## Current Rollout Scope This repository is adding harness engineering in phases. -Phase 1 establishes: +Phase 1 established: - a documented harness workflow - execution-plan conventions - shared validation suites -- a pull request workflow that runs the PR suite +- a pull request workflow that runs the repository harness gate -Phase 2 adds: +Phase 2 established: - change-aware suite selection from git diff - a stabilized Storybook smoke harness - worktree-oriented orchestration defaults +Phase 3 hardens: + +- authoritative changed-suite execution in CI +- uploaded CI artifacts for harness, a11y, and browser test output +- a shared browser coverage contract +- orchestration preflight and plan-linked dispatch helpers +- harness self-tests for selection, artifact, and orchestration helpers + Future phases can layer on: - richer browser/app harnesses for interactive review diff --git a/docs/orchestration.md b/docs/orchestration.md index 8c04796..28fef92 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -12,6 +12,14 @@ Use the repository wrapper instead of calling `orch` directly: pnpm harness:orch -- [flags] ``` +The wrapper also exposes repo-local helper commands: + +```bash +pnpm harness:orch -- doctor --plan-file docs/exec-plans/my-plan.md +pnpm harness:orch -- plan inspect --plan-file docs/exec-plans/my-plan.md +pnpm harness:orch -- plan body --run cadence_ui_demo --task T1 --plan-file docs/exec-plans/my-plan.md +``` + The wrapper applies these defaults: - orchestration database: `.artifacts/orch/coord.db` @@ -20,13 +28,24 @@ The wrapper applies these defaults: - strict worktree mode for dispatch - `--base-ref` defaults to the current branch if not provided +Binary resolution order: + +1. `CADENCE_UI_ORCH_BIN` +2. `orch` on `PATH` +3. the legacy `~/.codex/skills/orch/assets/orch` location + +If no binary is found, `pnpm harness:orch -- doctor` prints the checked locations and exits +non-zero so the failure is explicit instead of looking like a broken wrapper. + ## Suggested Workflow 1. Write or update an execution plan in `docs/exec-plans/`. -2. Create an orchestration run. -3. Add tasks and dependencies that map to the execution plan. -4. Dispatch ready tasks into isolated worktrees. -5. Reconcile worker state and run the relevant harness suites before merge. +2. Inspect the plan task sketch with `pnpm harness:orch -- plan inspect --plan-file `. +3. Create an orchestration run. +4. Add tasks and dependencies that map to the execution plan. +5. Dispatch ready tasks into isolated worktrees, ideally with `--plan-file` so the wrapper can + generate the task body from the plan automatically. +6. Reconcile worker state and run the relevant harness suites before merge. ## Example @@ -67,9 +86,13 @@ pnpm harness:orch -- dispatch \ --run cadence_ui_harness_001 \ --task T1 \ --to default-worker \ - --body-file docs/exec-plans/task-t1.md + --plan-file docs/exec-plans/2026-03-24-harness-control-plane-hardening.md ``` +When `--plan-file` is provided without `--body-file`, the wrapper parses the plan's +`## Orchestration Task Sketch`, finds the matching task id, and writes a generated task body under +`.artifacts/orch/task-bodies//.md` before dispatching. + Inspect status: ```bash @@ -89,9 +112,20 @@ the plan: - task bodies should point back to the execution plan and the relevant harness suites - workers should validate the narrowest useful suites first, then report what remains +Suggested task sketch format: + +```md +- `T1`: stabilize docs smoke coverage +- `T2 -> T1`: reconcile a11y coverage and docs after the smoke slice lands +``` + +`T2 -> T1` means task `T2` depends on `T1`. + ## Safety Notes - dispatch from a committed or otherwise reviewable base when possible - keep shared integration files on the leader side when multiple workers are active - prefer one task per isolated write scope - use `status`, `blocked`, `answer`, and `reconcile` instead of ad hoc coordination +- run `pnpm harness:orch -- doctor` before the first dispatch on a new machine so binary discovery, + defaults, and plan parsing all fail fast