diff --git a/retrospectives/2026-11-18-verification.md b/retrospectives/2026-11-18-verification.md new file mode 100644 index 0000000..eba474b --- /dev/null +++ b/retrospectives/2026-11-18-verification.md @@ -0,0 +1,52 @@ +# How the first Countersign release was verified + +The protocol agent took the original seven handbook screenshots on Linux ARM64. It drove a packaged Electron Companion and a separate Chromium browser while curl ran in Docker Compose. For the selected-certificate image, the agent chose the second of two identities with the same display name, approved the request, and compared the fingerprint on screen with the one the mTLS site returned. That comparison is the evidence that the remote workload used the chosen certificate. [Original screenshots and walkthrough](https://github.com/zeroae/countersign/pull/2) · [Packaged demo driver](../examples/demo-integrator/compose_demo.py) + +## Findings, in priority order + +### 1. Keep the live handshake evidence alongside the illustration + +The original capture driver ran outside the repository. It started the demo with a disposable browser profile, SoftHSM2 identities and a GNOME keyring. Xvfb provided the Linux display and D-Bus provided the desktop session. A temporary desktop opener routed pairing URLs into the test browser. Playwright attached over the Chrome DevTools Protocol to **two processes**. It drove the browser confirmation and the Electron renderer separately. + +For the approval card, the agent waited for two identities and a disabled Approve button, then used a Playwright element screenshot before and after selection. To include browser chrome, the driver sized the Chromium window, located its X11 window with `xwininfo`, captured it with `xwd` and converted the result to PNG with FFmpeg. A second paired demo origin supplied the alternate brand. The wildcard and alternate-brand prompts were denied after capture. The parent separately served the built handbook and opened it in the T3 browser for the owner. + +The screenshots on current main have a different provenance. `pnpm screenshots` renders the real UI against fixture responses and a stubbed preload API, then annotates the images using measured element bounds. This makes a visual change reproducible. The executable evidence for certificate use lives in the packaged handshake test. [Fixture renderer](../scripts/capture_screenshots.mjs) · [Annotation script](../scripts/annotate_screenshots.py) · [Capture instructions](../docs/develop.md) + +**Suggested improvement.** Have the packaged handshake job attach a small result containing the tested commit, OS/architecture, number of identities, selected public fingerprint, upstream fingerprint and revocation result. Link that artifact when an image accompanies a behavioral claim. The existing two- and three-identity tests already perform the expensive part. [End-to-end workflow](../.github/workflows/e2e.yml) + +### 2. Make the verification handoff survive a stopped agent + +The parent resumed the protocol and issuer agents in the shared checkout. Their later reports added evidence that was absent from their initial handoffs. The following results are from their recorded work and the linked runs. + +| Agent | What it ran or inspected | What the result established | +| --- | --- | --- | +| Protocol | Ran Cargo tests and Clippy, inspected six hosted native jobs, then extracted the hosted Linux ARM64 Debian package and used its executable in the local Compose demo. | Each native job exercised 13 TLS handshake cases with the production addon. The separate Linux demo selected one of three certificates and matched the upstream fingerprint. Coverage was for the exercised software providers. [Native jobs](https://github.com/zeroae/countersign/actions/runs/44189458753) · [Desktop build](https://github.com/zeroae/countersign/actions/runs/44189459020) | +| Issuer and Gateway | Ran the Rust tests, an ACM-PCA SDK fixture and a multi-process Gateway fixture. Shortened a CRL deadline on an already idle connection, reproduced the failure on `2ee03d8`, and checked closure of both socket legs after the fix. Also checked the documented env exports against the executable and recovery against a temporary database. | Exercised worker and lease behavior locally. Live AWS Private CA and ZAE-ZCM qualification remained open. [Gateway fixture](../crates/countersign-gateway/tests/runtime.py) · [Implementation PR](https://github.com/zeroae/countersign/pull/1) | +| Broker | Reported 16 Broker and nine protocol tests passing locally. Exercised late signatures after revocation, concurrent decisions and recovery fencing. Later ran four widget tests and exercised the documented adapter in a simulated browser. | Checked Broker state transitions and the widget's ID-only link. The parent owned whole-site rendering. [Broker tests](../crates/countersign-broker/tests) · [Widget tests](../packages/widget/test) | +| Spec-only UAT | Read the original design at `9d507a1` and produced 33 stories with acceptance criteria. | Delivered a checklist for independent execution. The agent ran no product code. [UAT issue](https://github.com/zeroae/countersign/issues/3) | + +**Suggested improvement.** On interruption, persist the agent's last tested commit, exact command or hosted job URL, observed result and pending check in the PR. This distinguishes a locally passing change in a shared checkout from evidence for the uploaded artifact. + +### 3. Turn the one-off handbook smoke test into a reusable check + +The parent built MkDocs in a temporary Python environment, served the generated site on an ephemeral loopback port and launched Playwright Chromium. Its smoke script opened twelve handbook pages at 1440×1000 and 390×844, checked image loading, horizontal overflow and page errors, then exercised search and the dark palette. The script contained an absolute path into one machine’s Playwright cache. The docs workflow runs a strict MkDocs build. Those browser assertions still live outside CI. [Documentation workflow](../.github/workflows/docs.yml) + +**Suggested improvement.** Keep the viewport and search checks in a small tracked script, use Playwright’s installed-browser discovery, and run it against the built site on documentation PRs. Save a screenshot of the failing page and viewport on failure. + +## Reproducing the live check + +Follow the desktop prerequisites and fixture setup in the walkthrough, then run the automated verification with the packaged executable. On a headless Linux host with Xvfb, the command shape used by the parent was: + +```sh +timeout 240 xvfb-run -a .venv/bin/python examples/demo-integrator/compose_demo.py \ + --verify --app /absolute/path/to/countersign-companion \ + --browser /absolute/path/to/chromium +``` + +The launcher creates the isolated desktop state and D-Bus session. The CI harness also drives the packaged app with two and three identities under `dbus-run-session` and `xvfb-run`. Windows and macOS evidence came from their own hosted runners. Physical tokens, hardware PIN/touch prompts and real protected sites require separate qualification. [Linux walkthrough](../docs/try.md) · [CI entry point](../examples/demo-integrator/test/e2e.py) · [Platform evidence](../docs/reference/support.md) + +## How the agent tools were set up + +The parent selected the repository root as the Rust Analyzer workspace through the Rust tool connector and requested diagnostics. The connector reported unresolved Serde macro diagnostics early in the session. The agents also ran Cargo builds, tests and Clippy. For JavaScript the parent installed TypeScript and `typescript-language-server` under a temporary npm prefix, started the server with `--stdio`, initialized it with the workspace URI, opened the Companion source and requested document symbols. That probe established that the server could read the source. ESLint and the Node tests supplied the lint and runtime results. [Workspace checks](../.github/workflows/ci.yml) + +This retrospective used the parent’s and four children’s session records, the original local capture and smoke scripts, and repository workflows. The hosted job links identify historical runs. No platform qualification was rerun for this review.