Skip to content

feat(review): add opt-in post-review identity shadow - #720

Merged
zzwong merged 30 commits into
mainfrom
zzwong/mammoth-identity-shadow
Nov 10, 2026
Merged

zzwong merged 30 commits into
mainfrom
zzwong/mammoth-identity-shadow

Conversation

@zzwong

@zzwong zzwong commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Builds on the provider foundation in #714. Add cr review --identity-shadow --identity-shadow-config <path> to compare an explicitly configured System One model with the review's existing finding groups and prior-thread matching. The diagnostic runs after a newly completed review, its posting lock release, and completion-hook dispatch. It is off by default; a config path alone does nothing.

Current findings, verdicts, thread actions, posting, ledger records, fail-on behavior and normal output remain authoritative. The observer receives copied values and returns no decision to the review path. Resumed, retried, aborted and incomplete live reviews skip it. Dry-run opt-in can send diagnostic inputs to the configured inference provider while retaining the usual prohibition on Git-provider writes.

The implementation reuses the offline identity prompts, request preparation, provider adapter, receipts and replay machinery. Complete findings and eligible normalized prior opening bodies are retained in private run-owned artifacts, together with the observed baseline, bounded inference receipts and hash-bound comparisons. Baseline agreement measures agreement with current behavior; it is not an accuracy label or evidence of model qualification.

Limits and operation

The standalone strict YAML requires an HTTPS endpoint, model, exact expected response model, auth scheme and credential-store reference. Both complete pair universes are preflighted and output directories reserved before credentials or HTTP. The default qualification envelope is 40 calls, 327,680 aggregate request bytes, 6,144 state bytes, 8,192 request bytes, 10 seconds per call and a 30-second inference phase; oversized runs are refused rather than sampled. There are no automatic retries or profile/init defaults.

Credential lookup precedes the inference timer and retains the selected backend's existing synchronous bound. It can therefore add command latency after the authoritative review has completed. Local publication adds further time. Unknown prices or usage remain unknown; these limits do not promise a hard dollar ceiling. Ordinary diagnostic errors produce private receipts and sanitized stderr warnings without changing the completed review.

Validation

  • Full make check passed: tidy, formatting, zero lint issues, the complete Go test suite and build. Race checks passed for internal/identityexperiment, internal/identityshadow and internal/app.
  • A real local Live/pipeline/outbox/ledger regression compares shadow off/on for success, malformed output, cancellation, invalid config and comparison-publication failure. It verifies both native findings, all five posted actions, thread reply/resolution, fail-on behavior and unchanged authoritative state; the same posting lock is held at submission and released before observer entry, after completion-hook dispatch. Deliberately contradictory shadow predictions measure three disagreements without filtering findings. JSON coverage combines the actual pipeline presentation with every live-result field consumed by the command.
  • Main-branch integration retains the flat workspace layout and successful-checkout cleanup. The live equivalence regression derives exact workspace paths independently, preserves unknown paths and hashes, verifies both checkout locations are removed, and confirms diagnostic artifacts remain under their owning run.
  • Snapshot and observer regressions cover default-off zero work, selected-store cleanup, source eligibility, copied inputs, baseline isolation, complete-universe preflight, private artifact reservation/publication failures, inference budgets, retained accounting and all resumed/recovery exclusions.
  • Review follow-ups preserve unresolved task status in the aggregate receipt and manifest, including fully answered but contradictory cohorts. Added regressions independently verify the aggregate byte cap, the shared configured deadline after credential lookup, and exact prior-match direction. Temporary mutations removing each guard fail the intended tests; all original calls and findings remain accounted for.
  • A compiled CLI smoke verifies both flags, empty-fixture completion and budget refusal with zero provider calls and a nonexistent synthetic credential ref. Retained payload hashes, private permissions and independent offline replay all match. All feature verification used local fixtures, synthetic credentials and mock providers; no live review or paid inference was performed for these checks.

This feature does not enable the diagnostic on live reviews or change any model-selection or posting policy.

zzwong added 24 commits November 6, 2026 19:16
Add an explicit provider command for caller-owned cohort and prior-thread fixtures, with all-pair preflight and retained replay artifacts. Keep provider access isolated from live reviews and record partial failures, usage, and cost estimates without credentials.
Separate the reusable choice client from report identity prompts and make model, endpoint, authentication, limits and prices explicit caller settings. Preserve private replay artifacts and distinguish unknown costs from known zero.
Exercise TypeSafe authentication and resolved model handling with the shared identity adapter. Add small synthetic cohort and prior-thread fixtures for provider smoke checks.
Validate named choice answers and cache usage through a dedicated Decisions transport, retaining safe metadata on refusals and failed requests.
Reuse the System One codec for Clef and Clef-flash requests while validating Cloudflare envelopes, routing and usage. Mark provider context completeness as unverified.
Replace only the usage value so valid response metadata cannot inflate past the wire size guard. Retain known usage and the original response digest for ASCII and Unicode metadata.
Use one versioned report projection and replay runner for System One and Decisions. Preserve request provenance, retained originals and failure accounting while keeping cache-dependent costs explicitly unknown.
Reuse the shared identity question and replay runner for Cloudflare models. Require an explicit context acknowledgement and retain unverified context, usage and complete original records in private receipts.

@zzwong zzwong left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An inconsistent decision cycle can mark the diagnostic complete. Add regression coverage for aggregate-byte limits, the inference deadline and exact prior-pair request contents. The status defect affects diagnostic completeness without changing the authoritative verdict or posting.

Comment thread internal/identityshadow/compare_test.go Resolved
Comment thread internal/identityshadow/observer.go Resolved
Base automatically changed from zzwong/identity-credential-replay-contracts to zzwong/shared-identity-experiment-core November 10, 2026 14:24
Base automatically changed from zzwong/shared-identity-experiment-core to zzwong/offline-clef-identity November 10, 2026 14:35
Base automatically changed from zzwong/offline-clef-identity to zzwong/offline-openai-decisions-identity November 10, 2026 14:41
Add Decisions request and answer handling to provider-driven identity experiments while preserving refusals, cache uncertainty and original evidence. Include shared provider execution, selected-store credentials and replay contract enforcement.
Base automatically changed from zzwong/offline-openai-decisions-identity to zzwong/offline-mammoth-identity November 10, 2026 14:57
@zzwong
zzwong marked this pull request as ready for review November 10, 2026 15:00
@zzwong
zzwong deleted the branch zzwong/offline-mammoth-identity November 10, 2026 15:06
@zzwong zzwong closed this Nov 10, 2026
…ty-shadow

# Conflicts:
#	docs/architecture.md
#	internal/architecture/report_identity_test.go
@zzwong zzwong reopened this Nov 10, 2026
@zzwong
zzwong changed the base branch from zzwong/offline-mammoth-identity to main November 10, 2026 15:07
@zzwong zzwong closed this Nov 10, 2026
@zzwong zzwong reopened this Nov 10, 2026
@zzwong
zzwong merged commit 7970a83 into main Nov 10, 2026
12 checks passed
@zzwong
zzwong deleted the zzwong/mammoth-identity-shadow branch November 10, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant