Skip to content

feat(evaluations): resolve a served variation when variation is omitted - #299

Closed
donei003 wants to merge 2 commits into
mainfrom
cursor/eval-served-variation-700b
Closed

donei003 wants to merge 2 commits into
mainfrom
cursor/eval-served-variation-700b

Conversation

@donei003

@donei003 donei003 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Overview

#277 is on main. This PR targets main. The diff is only this change. Draft.

AIConfig.variation is optional.

  • A blank evaluation key is an error before any network call. feat(evaluations): reconcile offline evaluations by key #277 already rejects it; this PR pins "" and whitespace, including when variation is omitted.
  • When variation is set, the base is still that variation's latest management-API version. The flag is not evaluated.
  • When variation is omitted, run() calls extract_variation with {kind: "evaluation", key: <evaluation key>}. There is no caller context and no project-key fallback.
  • The version is the raw _ldMeta.version. The version-1 default inside extract_variation is not used. A missing or non-integer version, including True, is an EvaluationsError before any write.
  • A disabled variation, or a config that is not delivered, is an EvaluationsError before any write.
  • Content is that version from the management API, then the existing generation overrides. A served version older than the latest is valid. The SDK does not warn.
  • result.url is unchanged: {ui}/projects/{project}/ai/evaluations/{evaluationId}/runs/{runId}.

configKey, variationKey, and variationVersion are the next PR in this stack.

Decisions

Parameter overrides still merge key by key, as they do today. A key the caller sets wins, and other parameter keys stay. That is the layering already applied on top of a variation.

Testing

uv run pytest packages/client/tests -q

2221 passed. packages/client/api-report/launchdarkly_ai_server.txt is regenerated for AIConfig.variation: str | None = None.

Open in Web Open in Cursor 

@donei003
donei003 added this pull request to stack #300 November 12, 2026 21:10
@cursor

cursor Bot commented Nov 12, 2026

Copy link
Copy Markdown

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 113182a. Configure here.

@cursor

cursor Bot commented Nov 12, 2026

Copy link
Copy Markdown

bugbot run

@donei003 donei003 added the api-compatible An owner judged a flagged public API change compatible; passes the API compatibility check label Nov 12, 2026 — with Cursor
Base automatically changed from cursor/offline-evaluations-sdk-4a4c to main November 16, 2026 20:28
cursoragent and others added 2 commits November 16, 2026 12:28
AIConfig.variation is optional. When it is omitted, run() evaluates the
config for {kind: "evaluation", key: <evaluation key>} and uses that
variation's _ldMeta.version. A missing version is an error before any write.

Co-authored-by: Dylan O'Neill <donei003@users.noreply.github.com>
An omitted variation evaluates the AI Config through extract_variation,
which initializes the SDK with no options. Resolve the client first so the
evaluations sdk_key is used, as the judge path already does.

Co-authored-by: Dylan O'Neill <donei003@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible An owner judged a flagged public API change compatible; passes the API compatibility check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants