Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ The evaluations harness reads an LD-hosted dataset, reconciles a keyed offline e

Each row is also exported as one trace on the tracer provider `init_client` registered: a `launchdarkly.dataset_row` root with the generation, its tool calls, and each criterion's `evaluate` span beneath it, all sharing the row's `gen_ai.conversation.id`. A provider handler traces its own call; for a plain function the runner writes the `chat` and `execute_tool` spans. Prompts, output, tool payloads, and judge explanations follow the same [`capture_content`](#process-wide-defaults) gate as every other span. The built-in default sends them. Setting capture to false removes them, including on these evaluation spans. The `$ld:ai:offline-evals:*` events still carry no transcript.

Management API requests go to the `api_base_uri` option, then `LD_API_BASE_URI`, then `https://app.launchdarkly.com`; `LD_BASE_URI` is never used, because it may point at a relay proxy. Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. At either level a blank value counts as unset, and a trailing `/` is stripped. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts, plus each criterion's counts and pass-rate outcome in `criteria` where LaunchDarkly reports them. When the summary reports a criterion with a pass rate threshold (every criterion has one by default, see below), the harness then waits up to 30s for LaunchDarkly's verdict on the run and returns it as `result.verdict` (`"pass"` or `"fail"`). `result.passed` follows the verdict. The run fails when any criterion's pass rate, with errored rows counted against it, is below its threshold. Criteria without a pass rate threshold are left out of the verdict. If no verdict arrives, or LaunchDarkly returns a verdict this release does not recognise, `verdict` is `None`, a warning is logged, and `passed` falls back to the row counts: the run passes only when the completed summary has no failed, error, or pending rows. A pass rate threshold of `1.0` gets the all-rows gate the same way that no threshold does. An unrecognised verdict ends the wait at once, because LaunchDarkly records a run's verdict only once. The same row rule applies when LaunchDarkly reports no pass rate threshold for the run. `failed_rows` counts rows whose criteria were scored and did not meet their threshold, so a gate that ignored it would exit 0 on a run where every row failed its judge. Pass a **stable** evaluation `key` matching `^[a-z0-9][a-z0-9._-]*$` (at most 128 characters; validated before any network I/O — timestamp keys with `:` from older examples no longer work): `run()` GETs `…/offline-evaluations/{key}`, creates with POST (`key` + `name`) when missing, or always PATCHes a full config snapshot when the evaluation already exists (the server decides whether anything changed; PATCH omits `name` so a UI rename survives CI) — there is no PUT. Each call still creates a new run under the evaluation id.
Management API requests go to the `api_base_uri` option, then `LD_API_BASE_URI`, then `https://app.launchdarkly.com`; `LD_BASE_URI` is never used, because it may point at a relay proxy. Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. At either level a blank value counts as unset, and a trailing `/` is stripped. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts, plus each criterion's counts and pass-rate outcome in `criteria` where LaunchDarkly reports them. When the summary reports a criterion with a pass rate threshold (every criterion has one by default, see below), the harness then waits up to 30s for LaunchDarkly's verdict on the run and returns it as `result.verdict` (`"pass"` or `"fail"`). `result.passed` follows the verdict. The run fails when any criterion's pass rate, with errored rows counted against it, is below its threshold. Criteria without a pass rate threshold are left out of the verdict. If no verdict arrives, or LaunchDarkly returns a verdict this release does not recognise, `verdict` is `None`, a warning is logged, and `passed` falls back to the row counts: the run passes only when the completed summary has no failed, error, or pending rows. A pass rate threshold of `1.0` gets the all-rows gate the same way that no threshold does. An unrecognised verdict ends the wait at once, because LaunchDarkly records a run's verdict only once. The same row rule applies when LaunchDarkly reports no pass rate threshold for the run. `failed_rows` counts rows whose criteria were scored and did not meet their threshold, so a gate that ignored it would exit 0 on a run where every row failed its judge. Pass a **stable** evaluation `key` matching `^[a-z0-9][a-z0-9._-]*$` (at most 128 characters; a blank key is an error, and the key is validated before any network I/O — timestamp keys with `:` from older examples no longer work): `run()` GETs `…/offline-evaluations/{key}`, creates with POST (`key` + `name`) when missing, or always PATCHes a full config snapshot when the evaluation already exists (the server decides whether anything changed; PATCH omits `name` so a UI rename survives CI) — there is no PUT. Each call still creates a new run under the evaluation id.

```python
import asyncio
Expand Down Expand Up @@ -97,7 +97,7 @@ sys.exit(asyncio.run(main()))

**Pass one handler as `handler` or a list as `handlers`, not both.** Each handler must declare `provides_for`; the provider packages' `create_*_handler()` factories and `create_handler()` set it. The generation config and every judge select a handler from the list by provider and mode, with the same rule as `config()`: a handler that names the provider wins over a `"*"` handler, and the mode must match exactly. To evaluate your own application code, wrap it with `create_handler((provider, mode), fn)`.

**Start from an AI Config.** Pass `ai_config=AIConfig(key=..., variation=...)` instead of a full `generation`. The harness reads the AI Config's mode first, then the variation, then the model config it links, and runs each check as soon as the read that supplies its input returns. The mode selects the prompt field: `instructions` in agent mode, `messages` in completion and judge mode. `generation` can then override single fields, except `mode`. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.
**Start from an AI Config.** Pass `ai_config=AIConfig(key=...)` instead of a full `generation`. Omit `variation` to use the variation served for `{kind: "evaluation", key: <evaluation key>}` and the content of that `_ldMeta.version`. A missing version fails before any write. The caller passes no context, and there is no project-key fallback. Pass `variation="..."` to read that variation's latest version from the management API instead. The harness reads the AI Config's mode first, then the variation, then the model config it links, and runs each check as soon as the read that supplies its input returns. The mode selects the prompt field: `instructions` in agent mode, `messages` in completion and judge mode. `generation` can then override single fields, except `mode`. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.

Generation and criterion events are the only path by which row results reach LaunchDarkly, so `init_evaluations()` raises rather than creating a run that can never complete unless it can resolve an event transport: either an SDK key (`sdk_key` or `LD_SDK_KEY`) or a client already initialized through `init_client(client=...)`. Bringing your own client lets a process emit evaluation events without an SDK key in scope. Every generated row is emitted and flushed unconditionally; no feature flag gates event publishing. The harness then polls the summary endpoint until row accounting shows processing is complete.

Expand Down
4 changes: 2 additions & 2 deletions packages/client/api-report/launchdarkly_ai_server.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Public API of launchdarkly_ai_server. Generated by `make api-report`; do not edit.
add_cached_tokens_to_input(raw_usage: dict[str, Any]) -> SpanUsage
AIConfig [class, @dataclass(frozen=True)]
__init__(key: str, variation: str)
__init__(key: str, variation: str | None = None)
key: str
variation: str
variation: str | None = None
AiConfigRep = dict[str, Any]
all_skills() -> list[Skill] [async]
any_multimodal(turns: list[dict[str, Any]]) -> bool
Expand Down
50 changes: 34 additions & 16 deletions packages/client/src/launchdarkly_ai_server/evaluations/module.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,8 +358,12 @@ async def run(

``generation`` must set ``provider``, ``model`` and ``mode``
(``"completion"`` or ``"agent"``). Pass ``ai_config`` instead to start
from an existing AI Config variation. The AI Config supplies the mode.
The variation's model, provider, parameters, prompt and output format
from an existing AI Config. The AI Config supplies the mode. When
``variation`` is set, that variation's latest version is the base.
When it is omitted, the base is the variation served for
``{kind: "evaluation", key: <this run's key>}`` at that variation's
``_ldMeta.version``. A missing version fails before any write. The
variation's model, provider, parameters, prompt and output format
become the defaults, and ``generation`` can override them field by
field (``parameters`` merge key by key), except ``mode``. When
``tools`` is omitted the variation's tools are used, so each needs an
Expand All @@ -385,6 +389,7 @@ async def run(
run_tool_handlers = tool_handlers(run_tools)
pinned_tool_versions: dict[str, int] = {}
config_label = ""
client = await self._resolve_client()
if ai_config is None:
run_generation = self._validate_generation(
cast(dict[str, Any], dict(generation or {}))
Expand All @@ -393,14 +398,18 @@ async def run(
pool, run_generation, "generation"
)
else:
config_label = f"{ai_config.key!r}/{ai_config.variation!r}"
(
run_generation,
generation_handler,
ai_config_variation,
variation_key,
) = await self._read_ai_config(
ai_config, cast(GenerationOverrides | None, generation), pool
ai_config,
cast(GenerationOverrides | None, generation),
pool,
key,
)
config_label = f"{ai_config.key!r}/{variation_key!r}"
if tools is None and ai_config_variation.tool_versions:
raise EvaluationsError(
f"AI Config variation {config_label} uses tools "
Expand Down Expand Up @@ -431,7 +440,6 @@ async def run(
ld_judges = [
criterion for criterion in run_criteria if isinstance(criterion, Judge)
]
client = await self._resolve_client()

# The management API client is synchronous; running it in a worker thread
# keeps the caller's event loop free.
Expand Down Expand Up @@ -809,14 +817,24 @@ async def _read_ai_config(
ai_config: AIConfig,
overrides: GenerationOverrides | None,
pool: list[EvalHandler],
) -> tuple[GenerationConfig, EvalHandler, AIConfigVariation]:
evaluation_key: str,
) -> tuple[GenerationConfig, EvalHandler, AIConfigVariation, str]:
"""Read an AI Config variation and select its generation handler.

Each check runs as soon as the read that supplies its input returns:
the mode after the AI Config read, the prompt after the variation
read, and the provider after the model config read.
read, and the provider after the model config read. An omitted
``variation`` is resolved before those reads, so a missing served
version fails before any management-API write.
"""
label = f"AI Config {ai_config.key!r} variation {ai_config.variation!r}"
if ai_config.variation is None:
variation_key, version = await self._runner._served_variation(
ai_config.key, evaluation_key
)
Comment thread
cursor[bot] marked this conversation as resolved.
Resolved
else:
variation_key = ai_config.variation
version = None
label = f"AI Config {ai_config.key!r} variation {variation_key!r}"
config_mode = await asyncio.to_thread(
self._runner._fetch_ai_config_mode, self._project_key, ai_config.key
)
Expand All @@ -835,7 +853,8 @@ async def _read_ai_config(
self._runner._fetch_config_variation,
self._project_key,
ai_config.key,
ai_config.variation,
variation_key,
version=version,
)
prompt = _merge_generation(
AIConfigVariation.from_api(latest, None, mode).generation, overrides
Expand All @@ -853,7 +872,7 @@ async def _read_ai_config(
merged["mode"] = "agent" if mode == "agent" else "completion"
generation = self._validate_generation(merged)
handler = self._select_generation_handler(pool, generation, label)
return generation, handler, variation
return generation, handler, variation, variation_key

@staticmethod
def _validate_run_args(
Expand Down Expand Up @@ -944,12 +963,11 @@ def _validate_config_source(
"generation.mode cannot be set with ai_config: the AI Config "
"supplies the mode"
)
for name, value in (
("ai_config.key", ai_config.key),
("ai_config.variation", ai_config.variation),
):
if not value.strip():
raise EvaluationsError(f"{name} must not be blank")
if not ai_config.key.strip():
raise EvaluationsError("ai_config.key must not be blank")
# ``None`` means "the served variation". A blank string is still an error.
if ai_config.variation is not None and not ai_config.variation.strip():
raise EvaluationsError("ai_config.variation must not be blank")

@staticmethod
def _validate_generation(generation: Mapping[str, Any]) -> GenerationConfig:
Expand Down
83 changes: 76 additions & 7 deletions packages/client/src/launchdarkly_ai_server/evaluations/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,52 @@ def _finish_criterion_span(
span.set_attribute("error.type", code)


def _served_identity(config_key: str, extracted: Mapping[str, Any]) -> tuple[str, int]:
"""Read the served variation key and version from the raw flag payload.

``extract_variation`` copies ``_ldMeta.version`` onto ``meta`` and fills in
``1`` when the field is absent. That default must not become a run: only
the raw ``value`` still shows that the version was missing.
"""
value = extracted.get("value")
raw_meta: Any = value.get("_ldMeta") if isinstance(value, Mapping) else None
ld_meta = raw_meta if isinstance(raw_meta, Mapping) else {}
version = ld_meta.get("version")
# ``bool`` is an ``int``, so ``True`` must not become version 1.
if type(version) is not int:
raise EvaluationsError(
f"AI Config {config_key!r} served a variation with no integer "
f"_ldMeta.version (got {version!r})"
)
variation_key = ld_meta.get("variationKey")
if not isinstance(variation_key, str) or not variation_key.strip():
raise EvaluationsError(
f"AI Config {config_key!r} served a variation with no variation key"
)
return variation_key, version


def _select_variation_version(
versions: list[Mapping[str, Any]],
description: str,
version: int | None,
) -> Mapping[str, Any]:
"""Pick ``version``, or the greatest version when ``version`` is omitted."""
if version is None:
return max(versions, key=lambda item: int(item["version"]))
selected = next(
(
item
for item in versions
if type(item.get("version")) is int and item["version"] == version
),
None,
)
if selected is None:
raise EvaluationsError(f"LaunchDarkly {description} has no version {version}")
return selected


class EvaluationsRunner:
"""Private API operations and orchestration used by EvaluationsModule.run()."""

Expand Down Expand Up @@ -339,16 +385,39 @@ def _fetch_ai_config_mode(self, project_key: str, config_key: str) -> str:
)
return mode

async def _served_variation(
self, config_key: str, evaluation_key: str
) -> tuple[str, int]:
"""Return the variation key and version the SDK serves for this run.

The context is always ``{kind: "evaluation", key: evaluation_key}``.
There is no caller context and no project-key fallback. A disabled or
undelivered variation is an :class:`EvaluationsError`.
"""
context = {"kind": "evaluation", "key": evaluation_key}
try:
extracted = await extract_variation(config_key, context)
except Exception as error:
raise EvaluationsError(
f"Failed to resolve the served variation of AI Config "
f"{config_key!r} for evaluation {evaluation_key!r}: {error}"
) from error
return _served_identity(config_key, extracted)

def _fetch_config_variation(
self,
project_key: str,
config_key: str,
variation_key: str,
*,
version: int | None = None,
) -> Mapping[str, Any]:
"""Read the latest version of an AI Config variation by key.
"""Read one version of an AI Config variation by key.

Raises :class:`EvaluationsError` when the variation does not exist,
has no versions, or has a ``modelConfigKey`` that is not a string.
``version`` selects that version. Omit it to read the latest. A served
version older than the latest is valid and is not warned about. Raises
:class:`EvaluationsError` when the variation does not exist, has no
matching version, or has a ``modelConfigKey`` that is not a string.
"""
description = f"AI Config variation {config_key!r}/{variation_key!r}"
path = (
Expand All @@ -363,7 +432,7 @@ def _fetch_config_variation(
f"LaunchDarkly {description} was not found in project {project_key!r}"
) from error
raise
# The endpoint returns every version of the variation; evaluate the latest.
# The endpoint returns every version of the variation.
items = raw.get("items")
versions = [
item
Expand All @@ -372,15 +441,15 @@ def _fetch_config_variation(
]
if not versions:
raise EvaluationsError(f"LaunchDarkly {description} has no versions")
latest = max(versions, key=lambda item: int(item["version"]))
selected = _select_variation_version(versions, description, version)
# Absent or empty means that the variation links no model config.
model_config_key = latest.get("modelConfigKey")
model_config_key = selected.get("modelConfigKey")
if model_config_key is not None and not isinstance(model_config_key, str):
raise EvaluationsError(
f"LaunchDarkly {description} has a non-string modelConfigKey: "
f"{model_config_key!r}"
)
return latest
return selected

def _fetch_linked_model_config(
self, project_key: str, variation: Mapping[str, Any]
Expand Down
Loading
Loading