Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ elif result and task.evaluation_metric_key:
> **Experimental.** Import shadow requests from `launchdarkly_ai_server.experimental.shadow_requests`.
> These names may change in a minor release, so review the changelog when you upgrade.

A shadow request runs a candidate variation on real traffic without serving it. Use `shadow_config()` in place of `config()` at the call sites you want to shadow. It takes the same arguments, and `invoke()` and `stream()` return exactly what `config()` returns. When the context matches a targeting rule or the fallthrough that has shadows attached, each sampled shadow runs the same input on its own variation once the served variation succeeds. Its response is discarded. Its events are the normal `$ld:ai:*` events for that variation, with the served variation's `runId` and `isShadow: true`, so you can compare the two on the usual AI metrics. A shadow never delays or changes the served response, and its failures are only tracked.
A shadow request runs a candidate variation on real traffic without serving it. Use `shadow_config()` in place of `config()` at the call sites you want to shadow. It takes the same arguments, and `invoke()` and `stream()` return exactly what `config()` returns. When the context matches a targeting rule or the fallthrough that has shadows attached, each sampled shadow runs the same input on its own variation once the served variation succeeds. The shadow variation's judges score its response, which is then discarded. Its events are the normal `$ld:ai:*` events for that variation, with the served variation's `runId` and `isShadow: true`, so you can compare the two on the usual AI metrics. A shadow never delays or changes the served response, and its failures are only tracked.

Shadows run in the background by default, each with a 30 second limit (`shadow_timeout_s`). `shutdown()` cancels any that are still running. Pass `on_task` to run them yourself instead: it receives a picklable `ShadowTask` for each sampled shadow, and `run_shadow(task, handlers)` runs it later and never raises.

Expand All @@ -406,7 +406,7 @@ caller = shadow_config(key="my-ai-config", handler=handlers, on_task=queue.put_n
shadow = await run_shadow(await queue.get(), handlers)
```

A shadow runs as a plain `invoke()`, even when the served call streamed, with no tools and no judges. Pass handlers for every provider your shadow variations use, since a shadow with no matching handler is skipped. Plain `config()` never runs shadows.
A shadow runs as a plain `invoke()`, even when the served call streamed, with no tools. Its own judges run on its response after it succeeds, with their usual sampling rates, and each score is tracked under the shadow variation with `isShadow: true`. Each judge that fails is logged and skipped, and the others still score. Judges use the client already set up and never start one, so they are skipped after `shutdown()`. `ShadowResult.judge_results` holds the scores, and is present only when at least one judge scored. `skip_judges` applies to the served variation only. Pass handlers for every provider your shadow variations use, since a shadow with no matching handler is skipped. Plain `config()` never runs shadows.

### `graph(key, **options)`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
run_shadow(task: ShadowTask, handlers: list[ProviderHandler]) -> ShadowResult [async]
shadow_config(*, key: str, handler: ProviderHandler | list[ProviderHandler] | None = None, tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None, registry: Any = None, skip_judges: bool = False, shadow_timeout_s: float = 30.0, on_task: Callable[[ShadowTask], None] | None = None) -> ConfigInstance
ShadowResult [class, @dataclass(frozen=True)]
__init__(usage: dict[str, int], track_data: dict[str, Any], error: str | None = None)
__init__(usage: dict[str, int], track_data: dict[str, Any], error: str | None = None, judge_results: dict[str, JudgeResult] | None = None)
error: str | None = None
judge_results: dict[str, JudgeResult] | None = None
track_data: dict[str, Any]
usage: dict[str, int]
ShadowTask [class, @dataclass(frozen=True)]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
LaunchDarkly attaches to the matched targeting rule or the fallthrough. The served
variation runs exactly as ``config()`` runs it. Once it succeeds, each sampled shadow
runs on the same input in the background, or is handed to ``on_task`` to run later with
``run_shadow()``. A shadow's response is discarded; it is tracked on the normal
``$ld:ai:*`` events with ``isShadow``. These names may change in a minor release, so
``run_shadow()``. The shadow variation's judges score its response, which is then
discarded; the shadow and its judge scores are tracked on the normal events with
``isShadow``. These names may change in a minor release, so
review the changelog when you upgrade.
"""

Expand All @@ -25,15 +26,27 @@

from opentelemetry import context as otel_context

from ..client import ConfigInstance, _InternalConfigInstance
from ..client import (
ConfigInstance,
_InternalConfigInstance,
_resolve_output_format_response,
)
from ..conversation import _CONV_KEY, _conversation_id_from
from ..lifecycle import _fetch_variation, _parse_variation
from ..judge_scoring import judge_text
from ..judges import _run_judges as _run_core_judges
from ..lifecycle import (
_existing_client,
_extract_existing_variation,
_fetch_variation,
_parse_variation,
)
from ..registry import resolve_handlers
from ..sdk_usage import report_usage
from ..shutdown_hooks import register_shutdown_hook
from ..tracking import execute_and_track, make_track_data
from ..types import (
AiConfigRep,
JudgeResult,
LDContext,
NativeTool,
ProviderHandler,
Expand All @@ -52,15 +65,16 @@

_FALLTHROUGH = "fallthrough"
_PENDING_SHADOWS = "__ld_pending_shadows"
_SHADOW_ONLY_KEYS = ("tools", "judgeConfiguration", "shadowConfiguration")
_SHADOW_ONLY_KEYS = ("tools", "shadowConfiguration")
_BUCKET_SCALE = float(0xFFFFFFFFFFFFFFF)


@dataclass(frozen=True)
class ShadowTask:
"""
Everything needed to run and track one sampled shadow, with no flag evaluation.
Plain data, so it survives ``pickle``. Run it with ``run_shadow(task, handlers)``.
Everything needed to run and track one sampled shadow. Running it evaluates no
flag except the shadow's judges. Plain data, so it survives ``pickle``. Run it
with ``run_shadow(task, handlers)``.
"""

config_key: str
Expand All @@ -79,11 +93,16 @@ class ShadowTask:

@dataclass(frozen=True)
class ShadowResult:
"""The outcome of one shadow run. There is no response text: it is discarded."""
"""
The outcome of one shadow run. There is no response text: it is discarded.
``judge_results`` holds the shadow variation's judge scores, present only when
at least one judge scored.
"""

usage: dict[str, int]
track_data: dict[str, Any]
error: str | None = None
judge_results: dict[str, JudgeResult] | None = None


_running: set[asyncio.Task[ShadowResult]] = set()
Expand Down Expand Up @@ -430,11 +449,48 @@ async def _run(task: ShadowTask, handlers: list[ProviderHandler]) -> ShadowResul
span_attributes=task.span_attributes,
feature_flag_event=False,
)
judge_results = await _run_judges(task, handler, handlers, result)
except _ShadowTimeout:
return _failed(task, "timeout")
except Exception as err:
return _failed(task, str(err) or type(err).__name__)
return ShadowResult(usage=result["usage"], track_data=result["track_data"])
return ShadowResult(
usage=result["usage"],
track_data=result["track_data"],
judge_results=judge_results or None,
)


async def _run_judges(
task: ShadowTask,
handler: ProviderHandler,
handlers: list[ProviderHandler],
result: dict[str, Any],
) -> dict[str, JudgeResult]:
"""Scores the shadow's response with its own judges, outside the shadow timeout.

Judges resolve through the client already set up and never start one, so
with no live client they are skipped."""
if _existing_client() is None:
return {}
try:
response = _resolve_output_format_response(
result["response"], task.shadow_config.get("outputFormat")
)
return await _run_core_judges(
config=task.shadow_config,
user_context=task.user_context,
handler=handler,
handlers=handlers,
user_input=task.user_input,
trajectory=result.get("trajectory", ""),
llm_response=judge_text(response),
base_track_data=result["track_data"],
resolve=_extract_existing_variation,
)
except Exception:
logger.warning("Shadow request judges failed", exc_info=True)
return {}


@contextmanager
Expand Down
36 changes: 34 additions & 2 deletions packages/client/src/launchdarkly_ai_server/judges.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import logging
import random
from collections.abc import Callable, Iterator, Mapping
from collections.abc import Awaitable, Callable, Iterator, Mapping
from contextlib import contextmanager
from typing import Any

Expand Down Expand Up @@ -113,6 +113,38 @@ async def run_judges(
unchanged, and so is a judge for a config with no tools.
"""
from .lifecycle import extract_variation

return await _run_judges(
config=config,
user_context=user_context,
handler=handler,
handlers=handlers,
user_input=user_input,
llm_response=llm_response,
base_track_data=base_track_data,
tool_handlers=tool_handlers,
graph_key=graph_key,
trajectory=trajectory,
resolve=extract_variation,
)


async def _run_judges(
*,
config: AiConfigRep,
user_context: LDContext,
handler: ProviderHandler,
handlers: list[ProviderHandler] | None,
user_input: str | None,
llm_response: str,
base_track_data: TrackData,
tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None,
graph_key: str | None = None,
trajectory: str = "",
resolve: Callable[[str, LDContext], Awaitable[dict[str, Any]]],
) -> dict[str, JudgeResult]:
""":func:`run_judges`, resolving each judge's variation with ``resolve``.
A judge whose resolution raises is logged and skipped."""
from .tracking import execute_and_track

judge_results: dict[str, JudgeResult] = {}
Expand All @@ -134,7 +166,7 @@ async def run_judges(
judge_key = judge["key"]

try:
variation = await extract_variation(judge_key, user_context)
variation = await resolve(judge_key, user_context)
judge_ai_config: AiConfigRep = variation["config"]
judge_meta = variation["meta"]

Expand Down
23 changes: 21 additions & 2 deletions packages/client/src/launchdarkly_ai_server/lifecycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,11 @@ def note_ai_invocation() -> None:
_ai_invoked = True


def _existing_client() -> LDClientInterface | None:
"""Returns the client already set up, or ``None``. Never starts one."""
return _client


def get_client() -> LDClientInterface:
"""
Returns the singleton LaunchDarkly client.
Expand Down Expand Up @@ -764,14 +769,28 @@ async def extract_variation(
return _parse_variation(config_key, raw)


async def _extract_existing_variation(
config_key: str, context: dict[str, Any]
) -> dict[str, Any]:
"""Like :func:`extract_variation`, but uses the client already set up and
never starts one, raising when there is none."""
raw, _ = await _fetch_variation(config_key, context, existing_client_only=True)
return _parse_variation(config_key, raw)


async def _fetch_variation(
config_key: str, context: dict[str, Any], *, with_reason: bool = False
config_key: str,
context: dict[str, Any],
*,
with_reason: bool = False,
existing_client_only: bool = False,
) -> tuple[Any, Any]:
"""The raw variation value, and its evaluation reason when ``with_reason`` is set
and the client has ``variation_detail``. Otherwise the reason is ``None``."""
from .utils import to_ld_context # late import avoids circular dependency

await init_client()
if not existing_client_only:
await init_client()
client = get_client()
# The real ldclient.variation() expects an ldclient.Context, not a plain dict.
# Convert when running against the real SDK; test mocks accept dicts directly.
Expand Down
Loading
Loading