Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agentenv/config.example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,10 @@ agent_default = "local" # agents too
# default_a2a_agent_id = "my-agent" # what a deploy_agent step that names no agent deploys; by default the
# # agents bundle's Claude Code, @local/agentenv-framework/agents/claude-code

# [trajectories]
# formats = ["atif/1"] # the formats beside native a run prescribes to each agent whose trajectory it reads;
# # none by default, so agents build native only. Per run: user_overrides.trajectory_formats

# [envs]
# default_gateway_env_id = "default" # the gateway every gateway deploy runs
# default_service_db_env_id = "default-db" # the service-db local Postgres state runs from
Expand Down
11 changes: 10 additions & 1 deletion AGENTS.md

Large diffs are not rendered by default.

36 changes: 30 additions & 6 deletions packages/agentenv-protocol/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -612,6 +612,7 @@ from agentenv_protocol.computer import (
class MyDriver(ComputerDriverEnvironment):
buttons = ("left", "right") # features, on the card
max_click_count = 2
screenshot_delay_ms = 300 # how long after an input its screenshot is taken

@display
async def size(self) -> Display: ... # width, height, scale
Expand Down Expand Up @@ -668,7 +669,11 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
`computer_scroll`, `computer_type`, `computer_key` (W3C key names) and `computer_wait`, plus the
group tools. Every result is structured, with a one-line summary as its text block; input results
carry `applied`, `delivery` and `others_since_frame`, and `screenshot: true` returns a fresh frame
after acting. Click and scroll leave out `x` and `y` to act at the pointer on a computer with
taken `screenshot_delay_ms` after the input, so a smooth scroll or the page a key opens, which renders a couple
of hundred milliseconds after the input returns, shows: the driver class's `screenshot_delay_ms` (500 unless it
sets its own), or the deploy's setting of that name, 0 capturing at once. A driver's own tools don't wait: they keep
their own timing. Click and scroll leave out `x` and `y`
to act at the pointer on a computer with
`computer-pointer/v1`; elsewhere a call without them is refused `invalid_argument`. With the page
group, `computer_navigate` opens a URL in the active tab, `computer_history` goes `back`, `forward`
or `reload`s, `computer_tabs` lists, opens, switches and closes tabs by the browser's own ids and
Expand Down Expand Up @@ -719,8 +724,9 @@ other exception from an input operation is reported `delivery_failed` with `appl

**Settings** reach the driver as one JSON value in `AGENTENV_COMPUTER_SETTINGS`
(`ComputerSettings`): `default_space`, `default_norm_scale`, `queue_wait_s`,
`tap_hold_ms`, `type_delay_ms`, `max_hold_ms`, `layout`, `drag_duration_ms`, `exec_output_bytes`
and `exec_timeout_s`. A driver reads them as `self.settings`, for example its tap hold in `@press`.
`tap_hold_ms`, `type_delay_ms`, `max_hold_ms`, `layout`, `drag_duration_ms`, `exec_output_bytes`,
`exec_timeout_s` and `screenshot_delay_ms`, which replaces the driver's own. A driver reads them as `self.settings`,
for example its tap hold in `@press`.

Every caller is `default` until agents carry an identity of their own; `caller_identity(context)`
names the caller whose latest frame and others' input a call uses.
Expand Down Expand Up @@ -1623,9 +1629,27 @@ responses, the format it serves.
enable(TRAJECTORY_V1, live=True, native="my-cli-events/1", atif=MyCliAtif)
```

The card then lists `atif/1` in `formats` too; `default` stays `native`. The
SDK keeps one ATIF document per A2A context, run through one adapter instance,
and feeds it every event the agent appends, as it appends it. A native
The card then lists `atif/1` in `formats` too; `default` stays `native`.

The agent builds `atif/1` only when prescribed: a run that reads its copies
says so first with the trajectory extension's `prescribe`
(`POST /ext/trajectory/prescribe`, `{formats: ["atif/1"]}`, each a format the
card offers; it answers every format the agent now builds, native first).

- A prescription holds for each A2A context started after it. A context keeps the
formats it started with, so no copy misses its first turns, and a later call
sets them for the contexts after it.
- It's one setting for the whole agent. An agent that serves several runs builds,
for each new context, what the last call prescribed, whichever run made it.
- With no call the agent builds native only: no ATIF document, no media spool,
and no turn count of its own. A snapshot restored into it still records the
count the snapshot carries, so a context prescribed `atif/1` later numbers its
turns on from there.
- An `atif/1` read of a context that wasn't prescribed it is refused with `409
format_not_prescribed`. A read that names no format always gets native.

For a context prescribed `atif/1`, the SDK keeps one ATIF document, run through
one adapter instance, and feeds it every event the agent appends, as it appends it. A native
trajectory returned at task end runs through the adapter then. The agent's own
code feeds the adapter's aux files, records that aren't in the native log such
as a hook's notes, with `request.trajectory.aux(name, record)`, where they
Expand Down
2 changes: 1 addition & 1 deletion packages/agentenv-protocol/pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "agentenv-framework-protocol"
version = "0.1.618"
version = "0.1.624"
description = "Open data-plane protocol and server SDK for agent environments."
readme = "README.md"
license = "Apache-2.0"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@
TrajectoryEventsResponse,
TrajectoryMediaRequest,
TrajectoryMediaResponse,
TrajectoryPrescribeRequest,
TrajectoryPrescribeResponse,
TrajectoryObjectsResponse,
AtifEventsResponse,
AtifObjectsResponse,
Expand Down Expand Up @@ -260,6 +262,8 @@
"TrajectoryLog",
"TrajectoryMediaRequest",
"TrajectoryMediaResponse",
"TrajectoryPrescribeRequest",
"TrajectoryPrescribeResponse",
"TrajectoryObjectsResponse",
"TrajectoryState",
"TrajectoryUploadedObjects",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
import logging
from collections import OrderedDict
from dataclasses import dataclass
from typing import Any, Callable, Sequence
from typing import Any, Callable, Iterable, Sequence

from ..atif import FORMAT as ATIF_FORMAT
from ..atif import Adapter
from ._atif_copy import AtifCopy, AtifPage, TurnStart
from ._media_spool import MediaSpool
Expand All @@ -21,6 +22,15 @@
MAX_TRAJECTORY_READ_BYTES = 4 * 1024 * 1024


class NotPrescribed(LookupError):
"""A read of a format the conversation it names was not prescribed when it started."""

def __init__(self, context_id: str, format: str) -> None:
super().__init__(f"context {context_id!r} was not prescribed {format}, so it has no {format} copy")
self.context_id = context_id
self.format = format


@dataclass(slots=True)
class _TaskRecord:
context_id: str
Expand Down Expand Up @@ -104,6 +114,16 @@ def __init__(
self._tasks: dict[str, _TaskRecord] = {}
self._contexts: OrderedDict[str, list[str]] = OrderedDict()
self._copies: dict[str, AtifCopy] = {}
#: The formats beside native a conversation started from now on is built in: none until prescribed.
self._prescribed: frozenset[str] = frozenset()
#: Each started conversation's formats beside native, fixed as its first task starts.
self._built: dict[str, frozenset[str]] = {}

def prescribe(self, formats: Iterable[str]) -> frozenset[str]:
"""Build ``formats`` beside native for each conversation started from now on; one already started keeps the
formats it started with, so no copy misses its first turns."""
self._prescribed = frozenset(formats) - {self._native}
return self._prescribed

def register(self, task_id: str, context_id: str) -> TrajectoryLog:
"""The task's log, created ``pending`` the first time the task is seen."""
Expand Down Expand Up @@ -132,7 +152,8 @@ def start(self, task_id: str, turn: TurnStart | None = None) -> None:
started = sum(1 for other in order if self._tasks[other].started)
order.insert(started, task_id)
record.started = True
if self._atif is not None and turn is not None:
built = self._built.setdefault(record.context_id, self._prescribed)
if self._atif is not None and turn is not None and ATIF_FORMAT in built:
copy = self._copies.get(record.context_id)
if copy is None:
copy = self._copies[record.context_id] = AtifCopy(
Expand All @@ -153,6 +174,7 @@ def restored(self, context_id: str, turns: int) -> None:
new document, since the conversation is the snapshot's now; an agent that serves atif/1 only."""
self._turns.set(context_id, turns)
self._copies.pop(context_id, None)
self._built.pop(context_id, None)
self.media.drop(context_id)

def request_cancel(self, task_id: str) -> None:
Expand Down Expand Up @@ -281,7 +303,7 @@ def atif_task(self, task_id: str) -> AtifTaskRead | None:
"""The ended task's atif/1 document as it stood when it ended, with its aux log and served
turn context, or None when there is none."""
record = self._tasks.get(task_id)
copy = self._copies.get(record.context_id) if record is not None else None
copy = self._copy(record.context_id) if record is not None else None
if copy is None:
return None
self._touch(record.context_id)
Expand All @@ -290,7 +312,7 @@ def atif_task(self, task_id: str) -> AtifTaskRead | None:

def atif_context(self, context_id: str) -> dict[str, Any] | None:
"""The context's atif/1 live page, or None when it has no closed step yet."""
copy = self._copies.get(context_id)
copy = self._copy(context_id)
if copy is None:
return None
self._touch(context_id)
Expand All @@ -301,7 +323,7 @@ def atif_task_page(
) -> AtifCursor | None:
"""A cursor page of the task's closed atif/1 steps, in the task's state."""
record = self._tasks.get(task_id)
copy = self._copies.get(record.context_id) if record is not None else None
copy = self._copy(record.context_id) if record is not None else None
if copy is None:
return None
self._touch(record.context_id)
Expand All @@ -314,14 +336,21 @@ def atif_context_page(
) -> AtifCursor | None:
"""A cursor page of the context's closed atif/1 steps, on the task the context is on and in
the context's state."""
copy = self._copies.get(context_id)
copy = self._copy(context_id)
records = self._context_records(context_id) if copy is not None else None
if copy is None or records is None:
return None
task_id, state = self._context_progress(context_id, records)
page = copy.page(after, max_steps, max_bytes)
return AtifCursor(page=page, context_id=context_id, task_id=task_id, state=state)

def _copy(self, context_id: str) -> AtifCopy | None:
"""The context's atif/1 copy, None before it has one; NotPrescribed when it started without atif/1."""
built = self._built.get(context_id)
if built is not None and ATIF_FORMAT not in built:
raise NotPrescribed(context_id, ATIF_FORMAT)
return self._copies.get(context_id)

def _context_progress(
self, context_id: str, records: list[_TaskRecord]
) -> tuple[str, TrajectoryState]:
Expand Down Expand Up @@ -362,6 +391,7 @@ def _evict(self, *, keep: str) -> None:
continue
del self._contexts[context_id]
self._copies.pop(context_id, None)
self._built.pop(context_id, None)
self.media.drop(context_id)
for task_id in task_ids:
del self._tasks[task_id]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,19 @@ class TrajectoryMediaResponse(ExtensionResponse):
failed: dict[str, str] = Field(default_factory=dict)


class TrajectoryPrescribeRequest(ExtensionRequest):
"""The trajectory formats beside native the agent builds for each conversation it starts from now on, each one
its card's ``formats`` lists; none for native only, which it always builds."""

formats: list[str] = Field(default_factory=list)


class TrajectoryPrescribeResponse(ExtensionResponse):
"""Every format the agent builds for a conversation it starts from now on: native first."""

formats: list[str]


class AtifUploadedObjects(ExtensionResponse):
trajectory: Uploaded
aux: Uploaded | None = None
Expand Down Expand Up @@ -1227,8 +1240,8 @@ def _configure_trajectory(
return ExtensionConfiguration(
wire_params={"formats": formats, "native": native, "default": native} if native else {},
options={"native": native, **({"atif": atif} if atif is not None else {})} if native else {},
# An agent that serves atif/1 hands over its copies' media files, which the worker pulls at each turn's end.
features={"media"} if atif is not None else frozenset(),
# An agent that can build atif/1 builds it only when prescribed, and hands over its copies' media files.
features={"media", "prescribe"} if atif is not None else frozenset(),
variants=(
{"get": {"task_events", "context_events", "context", "context_objects"}} if live else {}
),
Expand Down Expand Up @@ -1389,7 +1402,20 @@ def _configure_trajectory(
sdk_only=True,
),
}
)
),
"prescribe": OperationGroup(
operations={
"prescribe": _op(
"prescribe",
"POST",
"/ext/trajectory/prescribe",
SDK,
request_model=TrajectoryPrescribeRequest,
response_model=TrajectoryPrescribeResponse,
sdk_only=True,
),
}
),
},
configuration_validator=_configure_trajectory,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
from ._atif_copy import TurnStart
from ._secrets import Secrets
from ._snapshot_state import SnapshotStates
from ._trajectory_log import MAX_TRAJECTORY_READ_BYTES, TaskTrajectories
from ._trajectory_log import MAX_TRAJECTORY_READ_BYTES, NotPrescribed, TaskTrajectories
from ._triggers import TriggerEngine, TriggerError
from .extensions import (
AGENT_CONFIG_V1,
Expand Down Expand Up @@ -69,6 +69,8 @@
AtifUploadedObjects,
TrajectoryEventsResponse,
TrajectoryMediaRequest,
TrajectoryPrescribeRequest,
TrajectoryPrescribeResponse,
TrajectoryObjectsResponse,
TrajectoryState,
TrajectoryUploadedObjects,
Expand Down Expand Up @@ -147,6 +149,16 @@ def _config_wire_names(config: type[AgentConfig]) -> dict[str, str]:
return result



def _not_prescribed(exc: NotPrescribed) -> AgentEnvError:
return AgentEnvError(
"format_not_prescribed",
f"{exc.format} was not prescribed for context {exc.context_id!r} when it started, so the agent built none; "
"prescribe it before a conversation's first message (the trajectory extension's prescribe)",
status=409,
)


def _config_write_only_wire_names(config: type[AgentConfig]) -> set[str]:
"""Return wire names explicitly marked as write-only in the config schema."""
wire_names = _config_wire_names(config)
Expand Down Expand Up @@ -684,6 +696,7 @@ def handlers(self) -> dict[tuple[str, str], Callable]:
(SKILL_CONFIG_V1.uri, "list"): self.skill_list,
(TRAJECTORY_V1.uri, "get"): self.trajectory_get,
(TRAJECTORY_V1.uri, "media"): self.trajectory_media,
(TRAJECTORY_V1.uri, "prescribe"): self.trajectory_prescribe,
(TRIGGERS_V1.uri, "register"): self.triggers_register,
(TRIGGERS_V1.uri, "decide"): self.triggers_decide,
(TRIGGERS_V1.uri, "state"): self.triggers_state,
Expand Down Expand Up @@ -905,6 +918,8 @@ def _atif_events(
request.context_id, request.after, max_steps, MAX_TRAJECTORY_READ_BYTES
)
unknown = f"Unknown context '{request.context_id}'"
except NotPrescribed as exc:
raise _not_prescribed(exc) from exc
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
if cursor is None:
Expand Down Expand Up @@ -934,7 +949,10 @@ async def _atif_document(
through the request's grants, as the copy serves it, its secrets scrubbed."""
served: dict[str, Any] = {}
if isinstance(request, (TaskTrajectoryRequest, TaskObjectTrajectoryRequest)):
read = self.task_trajectories.atif_task(request.task_id)
try:
read = self.task_trajectories.atif_task(request.task_id)
except NotPrescribed as exc:
raise _not_prescribed(exc) from exc
if read is None:
raise HTTPException(
status_code=404,
Expand All @@ -945,7 +963,10 @@ async def _atif_document(
else:
if isinstance(request, ContextObjectTrajectoryRequest):
_refuse_turn_objects(request)
document = self.task_trajectories.atif_context(request.context_id)
try:
document = self.task_trajectories.atif_context(request.context_id)
except NotPrescribed as exc:
raise _not_prescribed(exc) from exc
if document is None:
raise HTTPException(
status_code=404,
Expand All @@ -966,6 +987,17 @@ async def _atif_document(
)
return AtifObjectsResponse(objects=uploaded, **versions).model_dump(mode="json", exclude_none=True)

async def trajectory_prescribe(self, request: TrajectoryPrescribeRequest) -> dict[str, Any]:
"""Build ``request.formats`` beside native for each conversation the agent starts from now on; each must be
one the card offers."""
for format in request.formats:
self.check_trajectory_format(format)
built = self.task_trajectories.prescribe(request.formats)
native = self.trajectory_formats[0]
return TrajectoryPrescribeResponse(
formats=[native, *(format for format in self.trajectory_formats if format in built)]
).model_dump(mode="json")

async def trajectory_media(self, request: TrajectoryMediaRequest) -> dict[str, Any]:
"""Upload the media files of a context's atif/1 copy the worker asks for, from the spool (``MediaSpool.push``)."""
pushed = await self.task_trajectories.media.push(request.context_id, request.objects)
Expand Down
25 changes: 22 additions & 3 deletions packages/agentenv-protocol/src/agentenv_protocol/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,28 @@ def request_headers(url: str, headers: dict[str, str] | None = None) -> dict[str

def granted_headers(url: str, grants: dict[str | None, dict[str, str]]) -> dict[str, str]:
"""What ``grants``, env addresses and the headers that carry their control tokens, give a call to ``url``: those of
the address it is or is under, and none anywhere else."""
return next((sent for base, sent in grants.items()
if base and (url == base.rstrip("/") or url.startswith(f"{base.rstrip('/')}/"))), {})
the address it resolves to or under, and none anywhere else."""
where = _resolved(url)
if where is None:
return {}
for base, sent in grants.items():
address = _resolved(base) if base else None
if address is not None and where[0] == address[0] and where[1][:len(address[1])] == address[1]:
return sent
return {}


def _resolved(url: str) -> tuple[tuple[str, str, int | None], list[str]] | None:
"""``url``'s origin and path segments as its server routes them, dot segments resolved; None for a relative url or
one whose path still holds a dot segment once percent-decoded, which a server may resolve out of its address."""
try:
parsed = httpx.URL(url)
except httpx.InvalidURL:
return None
segments = parsed.path.rstrip("/").split("/")
if not parsed.is_absolute_url or any(segment in (".", "..") for segment in segments):
return None
return (parsed.scheme, parsed.host, parsed.port), segments


async def _rpc(base_url: str, method: str, params: dict, timeout: int, verify: bool,
Expand Down
Loading
Loading