Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,13 +153,15 @@ share.
Shared:
- `_openenv.py`: the session on a server `openenv/` makes that each helper plays an episode through (`Session`: a
background start, tool calls, the turns left open ended until scored), with the Task API reads, the per-toolset probe,
`tasks_and_tools` (what the row writers read: the tasks, the split names checked, each toolset's tools) and an
in-process `Server`.
`tasks_and_tools` (what the row writers read: the tasks, the split names checked, each toolset's tools, and for a
toolset whose tasks apply server config the probed task's first-turn tools, which rows carry as `first_turn_tools`
and an episode's first turn must then offer exactly) and an in-process `Server`.
- `_tools.py`: turns an env's tools into typed, documented Python methods from their JSON Schema, as transformers'
`get_json_schema` reads them.
- `_chat.py`, the rows and episode the Miles and slime helpers share:
- `write_rows` and its `command` write a server's tasks as `<split>.jsonl` rows of chat `messages` and `metadata`
(the episode's split, index, task id and tools, and the server it plays on with its deadlines and turn cap).
(the episode's split, index, task id and tools, `first_turn_tools` when its task applies server config, and the
server it plays on with its deadlines and turn cap).
- `play` plays a row's episode, the model chatting at a trainer's OpenAI chat endpoint that records its turns, with
the row's tools. It answers a call whose arguments aren't a JSON object as the model's error, and returns the reward,
or flags an episode the env or the session failed, whose model missed the row's `model_timeout` or that ended
Expand All @@ -180,14 +182,14 @@ Per trainer:
- `python -m agent_env.training.nemo_gym SOURCE ROOT` writes a bundle's NeMo Gym resources server into a Gym root,
`NEMO_GYM_EXTRA_ROOTS`'s or a Gym checkout's: `app.py`; a `pyproject.toml`, since uv applies the nearest one's
`[tool.uv]` and a checkout's `exclude-dependencies` would drop `cryptography`; a config with `simple_agent` and the
`legacy_agent` environment server; and `data/<split>.jsonl` rows with each task's tools.
`legacy_agent` environment server; and `data/<split>.jsonl` rows with each task's tools (and `first_turn_tools` when it applies server config).
- `server.AgentEnvResourcesServer` is that server, run in the venv Gym builds: a row's seed starts its episode, a
catch-all route calls its tools, and `verify` ends the turns left open with the model's last reply and returns the
reward, masking a rollout the environment failed. With `reward_components` it also returns each score by step id,
their sum the reward, for NeMo RL's GDPO.
- `verl/` (`verl` extra, installed into verl's own environment):
- `python -m agent_env.training.verl URL OUT` writes a server's tasks as verl's `<split>.parquet` rows (the task's
prompt, `agent_name: agentenv`, the row's tools in `extra_info`) and `agent_loop.yaml`. It reads a server that asks
prompt, `agent_name: agentenv`, the row's tools and `first_turn_tools`, null without server config, in `extra_info`) and `agent_loop.yaml`. It reads a server that asks
for a token with the one `--token-var` names, and with `--tokenizer` counts each prompt with its tools as verl
renders it (`--chat-template-kwargs`), refusing rows over `--max-prompt-length` before writing.
- `loop.AgentEnvAgentLoop` is verl's `ToolAgentLoop` with the row's tools: each call a call on the row's episode, the
Expand All @@ -201,7 +203,8 @@ Per trainer:
`max_tool_response_length`.
- `skyrl/` (`skyrl` extra, installed into SkyRL's own environment):
- `python -m agent_env.training.skyrl URL OUT --tokenizer MODEL` writes a server's tasks as SkyRL's `<split>.parquet`
rows (`env_class: agentenv`, the episode's split, index, task id and tools, the server's settings). Each row has its
rows (`env_class: agentenv`, the episode's split, index, task id, tools and `first_turn_tools` (null without
server config), the server's settings). Each row has its
tools rendered into its system prompt by the model's chat template, checked to render as `tools=` would, since
SkyRL's generator passes the template no tools; rows over `--max-prompt-length`, counted as SkyRL's filter and
generator count them, are refused.
Expand Down
20 changes: 16 additions & 4 deletions src/agent_env/training/_chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,17 @@
import click
import httpx

from agent_env.training._openenv import Session, Shared, function_tool, key, messages, tasks_and_tools, token_in
from agent_env.training._openenv import (
FIRST_TURN_TOOLS,
Session,
Shared,
exactly,
function_tool,
key,
messages,
tasks_and_tools,
token_in,
)


@click.command()
Expand Down Expand Up @@ -50,8 +60,8 @@ def write_rows(url: str, out: str | Path, *, tool_timeout: float = 60.0, reset_t
rollout may take, and the environment variable holding the token a server given one asks for, read in each of the
trainer's rollout processes."""
url = url.rstrip("/")
_, tasks, probed = tasks_and_tools(url, local="serve the bundle with agent_env.openenv.build_app first",
timeout=reset_timeout, token=token_in(token_var))
_, tasks, probed, firsts = tasks_and_tools(url, local="serve the bundle with agent_env.openenv.build_app first",
timeout=reset_timeout, token=token_in(token_var))
tools = {each: [function_tool(tool) for tool in found] for each, found in probed.items()}
server = {"url": url, "tool_timeout": tool_timeout, "reset_timeout": reset_timeout, "model_timeout": model_timeout,
"max_turns": max_turns, **({"token_var": token_var} if token_var else {})}
Expand All @@ -61,6 +71,7 @@ def write_rows(url: str, out: str | Path, *, tool_timeout: float = 60.0, reset_t
(out / f"{split}.jsonl").write_text("".join(json.dumps({
"messages": messages(task["system_prompt"], task["prompt"]),
"metadata": {"split": split, "index": task["index"], "task_id": task["id"], "tools": tools[key(task)],
**({FIRST_TURN_TOOLS: firsts[key(task)]} if key(task) in firsts else {}),
"agentenv": server},
}) + "\n" for task in described))
return out
Expand Down Expand Up @@ -89,7 +100,8 @@ async def play(base_url: str, prompt: list[dict[str, Any]], request_kwargs: dict
server, tools = row["agentenv"], row["tools"]
episode = _Episode(Shared(server["url"], frozenset(tool["function"]["name"] for tool in tools),
server["tool_timeout"], server["reset_timeout"], patience=server["reset_timeout"],
waits_for_server=True, token=token_in(server.get("token_var"))))
waits_for_server=True, token=token_in(server.get("token_var")),
exactly=exactly(row.get(FIRST_TURN_TOOLS))))
counts = {"agentenv_tool_calls": 0, "agentenv_invalid_calls": 0}
try:
await episode._start(row["split"], int(row["index"]), row["task_id"])
Expand Down
40 changes: 31 additions & 9 deletions src/agent_env/training/_openenv.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
FAILED = "The environment failed to answer, so this episode won't be scored."
ENDED = "The episode has ended."
END_TURN = END_TURN_TOOL.name
FIRST_TURN_TOOLS = "first_turn_tools"
CONNECT_SECONDS = 60.0
CLOSE_SECONDS = 300.0
#: How much longer than a call's deadline the client waits for the server to end the call, before dropping the session.
Expand Down Expand Up @@ -64,8 +65,9 @@ class NamedToolset:
class Shared:
"""What a session needs of its server: where it is, the tools the model is shown, which an episode's first turn
must offer no more than, the deadlines, how long to wait for room on the server, whether to wait for a server that
refuses connections, as one restarting does, as long as a reset may take, and the token it asks for. The key makes
it equal to its pickled copies only."""
refuses connections, as one restarting does, as long as a reset may take, the token it asks for, and for a toolset
whose tasks apply server config, the tools its probed task offered at its first turn, which an episode's first turn
must offer exactly. The key makes it equal to its pickled copies only."""

url: str
offered: frozenset[str]
Expand All @@ -74,6 +76,7 @@ class Shared:
patience: float = REOPEN_PATIENCE
waits_for_server: bool = False
token: str | None = field(default=None, repr=False)
exactly: frozenset[str] | None = None
key: str = field(default_factory=lambda: uuid.uuid4().hex)


Expand All @@ -90,8 +93,8 @@ def __init__(self, url: str, limit: str):


class Misgrouped(Failed):
"""An episode offering a tool the model isn't shown: a toolset that misses what tells tasks apart, which no rollout
fixes."""
"""An episode offering a tool the model isn't shown, or with server config offering other first-turn tools than its
toolset's probed task: a toolset that misses what tells tasks apart, which no rollout fixes."""


@dataclass
Expand Down Expand Up @@ -156,6 +159,10 @@ async def _begin(self, rollout: Rollout, split: str, index: int, expected: str |
if not (offered := frozenset(tool["name"] for tool in listed.observation["tools"])) <= self._shared.offered:
raise Misgrouped(f"task {played!r} offers the tools {sorted(offered)}, not the "
f"{sorted(self._shared.offered)} {self.offered_by}")
if (exactly := self._shared.exactly) is not None and offered != exactly:
raise Misgrouped(f"task {played!r} offers the tools {sorted(offered)} at its first turn, not the "
f"{sorted(exactly)} its toolset's first task does there, so serve them in separate "
"bundles")

async def _step(self, name: str, arguments: dict[str, Any]) -> str:
async with self._lock:
Expand Down Expand Up @@ -391,16 +398,31 @@ def read_tasks(url: str, *, local: str, token: str | None = None) -> tuple[str,


def tasks_and_tools(url: str, *, local: str, timeout: float, token: str | None = None
) -> tuple[str, dict[str, list[dict[str, Any]]], dict[str, tuple[dict[str, Any], ...]]]:
) -> tuple[str, dict[str, list[dict[str, Any]]], dict[str, tuple[dict[str, Any], ...]],
dict[str, list[str]]]:
"""What a writer of a split's rows reads: the server's env name, each split's tasks, refusing a split whose name
can't name a data file, and every tool each toolset's turns offer, by the toolset's key, found by playing its
first task through (``probe``), so reading deploys those."""
can't name a data file, every tool each toolset's turns offer, by the toolset's key, found by playing its first
task through (``probe``), so reading deploys those, and for each toolset whose tasks apply server config, the
tools that task offered at its first turn, which a row carries as ``FIRST_TURN_TOOLS``."""
name, tasks = read_tasks(url, local=local, token=token)
if unfit := [split for split in tasks if not _FILE.fullmatch(split)]:
raise ValueError(f"the server's splits {unfit} can't name a data file: a split's name must be letters, "
"digits, '.', '_' and '-', not starting with '.'")
return name, tasks, {each: asyncio.run(probe(url, toolset, timeout, token=token))[1]
for each, toolset in toolsets(name, tasks).items()}
probed = {each: asyncio.run(probe(url, toolset, timeout, token=token))
for each, toolset in toolsets(name, tasks).items()}
return (name, tasks, {each: tools for each, (_, tools) in probed.items()},
{each: sorted(first) for each, (first, _) in probed.items() if applies_config(each)})


def applies_config(each: str) -> bool:
toolset: Toolset = json.loads(each)
return toolset.get("config", False)


def exactly(names: list[str] | str | None) -> frozenset[str] | None:
if isinstance(names, str):
names = json.loads(names)
return None if names is None else frozenset(names)


def toolsets(name: str, tasks: dict[str, list[dict[str, Any]]]) -> dict[str, NamedToolset]:
Expand Down
7 changes: 4 additions & 3 deletions src/agent_env/training/nemo_gym/prepare.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import yaml

from agent_env.openenv import build_app
from agent_env.training._openenv import Server, key, messages, tasks_and_tools
from agent_env.training._openenv import FIRST_TURN_TOOLS, Server, key, messages, tasks_and_tools

#: The routes a resources server serves itself, which a tool's route can't take.
GYM_ROUTES = frozenset({"seed_session", "verify", "close_session", "aggregate_metrics", "reverify_mode"})
Expand Down Expand Up @@ -67,8 +67,8 @@ def write_server(source: str | Path, root: str | Path, *, name: str | None = Non
served = None if url else Server(build_app(source, reward=summed if reward_components else reward, sandbox=sandbox,
max_concurrent_envs=1))
try:
env_name, tasks, probed = tasks_and_tools(url or served.url, local="name the bundle's folder instead of its "
"server", timeout=reset_timeout)
env_name, tasks, probed, firsts = tasks_and_tools(url or served.url, local="name the bundle's folder instead "
"of its server", timeout=reset_timeout)
tools = {each: [_function(tool) for tool in found] for each, found in probed.items()}
finally:
if served is not None:
Expand Down Expand Up @@ -96,6 +96,7 @@ def write_server(source: str | Path, root: str | Path, *, name: str | None = Non
"responses_create_params": {"input": messages(task["system_prompt"], task["prompt"]),
"tools": tools[key(task)]},
"agent_ref": agent, "split": split, "index": task["index"], "task_id": task["id"],
**({FIRST_TURN_TOOLS: firsts[key(task)]} if key(task) in firsts else {}),
}) + "\n" for task in described))
return server

Expand Down
8 changes: 5 additions & 3 deletions src/agent_env/training/nemo_gym/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
"Python 3.13.14 or later as NeMo Gym is") from exc

from agent_env.openenv import build_app
from agent_env.training._openenv import STOP_SECONDS, Server, Session, Shared
from agent_env.training._openenv import STOP_SECONDS, Server, Session, Shared, exactly
from agent_env.training.nemo_gym.prepare import REWARD_AND_COMPONENTS, summed

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -89,14 +89,16 @@ def _checked(self) -> "AgentEnvResourcesServerConfig":


class AgentEnvSeedSessionRequest(BaseSeedSessionRequest):
"""A row: the tools it shows the model, and the task it plays."""
"""A row: the tools it shows the model, the task it plays, and for a task that applies server config, the tools its
first turn must offer exactly."""

model_config = ConfigDict(extra="allow")

responses_create_params: dict[str, Any]
split: str
index: int
task_id: str | None = None
first_turn_tools: list[str] | None = None


class AgentEnvVerifyRequest(BaseVerifyRequest):
Expand Down Expand Up @@ -139,7 +141,7 @@ async def seed_session(self, request: Request, body: AgentEnvSeedSessionRequest)
session_id = request.session[SESSION_ID_KEY]
offered = frozenset(tool["name"] for tool in body.responses_create_params.get("tools") or ())
session = _Session(Shared(self._url, offered, self.config.tool_timeout, self.config.reset_timeout,
patience=self.config.reset_timeout))
patience=self.config.reset_timeout, exactly=exactly(body.first_turn_tools)))
if (previous := self._sessions.pop(session_id, None)) is not None:
await previous.close()
self._sessions[session_id] = session
Expand Down
19 changes: 15 additions & 4 deletions src/agent_env/training/skyrl/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,17 @@
raise ImportError("agent_env.training.skyrl.env runs in SkyRL's trainer: install 'agentenv-framework[skyrl]' into "
"SkyRL's environment") from exc

from agent_env.training._openenv import CLOSE_SECONDS, GRACE_SECONDS, Rollout, Session, Shared, on_session_loop, token_in
from agent_env.training._openenv import (
CLOSE_SECONDS,
FIRST_TURN_TOOLS,
GRACE_SECONDS,
Rollout,
Session,
Shared,
exactly,
on_session_loop,
token_in,
)
from agent_env.training.skyrl.calls import Call, Unparsed, parse

logger = logging.getLogger(__name__)
Expand All @@ -37,7 +47,8 @@ def __init__(self, env_config: Any = None, extras: dict[str, Any] | None = None)
+ 2 * GRACE_SECONDS + CLOSE_SECONDS
self._session = _Session(Shared(server["url"], frozenset(self._tools), server["tool_timeout"],
server["reset_timeout"], patience=server["reset_timeout"],
waits_for_server=True, token=token_in(server.get("token_var"))))
waits_for_server=True, token=token_in(server.get("token_var")),
exactly=exactly(self._episode.get(FIRST_TURN_TOOLS))))
self._scores: dict[str, float] = {}
self._run = self._unparsed = 0
self._failed = False
Expand Down Expand Up @@ -90,8 +101,8 @@ def _stepped(self, action: str) -> BaseTextEnvStepOutput:
if rollout.failure is not None:
if rollout.misgrouped:
self.close()
raise ValueError(f"{self._task()}: {rollout.failure}. Its row lists the tools of its toolset's first "
"task, so a task must offer no tool that one doesn't")
raise ValueError(f"{self._task()}: {rollout.failure}. Its row lists the tools found by playing its "
"toolset's first task, so its toolset misses what tells its tasks apart")
raise _Failed(rollout.failure)
if not rollout.done:
return BaseTextEnvStepOutput(observations=answers, reward=0.0, done=False, metadata={})
Expand Down
Loading
Loading