Skip to content

fix(training): refuse an OpenEnv episode whose first-turn tools differ from its config group's probed task - #840

Open
cikowski wants to merge 6 commits into
mc/ugml-1707-reveal-toolsfrom
mc/ugml-1837-reveal-grouping
Open

cikowski wants to merge 6 commits into
mc/ugml-1707-reveal-toolsfrom
mc/ugml-1837-reveal-grouping

Conversation

@cikowski

@cikowski cikowski commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Tasks that apply server config share one probe, because their config replies can reveal tools and no static key tells apart what they reveal. When a later task in that group offers more tools than the probed one, its rollout is already refused (Misgrouped). When it offers fewer, it trains silently on a tool list that includes tools its episode refuses. With this PR, a trainer checks an episode of such a group against the tools the probed task offered at its first turn, and refuses an episode whose first turn offers any other set, in either direction.

  • tasks_and_tools also returns, for each toolset whose tasks apply server config, the tools its probed task offered at its first turn. probe already measured these.
  • The rows carry that list as an optional first_turn_tools field: chat (Miles, slime), verl and SkyRL (a new nullable string column in their parquet schemas) and NeMo Gym (a top-level row field its seed request reads). TRL passes it straight to Shared.exactly.
  • Session refuses (Misgrouped) an episode whose first-turn tools differ from Shared.exactly. A row without the field skips this check and keeps today's check: it refuses only tools beyond the ones the row shows. That covers rows written before this change and toolsets without config.
  • The comparison uses the first turn because a row shows the tools of every turn combined, so a multi-turn task would fail an equality check against that list.

Rollout

New rows for config toolsets carry the field. Old rows, and toolsets without config, behave as today. A bundle that mixes tasks whose configs reveal different tools now fails loudly where it used to train silently:

  • TRL stops training with a RuntimeError, and SkyRL raises ValueError, as both already do for an episode offering more tools.
  • verl and NeMo Gym already handle a rollout that offers more tools by leaving it unscored (verl after its retries) or masked. Chat (Miles, slime) flags it with reward 0, and keep_scored_groups drops its group. Now a rollout that offers fewer is left out the same way.

No server change and no extra probe. Merge after #833, which this is stacked on (its base is #833's branch); retarget to main once #833 lands. Ships with the next tagged release. Revert: revert this PR.

Validation

Head d778fb383.

Done when Status Evidence Limits
A task whose config reveals fewer tools than its group's probed task is refused MET chat test_a_task_whose_config_reveals_other_tools_than_its_probed_tasks_is_refused_and_an_older_row_plays[fewer] and TRL test_a_rollout_whose_config_reveals_fewer_tools_than_its_classs_stops_training[fewer], through the real server, probe and Session; both fail on the base branch
One that reveals more is refused MET the same chat test [more] (the existing check), and test_a_task_revealing_at_its_first_turn_what_its_probed_task_reveals_later_is_refused, which only the new check refuses
Tasks whose configs reveal the same tools, or nothing, share one probe whatever their args, and play MET test_tasks_whose_configs_reveal_the_same_tools_share_one_probe_whatever_their_args_and_play[reveal-the-same, reveal-nothing]: one probe through write_rows / tasks_and_tools
Every row trainer carries the field, and an older row without it plays MET writer round-trips for verl, SkyRL and NeMo Gym (test_a_row_of_a_task_with_server_config_keeps_its_toolsets_first_turn_tools), which fail when the parquet column is dropped; the chat [fewer] test plays the same row with the field removed
verl, SkyRL and NeMo Gym rollouts read the field MET tst/verl/loop_test.py and tst/skyrl/generator_test.py: a row whose stored first_turn_tools differs is refused and one that matches plays; tst/nemo_gym/server_test.py: a differing row is masked run in CI (the verl, SkyRL and NeMo Gym workflows), not on this machine
The first-turn choice is pinned MET test_a_config_between_turns_is_compared_on_the_first_turns_tools_not_on_every_turns fails when the field is built from every turn's tools
  • On d778fb383: ruff check: all checks passed. Unit suite (pytest tst/unit packages/agentenv-protocol/tests -n 3 --timeout 300): 17587 passed, 70 skipped, 9 failed, the 9 that also fail on the base branch on this machine (Windows disk recipe, computer exec, MCP CLI builder, docker GitHub clone). None is in this diff.

Not in this PR

Known limitations:

  • A config applied after the first turn is not compared. A later turn that offers fewer tools than the probed task's same turn still trains on the combined list. Comparing every turn needs a per-turn tool list in each row, a larger row format change. Config steps normally run before the first prompt.
  • Mixed bundles are refused, not trained: making them trainable (splitting a config group by what each config reveals, at one probe per distinct config) is a separate change.
  • verl and NeMo Gym leave a refused rollout out instead of stopping training, as they already do for an episode offering more tools.
  • Rows written from a different version of a bundle than the one served compare against the stored first-turn list, so a toolset's own first task can be refused, and the refusal suggests separate bundles where rewriting the rows is the fix. Rows are written from the served bundle in the documented flow.
  • A config step marked fail_task_on_error: false that fails in one rollout makes that rollout offer fewer tools, so it is refused: TRL and SkyRL stop, with an error that suggests separate bundles though both tasks carry the same config. Before this change it trained on tools the episode did not offer, which is the case this PR refuses. fail_task_on_error defaults to true, and no apply_server_config step in the bundles this was checked against opts out.

Quality bar

Head d778fb383.

Gate Ran on Result
Local CI (ruff check, unit suite) d778fb383 ruff clean; 17587 passed, the same 9 failures as the base branch
Baseline: the 9 failures on the base branch 3f747300 same 9 fail
Earlier rounds dd7b3551 to 113e3941e schema, test and refusal-text findings, all fixed; root-cause note for the refusal text
Test review: mutants on the changed lines d778fb383 PASS: 5 of 5 mutants of the refusal killed; the named tests fail on the base branch
Adversarial review: every refusal path, consumers, regressions d778fb383 PASS: every clause true on every path except the two waived limitations
Conformance review: design note, standards, this description d778fb383 PASS
Greptile d778fb383 5/5
CI d778fb383 52 green, 1 skipped
pr_gate.py check d778fb383 exit 0, every bar met

Relaxations: none.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge after the required base PR, #833.

Summary

This PR rejects training episodes whose first-turn tools differ from the task used to prepare their config group.

  • Config-based tool groups reject episodes with a different first-turn tool set.
  • Chat and NeMo Gym episodes check config-group tools against their saved first-turn set.
  • verl and SkyRL episodes compare config-group tools with their saved first-turn set.

Reviews (9) · Last reviewed commit: "wip r7" · Reviewed by Greptile

@cikowski
cikowski requested a review from a team as a code owner November 20, 2026 07:15
…he multi-turn more case, refusal texts name both directions, AGENTS.md row lines

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant