Repository navigation
Conversation
…r from its config group's probed task
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tasks that apply server config share one probe, because their config replies can reveal tools and no static key tells apart what they reveal. When a later task in that group offers more tools than the probed one, its rollout is already refused (
Misgrouped). When it offers fewer, it trains silently on a tool list that includes tools its episode refuses. With this PR, a trainer checks an episode of such a group against the tools the probed task offered at its first turn, and refuses an episode whose first turn offers any other set, in either direction.tasks_and_toolsalso returns, for each toolset whose tasks apply server config, the tools its probed task offered at its first turn.probealready measured these.first_turn_toolsfield: chat (Miles, slime), verl and SkyRL (a new nullable string column in their parquet schemas) and NeMo Gym (a top-level row field its seed request reads). TRL passes it straight toShared.exactly.Sessionrefuses (Misgrouped) an episode whose first-turn tools differ fromShared.exactly. A row without the field skips this check and keeps today's check: it refuses only tools beyond the ones the row shows. That covers rows written before this change and toolsets without config.Rollout
New rows for config toolsets carry the field. Old rows, and toolsets without config, behave as today. A bundle that mixes tasks whose configs reveal different tools now fails loudly where it used to train silently:
RuntimeError, and SkyRL raisesValueError, as both already do for an episode offering more tools.keep_scored_groupsdrops its group. Now a rollout that offers fewer is left out the same way.No server change and no extra probe. Merge after #833, which this is stacked on (its base is #833's branch); retarget to
mainonce #833 lands. Ships with the next tagged release. Revert: revert this PR.Validation
Head
d778fb383.test_a_task_whose_config_reveals_other_tools_than_its_probed_tasks_is_refused_and_an_older_row_plays[fewer]and TRLtest_a_rollout_whose_config_reveals_fewer_tools_than_its_classs_stops_training[fewer], through the real server,probeandSession; both fail on the base branch[more](the existing check), andtest_a_task_revealing_at_its_first_turn_what_its_probed_task_reveals_later_is_refused, which only the new check refusestest_tasks_whose_configs_reveal_the_same_tools_share_one_probe_whatever_their_args_and_play[reveal-the-same, reveal-nothing]: one probe throughwrite_rows/tasks_and_toolstest_a_row_of_a_task_with_server_config_keeps_its_toolsets_first_turn_tools), which fail when the parquet column is dropped; the chat[fewer]test plays the same row with the field removedtst/verl/loop_test.pyandtst/skyrl/generator_test.py: a row whose storedfirst_turn_toolsdiffers is refused and one that matches plays;tst/nemo_gym/server_test.py: a differing row is maskedtest_a_config_between_turns_is_compared_on_the_first_turns_tools_not_on_every_turnsfails when the field is built from every turn's toolsd778fb383:ruff check: all checks passed. Unit suite (pytest tst/unit packages/agentenv-protocol/tests -n 3 --timeout 300): 17587 passed, 70 skipped, 9 failed, the 9 that also fail on the base branch on this machine (Windows disk recipe, computer exec, MCP CLI builder, docker GitHub clone). None is in this diff.Not in this PR
Known limitations:
fail_task_on_error: falsethat fails in one rollout makes that rollout offer fewer tools, so it is refused: TRL and SkyRL stop, with an error that suggests separate bundles though both tasks carry the same config. Before this change it trained on tools the episode did not offer, which is the case this PR refuses.fail_task_on_errordefaults to true, and noapply_server_configstep in the bundles this was checked against opts out.Quality bar
Head
d778fb383.ruff check, unit suite)d778fb3833f747300dd7b3551to113e3941ed778fb383d778fb383d778fb383d778fb383d778fb383pr_gate.py checkd778fb383Relaxations: none.
🤖 Generated with Claude Code


Confidence Score: 5/5
The reviewed changes appear safe to merge after the required base PR, #833.Summary
This PR rejects training episodes whose first-turn tools differ from the task used to prepare their config group.
Reviews (9) · Last reviewed commit: "wip r7" · Reviewed by Greptile