Skip to content

fix(task-steps): a run-wide agent pick of another agent no longer takes the step's pinned version - #842

Open
cikowski wants to merge 2 commits into
mainfrom
fix/agent-override-version
Open

cikowski wants to merge 2 commits into
mainfrom
fix/agent-override-version

Conversation

@cikowski

@cikowski cikowski commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

A step's a2a_agent_version is the pinned version of the agent that step names. When a run picks a different agent for the whole run (user_overrides.a2a_agent_id: task run --a2a-agent-id, a hub run's agent pick) and gives no version, deploy_agent and install_agent asked the store for the picked agent at the step's pinned version. On main, a step pinned to theirs v4 run with a pick of mine asks for mine v4: a version that does not exist, or an unrelated build of the other agent.

Now a run-wide pick that names another agent is deployed or installed at the run-wide a2a_agent_version, or at its latest version when none is given. A pick that names the step's own agent (or the configured default agent, for a step that names none) keeps the step's pin.

Unchanged: a step's own step_params override, a run-wide version with no run-wide id (applies to deploy_agent steps only, as before), and deploy_user_sim_agent, which never takes the run-wide pick.

Rollout

How it was tested

Validation: head 61dcf8d3a7c597f5f79c061e5230ca370000fc36. Each case runs the step's execute() and reads the (id, version) it asks the agent store for.

Done when Status Evidence
a pick of another agent does not take the step's pin (deploy and install) MET test_a_pick_of_another_agent_does_not_take_the_steps_pinned_version x3: fails on main (('mine', 4)), passes on head
a pick of another agent with a version uses that version MET test_a_pick_of_another_agent_with_a_version_deploys_that_version x3; the install case fails on main
a pick of the step's own agent, and no pick at all, keep the pin MET test_a_pick_of_the_steps_own_agent_keeps_its_pin x3, test_a_step_with_no_pick_keeps_its_agent_and_pin x3
a step naming no agent keeps its pin only for the default agent MET test_a_step_naming_no_agent_keeps_its_pin_only_for_the_default_agent x2: fails on main
a pick still runs where the default agent cannot be read from the config MET test_a_pick_runs_where_the_default_agent_cannot_be_read x2: fails on main on the version, and without the ConfigError catch
  • .venv/bin/python -m pytest -n0 tst/unit/task_step/run_wide_agent_pick_test.py tst/unit/bundle/agent_override_test.py tst/unit/task_step/test_step_overrides_agent_and_model.py tst/unit/task_step/test_deploy_user_sim_agent.py tst/unit/task_step/install_agent_test.py tst/unit/cli/task_run_agent_pick_test.py: 144 passed.
  • ruff check: clean.
  • With main's two resolvers, 8 of the 18 cases in run_wide_agent_pick_test.py fail (the rows marked above: 3 + 1 + 2 + 2).
  • pytest tst/unit/task_step tst/unit/cli tst/unit/bundle -n 4: 5,793 passed, 14 skipped, 6 failed. The six (computer_exec_test.py, test_mcp_cli_builder.py) fail the same way with main's src on my machine, do not import this code, and pass in CI. A stale assertion of the old behaviour in tst/unit/bundle/agent_override_test.py is updated here.
  • Not run locally: integration suites (need Docker and a registry), the rest of tst/unit. The repo has no type-check or ruff format gate.
  • Screenshots: none (the only visible change is one --help line).

Quality bar

Head 61dcf8d3a7c597f5f79c061e5230ca370000fc36.

Gate Ran on Result
Local CI (ruff check, pytest on the touched suites) 61dcf8d 5,793 passed, 14 skipped, 6 failed (all six also on main, unrelated); ruff clean
Baseline: same tests with origin/main's resolvers 1db49dd 8 of 18 pick cases fail, as intended; the six unrelated local failures also on main
Test review: reran the tests and mutated the changed lines 61dcf8d PASS: 144 passed, 8 of 8 mutants killed, 8 of 18 cases fail on main as stated
Adversarial review: probes for bad input, config and caller edges 61dcf8d PASS: base-against-head matrix over stored id and pin, run-wide and step overrides and default-agent states; every difference is an intended one
Conformance review: the spec and this description's claims 61dcf8d PASS
Greptile 61dcf8d 5/5
CI 61dcf8d 34 of 34 green; one integration job rerun to green after a Docker Hub DNS failure in an unrelated test
pr_gate.py check 61dcf8d exit 0, every bar met

Relaxations: none.

Not in this PR

Known limitations:

  • Agents are compared by the id string, so a pick that names the step's agent through an alias counts as another agent and deploys latest (safe direction; resolving aliases here would need the installed-bundle lookup at step run time).
  • A pick over a step that names no agent reads the configured default agent to tell whether the pick is another agent. A default that the config cannot resolve is treated as another agent, but a default stored as a secret whose backend raises for that key fails the step, where main did not read it. Waived: catching every backend error here would hide an infrastructure failure that the same step hits on any run without a pick.
  • A pick of the step's own agent that also carries a run-wide version: deploy_agent takes that version and install_agent keeps its pin (pre-existing on main; install_agent has never read a run-wide version for its own agent).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Run-wide picks of a different agent now use the picked version, or latest when none is given, instead of inheriting the step’s pinned version.

  • A run-wide pick uses the picked agent’s version, not another agent’s pin.

cikowski documents these accepted limits: aliases count as different agents; a secret backend error while reading the default agent can fail the step; and same-agent picks retain the existing difference between deploy and install version choices. No new actionable issues were found in the changes since the previous review.

Reviews (4) · Last reviewed commit: "address review round 2: the pick's help ..." · Reviewed by Greptile

…es the step's pinned version

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@cikowski
cikowski requested a review from a team as a code owner November 20, 2026 10:36
…ng state which version a picked agent runs at

@agutmanstein-scale agutmanstein-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 61dcf8d. No blockers. Step overrides and a run-wide a2a_agent_version still win. Otherwise a run-wide pick of another agent gets its latest version and never the step's pin, and picking the step's own agent keeps the pin. install_agent behaves the same way. This is the UGML-1860 fix that synthetic-pipelines #2367 needs before re-registering.

@yzabelle-go yzabelle-go left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right fix: picking mine used to deploy mine at theirs's pin. The 144 touched tests pass locally, and merge-tree with main is clean. Two small comments: the CLI can't set the version the new help mentions, so a CLI pick always floats to latest, and the _resolve_agent_ref docstring now contradicts the own-agent case. Approving.

src/agent_env/task_step/task_steps/deploy_agent.py:335: This docstring still says an overridden id never takes the stored pin, but with this PR a pick of the step's own agent (or the default agent for an unnamed step) keeps it, as test_a_pick_of_the_steps_own_agent_keeps_its_pin checks. Worth updating it to the rule picks_another_agent now implements.

@click.option("--apply-trajectory-filter/--no-trajectory-filter", "apply_trajectory_filter", default=None, help="Force trajectory filtering on/off for rubrics_verifier (overrides the step's trajectory_filter setting)")
@click.option("--a2a-agent-id", default=None, type=str,
help="Override A2A agent id for deploy_agent steps; a user sim's agent keeps its own")
help="Override A2A agent id for deploy_agent and install_agent steps; an agent other than the step's own runs at the run's a2a_agent_version, else its latest, never the step's pinned version; a user sim's agent keeps its own")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This help names "the run's a2a_agent_version", but task run has no flag that sets it: line 422 only writes user_overrides["a2a_agent_id"]. So after this PR, task run --a2a-agent-id mine on a task pinned to theirs v4 always gets mine at latest, resolved separately in each step, and the only way to pin it is a hand-written --context-json. Could you add --a2a-agent-version next to --a2a-agent-id (written into user_overrides), or say "latest; pin via --context-json user_overrides.a2a_agent_version" in the help?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants