Skip to content

FRLG AutoStory: fault sweeps on the shared sweep runner - #521

Draft
dayfine wants to merge 8 commits into
frlg/stage20-leaguefrom
frlg/seam4-adopt
Draft

dayfine wants to merge 8 commits into
frlg/stage20-leaguefrom
frlg/seam4-adopt

Conversation

@dayfine

@dayfine dayfine commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Depends on #458 (base frlg/stage20-league) and #464 (nav/leg-trace, merged in for #513's shared runner); the diff also shows #464's commits until it lands in this stack.

FRLG's fault sweep carried its own copy of the sweep runner: SweepFailure/sweep_result(), the clean-first rule, the KNOWN_FAILING wrapper and the 30-seed wild-battle loop. This moves it onto the shared runner from #513: each class now builds SweepVariants for Pokemon::run_sweep(), wild battles run on run_seeds() (still 30 seeds, now logging mean and worst per measure), the class switch runs under sweep_clean_first(), and the list is a Pokemon::KnownFailing. The file keeps only FRLG's rows and how each fault class runs on its sim. Test names and rows are unchanged; the known-failing list keeps its 8 entries, and a new test checks that each entry still names a fault sweep test (KnownFailing::stale()). The only own-commit file is PokemonFRLG_AutoStoryFaultSweep_Tests.cpp; checked by a syntax-only compile with the lab's flags, Fork Draft CI runs the suites.

dayfine and others added 6 commits November 13, 2026 00:19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
…failing list

SimSweep: sweep_result() (the log lines and message both games' sweeps
print), run_sweep() over SweepVariants, sweep_clean_first() and
variant_failure(). SimSeeds: run_seeds() with uint32 seeds, 30 by
default, logging mean and worst per measure. KnownFailing: the
pass-while-failing wrapper and stale() for entries that name no test.
Unit tests for each; adoption in BDSP and FRLG follows per game.
Pokemon Navigation: shared fault sweep runner, seeded rows and known-failing list
…dopt

# Conflicts:
#	SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
#	SerialPrograms/cmake/SourceFiles.cmake
The sweeps build SweepVariants for Pokemon::run_sweep(), the wild battles
run on run_seeds() (30 seeds, as before), the clean-first rule is
sweep_clean_first() and the known-failing list is a Pokemon::KnownFailing,
with a new test that every entry names a fault sweep test. Test names and
the eight known-failing entries are unchanged.
… 47763e7)

Conflict in PokemonFRLG_AutoStoryFaultSweep_Tests.cpp sweep_mount_lost: keep seam4-adopt's Pokemon::sweep_result call with stage20-league's variant count of 1.
make_UNIT_TESTS_ALL() passed the code-health function length limit (50 lines)
once the SimSweep, SimSeeds and KnownFailing suites were added.
@dayfine

dayfine commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Deferred until FRLG merges the shared core #648; the FRLG side gets redone on the lean branch #652 after that. Branch kept.

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: fa68899

Verdict

APPROVED

LGTM. No correctness findings; 4 minor and 3 nits below, none blocking. Scope: this PR's own commits (39e9c1a on PokemonFRLG_AutoStoryFaultSweep_Tests.cpp, fa68899 on ComputerPrograms_UnitTestRunner.cpp) and how they use the shared runner. The runner files themselves (Pokemon_SimSweep/SimSeeds/KnownFailing*) come in through the nav/leg-trace merge and were reviewed in #513; I read them only to check the calls.

Correctness

The sweep tests the same thing as before. Compared the Fork Draft CI logs of the base tip (78a0edc, run 43745607030) and this head (run 43770979623):

  • Fault sweep tests: 920 on base, 921 on head. The names are identical; the one added is Fault sweep: every known failing entry names a test.
  • Fault sweep <row> <class>: N variants, M fail. lines: 621 on each side, identical after sorting (same variant counts and fail counts for every row and class).
  • Failure table rows (| row | step | class | variant | stopped |): 15 on each side, identical.
  • Known failing until ... lines: 8 on each side, identical.
  • Wild battles: 88 rows on each side, with the same battle count and clean time per row, and the same pass counts (87 at 30/30, 02.02 at 28/30).
  • Whole suite: 35,600 passed, 0 failed (base: 35,581 passed, 0 failed).

By reading:

  • differs() and line_with_battle() now compare to_string() of the positions instead of the positions. to_string(MapPosition) prints the map and the tile, so the two comparisons agree.
  • Variants are now built first and run inside run_sweep(). The lambdas capture row and clean by reference; in every sweep both outlive the sweep_variants() call, including sweep_mount_lost()'s own riding clean.
  • Variant counts are variants.size() and match the old hand counts (DARK_CAVE_SEEDS + 1, warps.size(), moved_step_taps, bike_legs).
  • sweep_clean_first() logs the same "not swept, the row fails clean" line; KnownFailing::run() keeps the catch of Exception and the same three outcomes.
  • Nothing left half-moved in this file: no local SweepFailure, sweep_result, mean_of, GAME_ODDS_SEEDS or KNOWN_FAILING map remains.

One visible change, logs only: the wild-battle summary is now two lines (Seeds <row> wild battles: passed P/N; <measure> mean, worst; ... from run_seeds(), then Fault sweep <row> wild battles: N battles, clean time T s.) instead of one. The means match the old ones (02.02: 146.6 s, 38.2 tiles, 6.7 s per battle). I found no script on this branch or on agents that parses the old line.

CI on the head

Passed: Fork Draft CI (Ubuntu GCC build + unit tests), C++ Code Health, Check for Tabs, C++ CI Python Bindings, Runtime map reader, Story data codec. Skipped because the PR is a draft: the six C++ CI Serial Programs workflows (Mac, Mac Intel, Ubuntu Default, Ubuntu clang, Windows Clang, Windows Default), so MSVC and the clang builds have not compiled this head. Local code health against origin/frlg/stage20-league: 0 failures.

Minor

  1. /*param=*/ comments on new calls (code health param_comment), PokemonFRLG_AutoStoryFaultSweep_Tests.cpp:

    • line 443, run_row(row, script, {}, clean.scripted) in whole_row_variant()
    • line 651, the same call in the warp no-op variant
    • line 689, sweep_result(logger, row.id, ..., 1, {{"", "riding, no fault", clean.failure}})

    end_pose_variant()'s run_row(row, SimScript{}, {{c, tile}}, clean.scripted) has the same shape, though the linter does not flag it.

  2. "" sentinel converted to std::optional in three places. differs(), line_with_battle() and Test_AutoStoryFaultSweep::sweep() each write x.empty() ? std::nullopt : std::optional<std::string>(x). RowRun::failure and row_step_on_sim()'s return are the sentinels; make them std::optional<std::string>, or at least use one helper.

  3. fa68899 is in the wrong branch. add_tests_NavigationSimulation() fixes a function-length failure that #513's three add_tests_* lines cause, but it lives only on this FRLG branch. Any other stack that merges nav/leg-trace hits the same limit. It belongs on nav/leg-trace (#464). make_UNIT_TESTS_ALL() is still at exactly the 50-line limit, so the next suite added fails again.

  4. Merge order and description. The diff against frlg/stage20-league carries #513's 1,170 runner lines through the nav/leg-trace merge, and #464 is still open. Merging this first puts the runner into the FRLG stage branch ahead of its parent. The description is also out of date: #458 is closed, and "the only own-commit file" is no longer true since fa68899.

Nits

  1. Old name left in two places: the comment at line 735 ("is KNOWN_FAILING's, by name") and the message at line 869 ("KNOWN_FAILING names no such test:"). The list is known_failing() now.
  2. KnownFailingNamesInput, KnownFailingNamesExpectation and the one-row KNOWN_FAILING_NAMES_CASES table are empty ceremony: the test takes no input and has one case. A plain UnitTest with a fixed name says the same. The test itself is not vacuous; it catches a list entry that names no test.
  3. whole_row_variant() and end_pose_variant() return lambdas that hold row and clean by reference. Correct today; one line in their comments saying the variant must not outlive both would keep it so.

Out of scope, for a follow-up

  • PokemonFRLG_AutoStoryLavender_Tests.cpp:477 still has its own GAME_ODDS_SEEDS = 30 and a hand-written seed loop with its own summary (test_rock_tunnel_game_odds). It is a row test, not a fault sweep, so it is outside this PR's stated change, but it is the remaining copy of what run_seeds() does.
  • Code health reports test_only_api advisories on the shared headers (KnownFailing, SeedRun, SeedsResult, run_seeds, SweepVariant, run_sweep, sweep_clean_first, variant_failure, DEFAULT_SWEEP_SEEDS, SeedStats). These are #513's files, and the runner is test infrastructure used only from test files by design; they need an answer before an upstream handoff, not here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant