Skip to content

docs: design headless mode for SerialPrograms (--headless) - #657

Merged
dayfine merged 2 commits into
lab/mainfrom
lab/headless-cc-design
Nov 19, 2026
Merged

dayfine merged 2 commits into
lab/mainfrom
lab/headless-cc-design

Conversation

@dayfine

@dayfine dayfine commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Base: lab/main (356aa15e1)

Adds docs/design/Headless-CC.md anchored on lab/main, addressing all findings from re-review #5968689083:

  • B1 (macOS-only scope): PR 2 is macOS-only (--headless rejected outside macOS alongside --profile); apply_lab_startup_policy() explicitly handles StartupMode::HEADLESS (+ LabStartupPolicy_Tests). Cross-platform profile/device isolation, Windows SetConsoleCtrlHandler, and std::quick_exit are deferred to Future.
  • M1 (Readiness): Uses the controller status ("ready" / "none" / "", factored from LabChannel without requiring the channel) instead of unused ProgramState::NOT_READY; controller-free programs (UnitTestRunner) skip the wait.
  • M2 (PanelCatalog ownership): Scans PanelListDescriptor::panels() per lookup (no caching in PR 1) and converts the matching unique_ptr<PanelDescriptor> into an owning std::shared_ptr<const PanelDescriptor> stored by HeadlessHost.
  • M3 (Exit code 1 on lab/main): Defined via Listener::error() between RUNNING and STOPPED (excluding update_historical_stats_with_current()'s "Unable to save stats."), with ActionSource::PROGRAM_ERROR on lab/tooling noted second.
  • M4 (--options & per-device leases): Rejects ConsoleSetup in --options with exit 4; per-device lease UserSetupError fails fast with log + exit 5.
  • m1–m7: Fixes LabChannel branch note & Agent-Integration.md link, guards ControllerSettingsTable and skips STATS_FILE, drops redundant m_sleep_scope, bounds signal stop wait (130/143), clarifies GUI exit codes & open -n log/state reporting, specifies dynamic_cast<ProgramSession*> + UnitTestRunner acceptance options, and skips Discord/social SDK.

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: a8be72a

Design review of docs/design/Headless-CC.md, checked against the code at this SHA (base lab/tooling 79519ac) and against lab/main 356aa15. Docs only, so no compile or code-health run; the six C++ CI jobs were still in progress when I read the checks.

Verdict

NEEDS_REWORK

The direction is sound and the seams in §2 are real, but the doc is written against a different branch than its base, and it leaves out the four things that decide whether this is safe to run next to a live console: profile and serial device, settings writes, the exit path under the Qt thread rules, and dialogs. Findings: 4 blocking, 6 major, 4 minor.

What checks out

  • PanelHolder is implemented only by MainWindow; make_ui_component() is called only in MainWindow::load_panel() (MainWindow.cpp:427), so a session without widgets is possible.
  • ProgramSession::start_program(ActionSource, std::optional<std::string>), stop_program, current_state() and Listener::state_change exist as described.
  • LabChannel reaches the window only through LabChannelCallbacks{current_panel, select_program} and posts UI work with QMetaObject::invokeMethod(QCoreApplication::instance(), ...) (LabChannel.cpp:815).
  • ProgramSelect does own m_lists and select_program() as §3.2 says.

Blocking

  1. The doc describes lab/main, not its base lab/tooling. StartupMode, StartupPlan, parse_startup_arguments(), StartupArguments.*, StartupArguments_Tests.cpp, stop_application_services() and stop_thread_pools() do not exist on lab/tooling; they came in with #233 on lab/main, and neither branch is an ancestor of the other. On lab/tooling, Main.cpp parses argv inline at line 225, after QApplication, PERSISTENT_SETTINGS().read(), SerialPortPoller::instance().ports() and get_all_cameras(), so "malformed flags fail before Qt or hardware initialization" (§3.1) cannot be met there. --profile on lab/tooling is also macOS-only (Filesystem::set_startup_profile under #if defined(__APPLE__)), so "compatible with --profile" is false on Linux and Windows. Pick the base (I would say lab/main, or merge it into lab/tooling first) and rewrite §1, §3.1, §3.6 and §4 against it.
  2. Nothing stops a headless run from opening another console's serial port. --headless does not require --profile. make_panel() plus load_json_from_global() (§3.3) builds a ControllerSession, whose constructor calls m_descriptor->open_connection(logger) with whatever device the profile saved; with the default profile that is console A's port, the 2026-11-08 failure again. Startup also scans ports and cameras before any mode check. Required: --headless is rejected without --profile; it goes through the lab/main startup policy (apply_lab_startup_policy, reserve_lab_profile() lease); and the doc says what happens when the profile's device is already leased.
  3. --options is described as a one-run override but the design saves it. §3.4 step 6 calls m_current_panel->save_settings(), which does PERSISTENT_SETTINGS().panels[identifier] = to_json() and writes the file (PanelSession.cpp:37-43); report_new_panel_intent() and load_panel() in §3.3 save too. So the override becomes the profile's saved options, and N workers on one profile overwrite each other. Decide and state it: a one-shot run writes no settings (my recommendation), or --options is persistent and the doc says so. Also say whether headless selection writes ProgramCategory / the program-panel key, which decide what the GUI opens next on that profile.
  4. The exit path ignores the thread-adoption rules. "Tears down services cleanly" (§3.4) and "stop_thread_pools() run cleanly" (§3.6) are not what happens on Windows: with PA_ENABLE_QT_ADOPTION_WORKAROUND, main() ends in std::quick_exit(ret) because joining hangs. The doc must say the run's exit code is the value passed there, and that anything that must reach disk (settings, log, Lab/state) is flushed before it. Signals: a QSocketNotifier self-pipe is POSIX only; on Windows SetConsoleCtrlHandler runs the handler on a thread the OS creates, so it may only set a flag or post to the event loop. No new std::thread, std::mutex or std::async for the waiter; AGENTS.md's concurrency section is not mentioned anywhere in the doc.

Major

  1. QCoreApplication is not feasible as written; keep QApplication and create no window. Session-level code opens QMessageBox outside the three startup checks §3.6 lists: SerialPABotBase2_Connection.cpp:140 (Prolific port, on the connect path), ErrorReports.cpp:371 (send_all_unsent_reports), NewVersionCheck.cpp:145, HardwareValidation_x86.tpp. set_permissions() asks for camera and microphone access through qApp and a QObject; a headless camera run still needs that request. #207 could use QCoreApplication only because it touches no settings or devices. For PR 1: QApplication, no MainWindow, and an explicit list of dialogs turned into log line plus exit code. QCoreApplication can be a later, measured step.
  2. Video and audio capture are not covered. The doc should state that capture lives in the sessions (the backends hold a QVideoSink) and runs with no display widget, name what was checked, and cover audio. PR 2's acceptance (snapshot) depends on it. On macOS the lab rule is that the app gets camera frames only when launched with open -n, which does not hand back the exit code; say that direct exec with an exit code is for the mGBA link source and how a capture-card run reports its result.
  3. Exit code has no stated source, though one exists. ProgramSession::last_transition() already separates ActionSource::PROGRAM from PROGRAM_ERROR (ProgramTransition.h), and start_program() returns an error string. Define the table: finished, program error, stopped by signal, start refused, unknown id, controller never ready. Note that state_change is called from the program side, so the listener must post to the event loop, not act in the callback.
  4. "Waits briefly" for the controller is a fixed delay. Use the state the code has: ProgramState::NOT_READY leaving, or LabConsoleCommands::controller().status == "ready" as LabChannel's start does, with a timeout and its own exit code.
  5. PanelCatalog ownership does not fit the existing types. PanelEntry owns std::unique_ptr<PanelDescriptor>, PanelHolder::load_panel() takes std::shared_ptr<const PanelDescriptor> (today made by PanelListWidget), and PanelSession keeps a const PanelDescriptor&. A find_panel() returning const PanelDescriptor* cannot feed load_panel(). Say who owns the descriptor and that it outlives the session. select_program() in §3.3 also repeats PanelListWidget::handle_panel_clicked() (make, load JSON, load); share it.
  6. Slicing. PR 1 is three changes: a refactor of GUI code (ProgramSelect), new startup flags, and the host with Main.cpp wiring. The catalog extraction should be its own no-behavior-change PR. Signal handling is used by the one-shot run (§3.4 step 5) but scheduled in PR 2; it belongs with the one-shot. PR 2 (LabChannel without a window) is the part with the least proven need, since the lab already drives the GUI app; keep it out until the one-shot runner is in use.

Minor

  1. Prior work is not mentioned. #206/#207 (use_qt_ui(), SerialProgramsTests: own executable, QCoreApplication, own user folder, now on lab/main) are the headless precedent and upstream candidates. The doc should say why this is a flag in Main.cpp and not a second small executable on the same pattern, which would keep the change out of a file upstream owns, and how it relates to docs/design/Agent-Integration.md.
  2. on_busy() / on_idle() are called only by the program widgets through global_panel_holder(), so in headless they are never called; a SleepSuppressScope toggle there would not fire. Hold the scope for the length of the run instead.
  3. PA_MGBA_LINK does not appear in the source on lab/tooling (only VideoSource_MgbaLink), and "mGBA Slice 3-4 runner" has no link. The PR 1 acceptance "or a mock/no-op program" names no program; name one that needs no controller, and add a unit test for HeadlessHost with a fake panel.
  4. ActionSource::CLI "if desired": decide. LabHold and the state file read the source, so a headless start reported as UI would look like a user's click.

@dayfine
dayfine force-pushed the lab/headless-cc-design branch from a8be72a to 8da7692 Compare November 19, 2026 02:56
@dayfine
dayfine changed the base branch from lab/tooling to lab/main November 19, 2026 02:57

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: 8da7692

Re-review of docs/design/Headless-CC.md against the previous review (#pullrequestreview-5968597602, at a8be72a) and the code on lab/main 356aa15, which is now the PR base and the merge base. Docs only: no compile or code-health run. The PR head has no CI checks reported. The file has LF endings and no tabs.

Verdict

NEEDS_REWORK

The rework fixed most of the earlier review: base, profile requirement, settings on exit, QApplication, exit path and slicing are now right or close. Three things still do not match lab/main: the profile rule makes headless macOS-only while the doc designs for Windows and Linux, the readiness check waits on a state nothing ever sets, and the PanelCatalog ownership story does not fit how descriptors are made. Open: 1 blocking, 4 major, 7 minor.

Status of the earlier findings

# Earlier finding Status
1 Wrong base Partly. Base is lab/main and §1, §2, §3.1, §4 are written against it; Main.cpp:399, :331, :355, PanelListWidget.cpp:90, MainWindow.cpp:427 all check out. Still open: the macOS-only profile point (B1) and LabChannel placed on the wrong branch (m1).
2 Serial safety, --headless requires --profile Partly. Requirement, policy and profile lease are in §3.1. Open: platform scope (B1) and the per-device leases (M4).
3 --options must not be saved Resolved for the override itself: no save_settings(), no ProgramCategory / ProgramPanel write, no write on exit. Two remaining writers are not listed (m2).
4 Exit path, threads, signals Resolved. ret goes to std::quick_exit, log is flushed first (stop_application_services() already calls global_file_logger().stop() before it), POSIX and Windows handlers are separate, AGENTS.md rules are named. See m4 for the stop wait.
5 QCoreApplication infeasible Resolved. QApplication with no window; dialog list matches the code, plus SetupSettings.cpp:85.
6 Video/audio, macOS open -n Resolved at design level. "Reports results via log/state files" names no file or format; say which one the runner reads (m5).
7 Exit code source Partly. Table is there, but the source of 1 is given only for lab/tooling (M3).
8 Controller readiness Not resolved. The chosen signal is dead code on lab/main (M1).
9 PanelCatalog ownership Partly. shared_ptr return and a shared load helper are stated, but the ownership model does not match the code (M2).
10 Slicing Resolved. Catalog extraction is its own PR; signals ship with the one-shot run; LabChannel mode is deferred.
11 #206/#207 not mentioned Resolved.
12 on_busy() / on_idle() Resolved, but now redundant (m3).
13 PA_MGBA_LINK, unnamed test program Resolved. UnitTestRunner is a real identifier; see m6.
14 ActionSource::CLI Resolved (decided: CLI, once merged with lab/tooling).

Blocking

B1. --headless requires --profile, and --profile is rejected outside macOS. On lab/main, main() sets policy.profiles = REQUIRED only under __APPLE__; elsewhere apply_lab_startup_policy() returns "Settings profiles are supported only on macOS." for any plan with a profile (LabStartupPolicy.cpp), Filesystem::set_startup_profile is called only on macOS, and reserve_lab_profile() returns nullptr (LabDeviceRuntime.cpp:52). So as written, headless can never start on Linux or Windows, yet §3.4 says exit codes apply to "Linux/Windows runs" and §3.5 designs SetConsoleCtrlHandler and the quick_exit path for Windows. The earlier review raised the macOS-only profile under finding 1; the new text does not mention it. Pick one and say it: (a) PR 2 is macOS-only, --headless is rejected elsewhere, and the Windows signal design moves to "Future"; or (b) define what isolates settings and devices on Linux/Windows (profile support there is its own change) before headless is allowed. An implementer who "fixes" this by relaxing the profile rule off macOS removes the safety finding 2 asked for. Also note that apply_lab_startup_policy() returns early for any mode other than GUI, so a new HEADLESS mode must be added to that check and to LabStartupPolicy_Tests explicitly.

Major

M1. The readiness wait observes a state that is never set. Nothing on lab/main calls set_state(ProgramState::NOT_READY): m_state starts as STOPPED (ProgramSession.cpp:73) and set_state is only called with RUNNING, STOPPING and STOPPED. NOT_READY appears only in switch cases. So §3.5 step 2 passes at once with the controller still connecting, start_program() succeeds, and exit code 5 cannot happen; the run then fails later as a program error. The earlier review offered NOT_READY as one of two options; that option was wrong, and I should have checked it. Use the other one: the controller state LabChannel reports as "ready" / "none" (LabChannel.h:88, LabRequest.h:70, exercised by FakeController in LabChannel_Tests.cpp), factored so HeadlessHost can read it without the channel, with the timeout and exit 5. Say what a program with no console (UnitTestRunner, controller "") does: no wait.

M2. PanelCatalog ownership does not match how descriptors exist. PanelListDescriptor holds no descriptors: panels() calls make_panels() and returns a fresh std::vector<PanelEntry> on every call (PanelList.h), and PanelListWidget's constructor moves each unique_ptr<PanelDescriptor> out of that vector into its own shared_ptr map (PanelListWidget.cpp:69). There is nothing "owned by the catalog/list lifetime" for an aliasing shared_ptr to point at, and an aliasing pointer with no owning control block is a raw pointer with extra steps. Simpler and correct: find_panel(identifier) calls panels(), and converts the matching entry's unique_ptr into an owning std::shared_ptr<const PanelDescriptor>; the host keeps it next to the session, as §3.3 already lists. Say whether the catalog caches entries (then the GUI lists must read from the same cache, which is a behavior change for PR 1) or builds them per lookup as select_program() does today.

M3. The source of exit code 1 is not stated for the branch PR 2 lands on. lab/main has no ProgramTransition.h, ActionSource or last_transition(); start_program() takes no arguments. What it has: every error branch in ProgramSession::run_program() calls report_error(), which reaches Listener::error(message), and a normal finish does not. Define 1 as "error() received between RUNNING and STOPPED", and note the one false positive: update_historical_stats_with_current() pushes error("Unable to save stats.") after a good run. State the rule for lab/main first and the lab/tooling mapping second.

M4. --options and device opening. Two gaps on the path the earlier finding 2 was about:

  • §3.1 and §3.5 step 1 apply --options with a second load_json() after load_json_from_global(). For a console program, ConsoleProgramSession::load_json() passes ConsoleSetup to m_system.load_json(), which reloads the devices; LabChannel.cpp:129-139 strips that key for exactly this reason. Say whether an options file may carry ConsoleSetup. If yes, merge the two JSON values and load once, so the profile's saved device is never opened when the override names another. If no, reject the key with exit 4.
  • The doc covers only the profile lease. lab/main also has per-device leases (acquire_lab_serial, acquire_lab_capture, policy from lab-devices.json), which throw UserSetupError on the connect path. Say what the run does then: with M1 fixed it should be a log line and exit 5, not a 10 s wait and not a dialog.

Minor

  • m1. LabChannel is on lab/main (CommonFramework/Windows/LabChannel.{h,cpp}, Lab.md, LabRequest.h, tests), not only on lab/tooling as §1 and §4 say. Only ProgramTransition.h / ActionSource, VideoSource_MgbaLink and docs/design/Agent-Integration.md are lab/tooling-only. Link Agent-Integration.md by branch, as the mGBA note is.
  • m2. "Writes no settings" has two exceptions to list: ControllerSettingsTable saves through OptionsPanelWrapper<ConsoleSettingsPanel>().make_panel()->save_settings() when it meets a new controller (NintendoSwitch_ControllerSettings.cpp:588, from a connection thread), and every run updates STATS_FILE. Neither stores the --options override, so the fix is one sentence each, or a headless guard on the first.
  • m3. ProgramSession::run_program() already holds a SleepSuppressScope for the run (ProgramSession.cpp:323). Drop m_sleep_scope from HeadlessHost.
  • m4. On a signal the host calls stop_program() and waits for STOPPED with no bound. Give it a timeout and say what a second signal does. 130 is the convention for SIGINT; SIGTERM is 143.
  • m5. Exit codes against today's Main.cpp: a held profile lease returns 2 in the GUI path (Main.cpp:402), and check_resource_folder() / check_hardware() return 1. The table gives 3 for both. Fine as a headless-only choice, but say the GUI codes stay, or map them the same way.
  • m6. Acceptance and tests: UnitTestRunner defaults to substring mode with "PokemonSwSh" and reads UNIT_TEST_RESOURCE_PATH(), so name the options file the acceptance run uses and say whether failed tests change the exit code. A fake PanelSession is not a ProgramSession; HeadlessHost needs the same dynamic_cast<ProgramSession*> that LabChannel.cpp:58 uses, and --run on a settings panel should have a stated exit code (4).
  • m7. run_program() also starts Discord (run_on_start) and the social SDK from the profile's settings; say whether headless skips them. The PR description has no base banner (PR_WORKFLOW.md "Linking").

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: 8fffda0

Final pass on docs/design/Headless-CC.md against review #pullrequestreview-5968689083 (at 8da7692) and the code on lab/main 356aa15 (PR base and merge base). Docs only: no compile or code-health run. Checks at this SHA: code-health, no-tabs, build and the three native-leases jobs pass; the six C++ build / build jobs were still pending when I read them. The file has LF endings and no tabs.

Verdict

APPROVED

LGTM: ready to build PR 1 (PanelCatalog extraction) from. All twelve open findings are resolved in the text, and the line references I checked match lab/main. Nothing blocks PR 1. Four points below affect PR 2 only; fix them in the doc when PR 2 starts, not before PR 1.

Status of the open findings

# Finding Status Checked against lab/main
B1 Platform scope Resolved §3.1 and §4 make PR 2 macOS-only and reject --headless elsewhere; Windows handler and quick_exit moved to Future. The text says apply_lab_startup_policy() must stop returning early for HEADLESS and that LabStartupPolicy_Tests covers it; the early return is LabStartupPolicy.cpp if (plan.mode != StartupMode::GUI). LabDeviceRuntime.cpp:52 is right.
M1 Readiness Resolved Uses the controller status "ready" / "none" / "" with a timeout and exit 5; no-console programs skip the wait. LabChannel.h:88 and LabRequest.h:70 are right. See P2-c for where the status is computed.
M2 Catalog ownership Resolved Per-lookup scan of panels(), the entry's unique_ptr converted to an owning shared_ptr<const PanelDescriptor>, kept by the host next to the session, no cache in PR 1. Matches PanelList.h and ProgramSelect::select_program().
M3 Source of exit 1 Resolved Listener::error() between RUNNING and STOPPED, the stats-save message excluded, lab/tooling mapping second. Matches report_error() / push_error("Unable to save stats.") in ProgramSession.cpp. See P2-a.
M4 --options and device leases Resolved ConsoleSetup in --options is rejected with exit 4 before any device opens (LabChannel.cpp:129-139 is right); a per-device lease UserSetupError is a log line and exit 5. See P2-d for how the host sees it.
m1 LabChannel branch Resolved §1 places it on lab/main; Agent-Integration.md is linked by branch.
m2 Other settings writers Resolved NintendoSwitch_ControllerSettings.cpp:588 guarded, STATS_FILE skipped in headless.
m3 m_sleep_scope Resolved Dropped; ProgramSession.cpp:323 is right.
m4 Stop wait after a signal Resolved Bounded wait, second signal exits, 130 / 143.
m5 Exit codes vs Main.cpp Resolved Stated as headless-only; GUI keeps 2 (Main.cpp:402) and 1.
m6 Acceptance run, non-program panel Resolved Options file named, dynamic_cast<ProgramSession*> (LabChannel.cpp:58) with exit 4. See P2-a and P2-b.
m7 Discord, base banner Resolved Discord and social SDK skipped; the description opens with the base.

For PR 1 (not blocking; decide while coding)

  • Shared load helper, §3.2. PERSISTENT_SETTINGS().panels[JSON_PROGRAM_PANEL] is written with the list's display name (iter->first in PanelListWidget::handle_panel_clicked()), and the widget also writes "" for an unknown name. A helper that takes a descriptor does not have that key. Simpler than a bool persist_selection parameter: the helper does intent, make_panel(), load_json(), load_panel() and returns the error; the widget keeps its own settings write and its QMessageBox. Then the helper has no flag and no Qt include.
  • A test for find_panel(). PR 1's acceptance lists only tests that already exist. If PanelCatalog takes its lists in the constructor (the game factories are added by the caller, as ProgramSelect::add() does today), a fake PanelListDescriptor can test "known identifier returns a descriptor with that identifier, unknown returns null" without loading every game's icon.
  • The doc does not name the catalog's file. It includes every game's PanelListFactory, so it belongs next to Source/PanelLists.{h,cpp}, and goes into SourceFiles.cmake.

For PR 2 (fix in the doc before PR 2 starts)

  • P2-a. A failed unit test does not produce exit 1. UnitTestRunner::on_test_finished() only counts failures (m_total_failed, the FAILED_TESTS text); program() returns normally after runner.run(), so no Listener::error() fires and the run exits 0. The acceptance line "1 on test failure" needs either a throw from UnitTestRunner::program() when the count is not zero (a behavior change to that program) or a different acceptance check.
  • P2-b. UnitTestRunner is listed only in developer mode. NintendoSwitch_Panels.cpp:94 sits inside if (STATIC_GLOBALS.DEVELOPER_MODE). Without it, --run "UnitTestRunner" is an unknown identifier and exits 4. Say how the acceptance run gets developer mode in its profile.
  • P2-c. Where the controller status comes from. The only implementation is SwitchLabCommands::controller_status(PanelSession&) (NintendoSwitch_LabCommands.cpp:95), behind the LabConsoleCommands interface in LabChannel.h. "Factored from LabChannel.h:88" should name that function and say where the shared piece goes, so HeadlessHost in CommonFramework does not include a NintendoSwitch header.
  • P2-d. How the host sees a per-device lease failure. acquire_lab_serial() throws on the connection thread (SerialPABotBase2_Connection::open_guarded_serial_port()), acquire_lab_capture() in the camera and audio device constructors. None of these reach HeadlessHost as an exception. Name the signal it reads (a controller or video status, or a hook at the lease call); otherwise the run takes the 10 s readiness timeout the doc says it avoids.
  • P2-e. The example identifier "Pokemon FireRed/LeafGreen:AutoStory" is not the identifier form. Identifiers are <Game>:<Program> as in Lab.md and StatsDatabase.cpp (PokemonBDSP:AutoStory, PokemonFRLG:LuckyEggFarmer).

@dayfine
dayfine merged commit cd86fe3 into lab/main Nov 19, 2026
6 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant