Repository navigation
docs: design headless mode for SerialPrograms (--headless) - #657
Conversation
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: a8be72a
Design review of docs/design/Headless-CC.md, checked against the code at this SHA (base lab/tooling 79519ac) and against lab/main 356aa15. Docs only, so no compile or code-health run; the six C++ CI jobs were still in progress when I read the checks.
Verdict
NEEDS_REWORK
The direction is sound and the seams in §2 are real, but the doc is written against a different branch than its base, and it leaves out the four things that decide whether this is safe to run next to a live console: profile and serial device, settings writes, the exit path under the Qt thread rules, and dialogs. Findings: 4 blocking, 6 major, 4 minor.
What checks out
PanelHolderis implemented only byMainWindow;make_ui_component()is called only inMainWindow::load_panel()(MainWindow.cpp:427), so a session without widgets is possible.ProgramSession::start_program(ActionSource, std::optional<std::string>),stop_program,current_state()andListener::state_changeexist as described.LabChannelreaches the window only throughLabChannelCallbacks{current_panel, select_program}and posts UI work withQMetaObject::invokeMethod(QCoreApplication::instance(), ...)(LabChannel.cpp:815).ProgramSelectdoes ownm_listsandselect_program()as §3.2 says.
Blocking
- The doc describes
lab/main, not its baselab/tooling.StartupMode,StartupPlan,parse_startup_arguments(),StartupArguments.*,StartupArguments_Tests.cpp,stop_application_services()andstop_thread_pools()do not exist onlab/tooling; they came in with #233 onlab/main, and neither branch is an ancestor of the other. Onlab/tooling,Main.cppparsesargvinline at line 225, afterQApplication,PERSISTENT_SETTINGS().read(),SerialPortPoller::instance().ports()andget_all_cameras(), so "malformed flags fail before Qt or hardware initialization" (§3.1) cannot be met there.--profileonlab/toolingis also macOS-only (Filesystem::set_startup_profileunder#if defined(__APPLE__)), so "compatible with--profile" is false on Linux and Windows. Pick the base (I would saylab/main, or merge it intolab/toolingfirst) and rewrite §1, §3.1, §3.6 and §4 against it. - Nothing stops a headless run from opening another console's serial port.
--headlessdoes not require--profile.make_panel()plusload_json_from_global()(§3.3) builds aControllerSession, whose constructor callsm_descriptor->open_connection(logger)with whatever device the profile saved; with the default profile that is console A's port, the 2026-11-08 failure again. Startup also scans ports and cameras before any mode check. Required:--headlessis rejected without--profile; it goes through thelab/mainstartup policy (apply_lab_startup_policy,reserve_lab_profile()lease); and the doc says what happens when the profile's device is already leased. --optionsis described as a one-run override but the design saves it. §3.4 step 6 callsm_current_panel->save_settings(), which doesPERSISTENT_SETTINGS().panels[identifier] = to_json()and writes the file (PanelSession.cpp:37-43);report_new_panel_intent()andload_panel()in §3.3 save too. So the override becomes the profile's saved options, and N workers on one profile overwrite each other. Decide and state it: a one-shot run writes no settings (my recommendation), or--optionsis persistent and the doc says so. Also say whether headless selection writesProgramCategory/ the program-panel key, which decide what the GUI opens next on that profile.- The exit path ignores the thread-adoption rules. "Tears down services cleanly" (§3.4) and "
stop_thread_pools()run cleanly" (§3.6) are not what happens on Windows: withPA_ENABLE_QT_ADOPTION_WORKAROUND,main()ends instd::quick_exit(ret)because joining hangs. The doc must say the run's exit code is the value passed there, and that anything that must reach disk (settings, log,Lab/state) is flushed before it. Signals: aQSocketNotifierself-pipe is POSIX only; on WindowsSetConsoleCtrlHandlerruns the handler on a thread the OS creates, so it may only set a flag or post to the event loop. No newstd::thread,std::mutexorstd::asyncfor the waiter; AGENTS.md's concurrency section is not mentioned anywhere in the doc.
Major
QCoreApplicationis not feasible as written; keepQApplicationand create no window. Session-level code opensQMessageBoxoutside the three startup checks §3.6 lists:SerialPABotBase2_Connection.cpp:140(Prolific port, on the connect path),ErrorReports.cpp:371(send_all_unsent_reports),NewVersionCheck.cpp:145,HardwareValidation_x86.tpp.set_permissions()asks for camera and microphone access throughqAppand aQObject; a headless camera run still needs that request. #207 could useQCoreApplicationonly because it touches no settings or devices. For PR 1:QApplication, noMainWindow, and an explicit list of dialogs turned into log line plus exit code.QCoreApplicationcan be a later, measured step.- Video and audio capture are not covered. The doc should state that capture lives in the sessions (the backends hold a
QVideoSink) and runs with no display widget, name what was checked, and cover audio. PR 2's acceptance (snapshot) depends on it. On macOS the lab rule is that the app gets camera frames only when launched withopen -n, which does not hand back the exit code; say that direct exec with an exit code is for the mGBA link source and how a capture-card run reports its result. - Exit code has no stated source, though one exists.
ProgramSession::last_transition()already separatesActionSource::PROGRAMfromPROGRAM_ERROR(ProgramTransition.h), andstart_program()returns an error string. Define the table: finished, program error, stopped by signal, start refused, unknown id, controller never ready. Note thatstate_changeis called from the program side, so the listener must post to the event loop, not act in the callback. - "Waits briefly" for the controller is a fixed delay. Use the state the code has:
ProgramState::NOT_READYleaving, orLabConsoleCommands::controller().status == "ready"asLabChannel'sstartdoes, with a timeout and its own exit code. PanelCatalogownership does not fit the existing types.PanelEntryownsstd::unique_ptr<PanelDescriptor>,PanelHolder::load_panel()takesstd::shared_ptr<const PanelDescriptor>(today made byPanelListWidget), andPanelSessionkeeps aconst PanelDescriptor&. Afind_panel()returningconst PanelDescriptor*cannot feedload_panel(). Say who owns the descriptor and that it outlives the session.select_program()in §3.3 also repeatsPanelListWidget::handle_panel_clicked()(make, load JSON, load); share it.- Slicing. PR 1 is three changes: a refactor of GUI code (
ProgramSelect), new startup flags, and the host withMain.cppwiring. The catalog extraction should be its own no-behavior-change PR. Signal handling is used by the one-shot run (§3.4 step 5) but scheduled in PR 2; it belongs with the one-shot. PR 2 (LabChannelwithout a window) is the part with the least proven need, since the lab already drives the GUI app; keep it out until the one-shot runner is in use.
Minor
- Prior work is not mentioned. #206/#207 (
use_qt_ui(),SerialProgramsTests: own executable,QCoreApplication, own user folder, now onlab/main) are the headless precedent and upstream candidates. The doc should say why this is a flag inMain.cppand not a second small executable on the same pattern, which would keep the change out of a file upstream owns, and how it relates todocs/design/Agent-Integration.md. on_busy()/on_idle()are called only by the program widgets throughglobal_panel_holder(), so in headless they are never called; aSleepSuppressScopetoggle there would not fire. Hold the scope for the length of the run instead.PA_MGBA_LINKdoes not appear in the source onlab/tooling(onlyVideoSource_MgbaLink), and "mGBA Slice 3-4 runner" has no link. The PR 1 acceptance "or a mock/no-op program" names no program; name one that needs no controller, and add a unit test forHeadlessHostwith a fake panel.ActionSource::CLI"if desired": decide.LabHoldand the state file read the source, so a headless start reported asUIwould look like a user's click.
a8be72a to
8da7692
Compare
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: 8da7692
Re-review of docs/design/Headless-CC.md against the previous review (#pullrequestreview-5968597602, at a8be72a) and the code on lab/main 356aa15, which is now the PR base and the merge base. Docs only: no compile or code-health run. The PR head has no CI checks reported. The file has LF endings and no tabs.
Verdict
NEEDS_REWORK
The rework fixed most of the earlier review: base, profile requirement, settings on exit, QApplication, exit path and slicing are now right or close. Three things still do not match lab/main: the profile rule makes headless macOS-only while the doc designs for Windows and Linux, the readiness check waits on a state nothing ever sets, and the PanelCatalog ownership story does not fit how descriptors are made. Open: 1 blocking, 4 major, 7 minor.
Status of the earlier findings
| # | Earlier finding | Status |
|---|---|---|
| 1 | Wrong base | Partly. Base is lab/main and §1, §2, §3.1, §4 are written against it; Main.cpp:399, :331, :355, PanelListWidget.cpp:90, MainWindow.cpp:427 all check out. Still open: the macOS-only profile point (B1) and LabChannel placed on the wrong branch (m1). |
| 2 | Serial safety, --headless requires --profile |
Partly. Requirement, policy and profile lease are in §3.1. Open: platform scope (B1) and the per-device leases (M4). |
| 3 | --options must not be saved |
Resolved for the override itself: no save_settings(), no ProgramCategory / ProgramPanel write, no write on exit. Two remaining writers are not listed (m2). |
| 4 | Exit path, threads, signals | Resolved. ret goes to std::quick_exit, log is flushed first (stop_application_services() already calls global_file_logger().stop() before it), POSIX and Windows handlers are separate, AGENTS.md rules are named. See m4 for the stop wait. |
| 5 | QCoreApplication infeasible |
Resolved. QApplication with no window; dialog list matches the code, plus SetupSettings.cpp:85. |
| 6 | Video/audio, macOS open -n |
Resolved at design level. "Reports results via log/state files" names no file or format; say which one the runner reads (m5). |
| 7 | Exit code source | Partly. Table is there, but the source of 1 is given only for lab/tooling (M3). |
| 8 | Controller readiness | Not resolved. The chosen signal is dead code on lab/main (M1). |
| 9 | PanelCatalog ownership |
Partly. shared_ptr return and a shared load helper are stated, but the ownership model does not match the code (M2). |
| 10 | Slicing | Resolved. Catalog extraction is its own PR; signals ship with the one-shot run; LabChannel mode is deferred. |
| 11 | #206/#207 not mentioned | Resolved. |
| 12 | on_busy() / on_idle() |
Resolved, but now redundant (m3). |
| 13 | PA_MGBA_LINK, unnamed test program |
Resolved. UnitTestRunner is a real identifier; see m6. |
| 14 | ActionSource::CLI |
Resolved (decided: CLI, once merged with lab/tooling). |
Blocking
B1. --headless requires --profile, and --profile is rejected outside macOS. On lab/main, main() sets policy.profiles = REQUIRED only under __APPLE__; elsewhere apply_lab_startup_policy() returns "Settings profiles are supported only on macOS." for any plan with a profile (LabStartupPolicy.cpp), Filesystem::set_startup_profile is called only on macOS, and reserve_lab_profile() returns nullptr (LabDeviceRuntime.cpp:52). So as written, headless can never start on Linux or Windows, yet §3.4 says exit codes apply to "Linux/Windows runs" and §3.5 designs SetConsoleCtrlHandler and the quick_exit path for Windows. The earlier review raised the macOS-only profile under finding 1; the new text does not mention it. Pick one and say it: (a) PR 2 is macOS-only, --headless is rejected elsewhere, and the Windows signal design moves to "Future"; or (b) define what isolates settings and devices on Linux/Windows (profile support there is its own change) before headless is allowed. An implementer who "fixes" this by relaxing the profile rule off macOS removes the safety finding 2 asked for. Also note that apply_lab_startup_policy() returns early for any mode other than GUI, so a new HEADLESS mode must be added to that check and to LabStartupPolicy_Tests explicitly.
Major
M1. The readiness wait observes a state that is never set. Nothing on lab/main calls set_state(ProgramState::NOT_READY): m_state starts as STOPPED (ProgramSession.cpp:73) and set_state is only called with RUNNING, STOPPING and STOPPED. NOT_READY appears only in switch cases. So §3.5 step 2 passes at once with the controller still connecting, start_program() succeeds, and exit code 5 cannot happen; the run then fails later as a program error. The earlier review offered NOT_READY as one of two options; that option was wrong, and I should have checked it. Use the other one: the controller state LabChannel reports as "ready" / "none" (LabChannel.h:88, LabRequest.h:70, exercised by FakeController in LabChannel_Tests.cpp), factored so HeadlessHost can read it without the channel, with the timeout and exit 5. Say what a program with no console (UnitTestRunner, controller "") does: no wait.
M2. PanelCatalog ownership does not match how descriptors exist. PanelListDescriptor holds no descriptors: panels() calls make_panels() and returns a fresh std::vector<PanelEntry> on every call (PanelList.h), and PanelListWidget's constructor moves each unique_ptr<PanelDescriptor> out of that vector into its own shared_ptr map (PanelListWidget.cpp:69). There is nothing "owned by the catalog/list lifetime" for an aliasing shared_ptr to point at, and an aliasing pointer with no owning control block is a raw pointer with extra steps. Simpler and correct: find_panel(identifier) calls panels(), and converts the matching entry's unique_ptr into an owning std::shared_ptr<const PanelDescriptor>; the host keeps it next to the session, as §3.3 already lists. Say whether the catalog caches entries (then the GUI lists must read from the same cache, which is a behavior change for PR 1) or builds them per lookup as select_program() does today.
M3. The source of exit code 1 is not stated for the branch PR 2 lands on. lab/main has no ProgramTransition.h, ActionSource or last_transition(); start_program() takes no arguments. What it has: every error branch in ProgramSession::run_program() calls report_error(), which reaches Listener::error(message), and a normal finish does not. Define 1 as "error() received between RUNNING and STOPPED", and note the one false positive: update_historical_stats_with_current() pushes error("Unable to save stats.") after a good run. State the rule for lab/main first and the lab/tooling mapping second.
M4. --options and device opening. Two gaps on the path the earlier finding 2 was about:
- §3.1 and §3.5 step 1 apply
--optionswith a secondload_json()afterload_json_from_global(). For a console program,ConsoleProgramSession::load_json()passesConsoleSetuptom_system.load_json(), which reloads the devices;LabChannel.cpp:129-139strips that key for exactly this reason. Say whether an options file may carryConsoleSetup. If yes, merge the two JSON values and load once, so the profile's saved device is never opened when the override names another. If no, reject the key with exit4. - The doc covers only the profile lease.
lab/mainalso has per-device leases (acquire_lab_serial,acquire_lab_capture, policy fromlab-devices.json), which throwUserSetupErroron the connect path. Say what the run does then: with M1 fixed it should be a log line and exit5, not a 10 s wait and not a dialog.
Minor
- m1.
LabChannelis onlab/main(CommonFramework/Windows/LabChannel.{h,cpp},Lab.md,LabRequest.h, tests), not only onlab/toolingas §1 and §4 say. OnlyProgramTransition.h/ActionSource,VideoSource_MgbaLinkanddocs/design/Agent-Integration.mdarelab/tooling-only. LinkAgent-Integration.mdby branch, as the mGBA note is. - m2. "Writes no settings" has two exceptions to list:
ControllerSettingsTablesaves throughOptionsPanelWrapper<ConsoleSettingsPanel>().make_panel()->save_settings()when it meets a new controller (NintendoSwitch_ControllerSettings.cpp:588, from a connection thread), and every run updatesSTATS_FILE. Neither stores the--optionsoverride, so the fix is one sentence each, or a headless guard on the first. - m3.
ProgramSession::run_program()already holds aSleepSuppressScopefor the run (ProgramSession.cpp:323). Dropm_sleep_scopefromHeadlessHost. - m4. On a signal the host calls
stop_program()and waits forSTOPPEDwith no bound. Give it a timeout and say what a second signal does.130is the convention forSIGINT;SIGTERMis143. - m5. Exit codes against today's
Main.cpp: a held profile lease returns2in the GUI path (Main.cpp:402), andcheck_resource_folder()/check_hardware()return1. The table gives3for both. Fine as a headless-only choice, but say the GUI codes stay, or map them the same way. - m6. Acceptance and tests:
UnitTestRunnerdefaults to substring mode with"PokemonSwSh"and readsUNIT_TEST_RESOURCE_PATH(), so name the options file the acceptance run uses and say whether failed tests change the exit code. A fakePanelSessionis not aProgramSession;HeadlessHostneeds the samedynamic_cast<ProgramSession*>thatLabChannel.cpp:58uses, and--runon a settings panel should have a stated exit code (4). - m7.
run_program()also starts Discord (run_on_start) and the social SDK from the profile's settings; say whether headless skips them. The PR description has no base banner (PR_WORKFLOW.md "Linking").
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: 8fffda0
Final pass on docs/design/Headless-CC.md against review #pullrequestreview-5968689083 (at 8da7692) and the code on lab/main 356aa15 (PR base and merge base). Docs only: no compile or code-health run. Checks at this SHA: code-health, no-tabs, build and the three native-leases jobs pass; the six C++ build / build jobs were still pending when I read them. The file has LF endings and no tabs.
Verdict
APPROVED
LGTM: ready to build PR 1 (PanelCatalog extraction) from. All twelve open findings are resolved in the text, and the line references I checked match lab/main. Nothing blocks PR 1. Four points below affect PR 2 only; fix them in the doc when PR 2 starts, not before PR 1.
Status of the open findings
| # | Finding | Status | Checked against lab/main |
|---|---|---|---|
| B1 | Platform scope | Resolved | §3.1 and §4 make PR 2 macOS-only and reject --headless elsewhere; Windows handler and quick_exit moved to Future. The text says apply_lab_startup_policy() must stop returning early for HEADLESS and that LabStartupPolicy_Tests covers it; the early return is LabStartupPolicy.cpp if (plan.mode != StartupMode::GUI). LabDeviceRuntime.cpp:52 is right. |
| M1 | Readiness | Resolved | Uses the controller status "ready" / "none" / "" with a timeout and exit 5; no-console programs skip the wait. LabChannel.h:88 and LabRequest.h:70 are right. See P2-c for where the status is computed. |
| M2 | Catalog ownership | Resolved | Per-lookup scan of panels(), the entry's unique_ptr converted to an owning shared_ptr<const PanelDescriptor>, kept by the host next to the session, no cache in PR 1. Matches PanelList.h and ProgramSelect::select_program(). |
| M3 | Source of exit 1 |
Resolved | Listener::error() between RUNNING and STOPPED, the stats-save message excluded, lab/tooling mapping second. Matches report_error() / push_error("Unable to save stats.") in ProgramSession.cpp. See P2-a. |
| M4 | --options and device leases |
Resolved | ConsoleSetup in --options is rejected with exit 4 before any device opens (LabChannel.cpp:129-139 is right); a per-device lease UserSetupError is a log line and exit 5. See P2-d for how the host sees it. |
| m1 | LabChannel branch |
Resolved | §1 places it on lab/main; Agent-Integration.md is linked by branch. |
| m2 | Other settings writers | Resolved | NintendoSwitch_ControllerSettings.cpp:588 guarded, STATS_FILE skipped in headless. |
| m3 | m_sleep_scope |
Resolved | Dropped; ProgramSession.cpp:323 is right. |
| m4 | Stop wait after a signal | Resolved | Bounded wait, second signal exits, 130 / 143. |
| m5 | Exit codes vs Main.cpp |
Resolved | Stated as headless-only; GUI keeps 2 (Main.cpp:402) and 1. |
| m6 | Acceptance run, non-program panel | Resolved | Options file named, dynamic_cast<ProgramSession*> (LabChannel.cpp:58) with exit 4. See P2-a and P2-b. |
| m7 | Discord, base banner | Resolved | Discord and social SDK skipped; the description opens with the base. |
For PR 1 (not blocking; decide while coding)
- Shared load helper, §3.2.
PERSISTENT_SETTINGS().panels[JSON_PROGRAM_PANEL]is written with the list's display name (iter->firstinPanelListWidget::handle_panel_clicked()), and the widget also writes""for an unknown name. A helper that takes a descriptor does not have that key. Simpler than abool persist_selectionparameter: the helper does intent,make_panel(),load_json(),load_panel()and returns the error; the widget keeps its own settings write and itsQMessageBox. Then the helper has no flag and no Qt include. - A test for
find_panel(). PR 1's acceptance lists only tests that already exist. IfPanelCatalogtakes its lists in the constructor (the game factories are added by the caller, asProgramSelect::add()does today), a fakePanelListDescriptorcan test "known identifier returns a descriptor with that identifier, unknown returns null" without loading every game's icon. - The doc does not name the catalog's file. It includes every game's
PanelListFactory, so it belongs next toSource/PanelLists.{h,cpp}, and goes intoSourceFiles.cmake.
For PR 2 (fix in the doc before PR 2 starts)
- P2-a. A failed unit test does not produce exit
1.UnitTestRunner::on_test_finished()only counts failures (m_total_failed, theFAILED_TESTStext);program()returns normally afterrunner.run(), so noListener::error()fires and the run exits0. The acceptance line "1on test failure" needs either a throw fromUnitTestRunner::program()when the count is not zero (a behavior change to that program) or a different acceptance check. - P2-b.
UnitTestRunneris listed only in developer mode.NintendoSwitch_Panels.cpp:94sits insideif (STATIC_GLOBALS.DEVELOPER_MODE). Without it,--run "UnitTestRunner"is an unknown identifier and exits4. Say how the acceptance run gets developer mode in its profile. - P2-c. Where the controller status comes from. The only implementation is
SwitchLabCommands::controller_status(PanelSession&)(NintendoSwitch_LabCommands.cpp:95), behind theLabConsoleCommandsinterface inLabChannel.h. "Factored fromLabChannel.h:88" should name that function and say where the shared piece goes, soHeadlessHostinCommonFrameworkdoes not include aNintendoSwitchheader. - P2-d. How the host sees a per-device lease failure.
acquire_lab_serial()throws on the connection thread (SerialPABotBase2_Connection::open_guarded_serial_port()),acquire_lab_capture()in the camera and audio device constructors. None of these reachHeadlessHostas an exception. Name the signal it reads (a controller or video status, or a hook at the lease call); otherwise the run takes the 10 s readiness timeout the doc says it avoids. - P2-e. The example identifier
"Pokemon FireRed/LeafGreen:AutoStory"is not the identifier form. Identifiers are<Game>:<Program>as inLab.mdandStatsDatabase.cpp(PokemonBDSP:AutoStory,PokemonFRLG:LuckyEggFarmer).
Base:
lab/main(356aa15e1)Adds
docs/design/Headless-CC.mdanchored onlab/main, addressing all findings from re-review #5968689083:--headlessrejected outside macOS alongside--profile);apply_lab_startup_policy()explicitly handlesStartupMode::HEADLESS(+LabStartupPolicy_Tests). Cross-platform profile/device isolation, WindowsSetConsoleCtrlHandler, andstd::quick_exitare deferred to Future."ready"/"none"/"", factored fromLabChannelwithout requiring the channel) instead of unusedProgramState::NOT_READY; controller-free programs (UnitTestRunner) skip the wait.PanelCatalogownership): ScansPanelListDescriptor::panels()per lookup (no caching in PR 1) and converts the matchingunique_ptr<PanelDescriptor>into an owningstd::shared_ptr<const PanelDescriptor>stored byHeadlessHost.1onlab/main): Defined viaListener::error()betweenRUNNINGandSTOPPED(excludingupdate_historical_stats_with_current()'s"Unable to save stats."), withActionSource::PROGRAM_ERRORonlab/toolingnoted second.--options& per-device leases): RejectsConsoleSetupin--optionswith exit4; per-device leaseUserSetupErrorfails fast with log + exit5.LabChannelbranch note &Agent-Integration.mdlink, guardsControllerSettingsTableand skipsSTATS_FILE, drops redundantm_sleep_scope, bounds signal stop wait (130/143), clarifies GUI exit codes &open -nlog/state reporting, specifiesdynamic_cast<ProgramSession*>+UnitTestRunneracceptance options, and skips Discord/social SDK.