Repository navigation
CI: stop display container before switching driver modes - #3184
jpascucci-nv wants to merge 4 commits into
Conversation
Closes NVIDIA#3183 Stop NVDisplay.ContainerLocalSystem before invoking nvidia-smi -fdm and wait for every NVDisplay.Container process to exit. After a driver install, the display container can crash-loop while the GPU is cycled. If its teardown overlaps PnP removal, nvlddmkm can remain loaded and nvidia-smi may never recover. Bound the process drain at 60 seconds and report remaining PIDs on failure. Preserve the existing mode-specific logging and device-cycle stabilization behavior.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Summary
Priority: ➖ Normal Change: Bug fix · Severity of issue fixed: Medium Merge Risk: ⚪ Minimal · up to
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
- Review profile: CHILL
- Plan: Enterprise
- Run ID:
81b6d0cc-b448-46d9-87d9-b15a3b5a79cc
📒 Files selected for processing: 1
ci/tools/configure_driver_mode.ps1
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
|
rwgk
left a comment
rwgk
left a comment
There was a problem hiding this comment.
codex GPT-6.1-Sol ultra:
- [P2] A queued service recovery restart can reintroduce the race — configure_driver_mode.ps1:29–30. After a crash, the service can report
Stoppedwith no processes while Windows waits to restart it. The helper then returns immediately, allowing the container to restart during device cycling. Stopping the service does not cancel queued recovery; Microsoft documents that disabling it prevents the restart. Temporarily disable the service through the cycle, then restore its previous configuration.
| # This assumes we have the prior knowledge on which GPU can use which mode. | ||
| $driver_mode = $env:DRIVER_MODE | ||
| if ($driver_mode -eq "WDDM") { | ||
| Write-Output "Setting driver mode to WDDM..." |
There was a problem hiding this comment.
The Write-Output now appears before running Stop-DisplayContainer, maybe that'll be confusing?
How about:
Remove the three Write-Output here.
Conveniently, you have the $driver_mode variable already.
So below (after running Stop-DisplayContainer):
Write-Output "Setting driver mode to $driver_mode..."
nvidia-smi -fdm $driver_mode_code
While at it, maybe rename the $driver_mode variable to $driver_mode_name.
Temporarily disable NVDisplay.ContainerLocalSystem while changing driver modes so queued recovery actions cannot restart it during the device cycle. Handle in-flight starts, restore the original startup type in a finally block, and report the requested driver mode by name.
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
- Review profile: CHILL
- Plan: Enterprise
- Run ID:
49b77a5d-5a22-4942-b124-3e8b3e0da34a
📒 Files selected for processing: 1
ci/tools/configure_driver_mode.ps1
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
rwgk
left a comment
rwgk
left a comment
There was a problem hiding this comment.
codex GPT-6.1-Sol ultra:
No new findings in 12a6d75. The earlier recovery-restart concern is addressed: the service stays disabled through device cycling and NVML recovery, with its startup type restored in finally.
PowerShell parsing and mocked success/failure checks passed, including restoration after service-stop timeout, mode-setting failure, empty device discovery, and NVML recovery failure.
This branch has not been deployed
2 participants
Description
Closes #3183
Stop NVDisplay.ContainerLocalSystem before invoking nvidia-smi -fdm and wait for every NVDisplay.Container process to exit.
After a driver install, the display container can crash-loop while the GPU is cycled. If its teardown overlaps PnP removal, nvlddmkm can remain loaded and nvidia-smi may never recover. Bound the process drain at 60 seconds and report remaining PIDs on failure.
Preserve the existing mode-specific logging and device-cycle stabilization behavior.
Checklist