Skip to content

fix(sfcompute): keep starting nodes listed as pending on SSH 404 - #163

Draft
mmoulikk wants to merge 1 commit into
mainfrom
BREV-13508/keep-starting-sfcompute-nodes-listed
Draft

mmoulikk wants to merge 1 commit into
mainfrom
BREV-13508/keep-starting-sfcompute-nodes-listed

Conversation

@mmoulikk

@mmoulikk mmoulikk commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Issue

While a new SF Compute node is starting, SF Compute already lists it as running, but for a few seconds GET /integrations/brev/v1/instances/{id}/ssh answers 404 ("SSH details not currently available"). ListInstances leaves the node out of the list, and dev-plane reads a node missing from the list as terminated. The environment moves to instance_lost, and when dev-plane handles that change twice, on to terminated, which is final. Setup never runs, the console hides Delete, and the VM keeps running and billing. This hit 9 SF Compute environments since Oct 31 (BREV-13508).

Root cause

getSSHInfo returns every SSH lookup error to its caller. ListInstances skips any node whose lookup fails ("skipping instance due to SSH error"), and GetInstance returns the error. A 404 right after creation means the SSH details aren't ready yet, not that the node is gone, but the code treats both the same.

Changes

  • v1/providers/sfcomputev2/instance.go:
    • getSSHInfo now takes the instanceResponse. If the SSH lookup returns 404 and the node was created less than sshDetailsPendingWindow (45 min) ago, it logs sfcv2: SSH details not available yet, reporting instance as pending and returns no SSH details instead of an error. The node stays in ListInstances, and GetInstance stops failing.
    • sfcInstanceToBrevInstance reports a running node with no SSH hostname as pending, since Brev can't reach it yet.
    • After 45 minutes, and for any other error (429, 5xx, connection, bad JSON), nothing changes: the node is skipped as before. 45 minutes outlasts dev-plane's SSH wait during the build (up to 3 × 10 min), so the build's own timeout still catches a node whose SSH details never appear.
  • v1/providers/sfcomputev2/instance_test.go: three new tests.
  • One behavior change beyond the 404: a running node whose SSH answer has no hostname is now pending at any age (before: running with no address).
  • dev-plane only needs a github.com/brevdev/cloud bump. There, pending doesn't move an environment (only a change to terminated does), the poll rate doesn't depend on status, and time in pending isn't billed (a few seconds per new node).

Test

  • TestIsSSHDetailsPending: a 404 inside the window counts as pending; a 404 at or after the window, a 500, or a transport error doesn't.
  • TestListInstancesKeepsNewInstanceWithoutSSHDetailsAsPending (fake SF Compute API): a new node with an SSH 404 is listed as pending with no IP, a ready node is running with its SSH address, and a node older than the window with an SSH 404 is still left out.
  • TestGetInstanceReportsNewInstanceWithoutSSHDetailsAsPending: GetInstance returns pending instead of an error for a new node with an SSH 404.
  • go test ./v1/providers/sfcomputev2/ -race -count=1 passes, and golangci-lint v2.6.0 (CI's version, run with Go 1.25.10) reports 0 issues.
  • Dev and staging have no SF Compute credentials, so this was checked locally against a fake SF Compute API; the real-API tests in validation_test.go were not run.

@mmoulikk mmoulikk self-assigned this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant