Skip to content

fix(adapter): start on first use and report which adapter answered - #21

Merged
TheStreamCode merged 2 commits into
mainfrom
fix/adapter-health-and-lazy-start
Nov 17, 2026
Merged

TheStreamCode merged 2 commits into
mainfrom
fix/adapter-health-and-lazy-start

Conversation

@TheStreamCode

Copy link
Copy Markdown
Owner

Why

Follow-up to #20: on Hermes 0.22.5 the in-process adapter binds, credential resolution returns the loopback base_url, and inference still fails with FailoverReason.auth while the same adapter run standalone works. The report also shows two listeners (43219 and 43220) during one failing run and an orphaned LISTEN on 43219 after the process exits.

Reading the code, three things make that state possible and unobservable:

  1. _adapter_base_url() ran at import (base_url=_adapter_base_url() in the profile constructor). Any process that loads the plugin without inference — the admission probe that register()'s docstring mentions, hermes plugins validate, a linter — binds a port and then exits.
  2. start_adapter() is idempotent per process only, and the state lives on a sys attribute. A foreign listener on the first candidate port pushes the next process to the following port, so the resolved base_url differs between processes. The module comment on DEFAULT_PORTS ("so a persisted base URL stays stable across restarts") assumes the opposite.
  3. The adapter relays the upstream status and body verbatim, but logged only at debug and only when a rule fired. A refusal therefore reaches the user as a generic authentication failed (on hosts older than 0.23 a 403 client_attestation_denied is not classified separately) with no record of which adapter answered.

Reproduced with the plugin loaded as a package, exactly as a directory plugin is:

process A (imports the plugin): base_url=http://127.0.0.1:43219/v1
process B (same, while A is alive): base_url=http://127.0.0.1:43220/v1

After this change the same harness reports no listener after import and no listener after register(), and the loopback URL appears only at the first resolution.

What changed

  • The adapter starts on first use: runtime_base_url() and fetch_models() resolve it, register() no longer starts one. A probe process binds nothing, and each process owns the adapter it reports.
  • GET /__muse_adapter/health returns adapter_version, pid, port, passthrough, upstream. python muse_adapter.py --health http://127.0.0.1:43219/v1 queries it and exits non-zero when nothing answers, which is how a stale or foreign port becomes visible.
  • Every relayed response carries x-muse-adapter: <ADAPTER_VERSION>, so a captured response says whether the loopback hop happened at all.
  • A response with status 400 or above logs one warning line: method, path, upstream status, adapter version, port, pid and the rules that fired. No body, no credential.
  • ADAPTER_VERSION stays at 3: no wire rule changed, so the contract and the sibling repositories are unaffected.

Tests

python -m pytest -q → 59 passed (53 before, 6 added):

  • health answers with version, pid, port, passthrough and upstream;
  • a foreign listener on the first candidate port is not adopted, and the live adapter identifies itself;
  • every relayed response carries x-muse-adapter;
  • a 403 logs exactly one line with status, version, port and pid, and neither the body nor the credential appears;
  • the --health CLI reports a listening adapter and a dead port;
  • loading and registering no longer bind, and resolution does.

ruff check . and python .github/scripts/validate-plugin.py --path . pass.

What this does not prove

The field failure itself is not reproducible here: there is no 0.22.5 install and no subscription key in this environment. Two causes remain possible and the new signals separate them at the next run:

  • a stale or foreign adapter served the request — the warning line names its version/port/pid, and health on that port says whether it is this process' adapter;
  • the request bypassed the adapter — no warning line at any port and no x-muse-adapter header, which would make it the v0.6.0: loopback wire adapter never receives traffic — runtime_base_url not overridden #12 shape again (note the model.base_url override already documented in the README as a bypass).

#20 also asks for the failing status/body, which this does not expose: the adapter already relays them, and classify_api_error is what turns a 403 into a readable reason on 0.23.

Refs #20

The adapter was started at import, so every process that merely loaded the
plugin (the admission probe, `hermes plugins validate`, a linter) bound a
loopback port and then exited, leaving the listener behind. The next process
could not reuse that port, bound the following one, and every process reported
a different base URL; a loopback URL captured from one process pointed at an
adapter that another process owned.

Start the adapter on first use instead: `runtime_base_url()` and `fetch_models()`
resolve it, `register()` no longer starts one. Each process now owns exactly the
adapter it reports.

A refused request is also invisible today: Meta's 403 reaches the caller as a
generic "authentication failed" and the adapter only logged at debug when a
rule fired. Add the pieces that make the seam diagnosable:

- `GET /__muse_adapter/health` returns adapter_version, pid, port, passthrough
  and upstream, and `python muse_adapter.py --health <url>` queries it, so a
  port held by another process or by an older plugin is distinguishable;
- every relayed response carries `x-muse-adapter: <version>`;
- a response with status 400 or above logs one warning line with method, path,
  status, adapter version, port, pid and the rules that fired, never a body or
  a credential.

ADAPTER_VERSION stays at 3: no wire rule changed.

Refs #20
Review pass on #21:

- `fetch_models` no longer starts the adapter. Discovery is a GET, which the
  adapter relays untouched, and Meta serves `/v1/models` directly with a
  subscription key, so binding a port for it only adds a socket to a short-lived
  process. It also avoided a subtle change: passing `base_url=... or self.base_url`
  would have made core's own base URL look "custom" to the base implementation
  once the profile base URL moved to the loopback URL.
- The "import does not bind" assertion is now order-independent: it records the
  active adapter before loading the plugin and requires it to be unchanged,
  instead of assuming no adapter is running in the test process.
- `GET /__muse_adapter/health` tolerates a caller that hangs up mid-response,
  like the relay path already does.

Refs #20
@TheStreamCode
TheStreamCode merged commit 3c60c0d into main Nov 17, 2026
5 checks passed
@TheStreamCode
TheStreamCode deleted the fix/adapter-health-and-lazy-start branch November 17, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant