Skip to content

Add typed_handler decorator to validate recipe messages - #213

Open
ndevenish wants to merge 1 commit into
queue-subscribefrom
typed-handlers
Open

ndevenish wants to merge 1 commit into
queue-subscribefrom
typed-handlers

Conversation

@ndevenish

Copy link
Copy Markdown
Contributor

Adds workflows.recipe.typed.typed_handler, a decorator for recipe callbacks that validates incoming messages and recipe parameters with pydantic before the callback runs. This is so that we can replace the hand-written checks that services currently repeat, eg. isinstance(message, dict), parameters.get(...) with defaults, and building pydantic models by hand inside the callback.

class Parameters(pydantic.BaseModel):
    output_filename: Annotated[Path, NonEmpty]
    include: set[str] = set()

class JSONLines(CommonService):
    def initializing(self):
        self.subscribe("jsonlines", self.receive_msg)

    @typed_handler
    def receive_msg(self, rw, header, message: dict[str, Any], parameters: Parameters):
        ...
  • Only subscription methods with the @typed_handler are passed, so nothing changes by default
  • Only subscription methods with the @typed_handler accept the parameters object (which is pulled out of the RecipeWrapper) so they cannot be accidentally passed to CommonService.subscribe
  • Annotations can be pydantic models or anything a pydantic.TypeAdapter accepts.
  • If the callback declares a parameters argument, the recipe parameters are assembled, validated against its annotation, and passed in as a keyword argument.

The decorator has the following signature:

def typed_handler(
    func: Callable[..., Any] | None = None,
    *,
    on_invalid: Literal["nack", "drop"] = "nack",
    parameters_from: Sequence[str] | None = None,
    substitute: bool = False,
) -> Any:

Where

  • on_invalid defines the behaviour when a message fails parsing.
  • substitute=True replaces $name / ${name} in parameter string values with values from the recipe environment (and $$ with $). It is off by default.
  • and parameters_from handles merging of parameters from multiple dictionaries/locations

Where parameters come from

By default parameters is built only from the current recipe step's parameters. Services that let incoming messages supply or override parameters say so explicitly with parameters_from, listing sources highest precedence first:

@typed_handler(parameters_from=("message", "step"))                    # any message key overrides the step
@typed_handler(parameters_from=("message.parameters", "step"))         # only the message's "parameters" dict
@typed_handler(parameters_from=("step.ispyb_parameters", "message", "step"))

Sources are "step" or "message", optionally followed by .key to select a nested dictionary. Merging is shallow, a key present in a higher source wins even if its value is falsy, and missing or non-dictionary sources are skipped. These cover the precedence orders currently implemented by hand (mostly with ChainMapWithReplacement) in the dlstbx services. Invalid sources are rejected when the module is imported.

Also

  • Add NonEmpty validator, for fields such as Annotated[Path, NonEmpty]. A plain Path silently turns "" into Path("."), and min_length constraints can't be applied to a Path.
  • pydantic (>=2) is added as a dependency. It is only imported by workflows.recipe.typed.

Service callbacks can annotate their message, and an optional
parameters argument, with pydantic models. By default the parameters
come from the recipe step only; parameters_from lets a service merge
in values from the message, or nested dictionaries such as
message.parameters, in an explicit order of precedence. Services can
opt in to substituting values from the recipe environment. Messages
failing validation are logged and nacked without requeueing, or
optionally dropped.

CommonService.subscribe now rejects callbacks that can not be called
with (rw, header, message), and points callbacks declaring a
parameters argument at typed_handler, instead of failing when the
first message arrives.

Also adds a NonEmpty validator, for fields such as
Annotated[Path, NonEmpty]. pydantic is added as a dependency.
@ndevenish
ndevenish added this pull request to stack #214 October 26, 2026 15:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant