Repository navigation
fix: Respect variance, unions and Any when matching devices to plans - #2
DiamondJoseph wants to merge 2 commits into
Conversation
Device type arguments were compared with issubclass(declared, annotated) whatever variance the annotated type declares. bluesky declares Movable as contravariant, so a Movable[int] parameter accepted a Movable[bool] device and rejected a Movable[object] one. - Compare each type argument using the variance of its type parameter - Compare unions member by member instead of raising TypeError - Treat Any in an annotation as "must work with everything", so that Movable[Any] accepts only a device that can be set to anything - Say what a rejected device declares in the error message - Name unions in messages and plan schemas instead of raising AttributeError - Offer in the schema of a plan only the devices its parameter accepts Document the rules in docs/explanations/type_validators.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Assisted-by: Claude Fable 5.2 via Claude Code
DeviceModel.from_device used the __name__ of each type argument. A union such as int | str does not have one, so a single device declaring a union made GET /devices fail. Name each member of a union, and fall back to the string form of anything else without a name. Test GET /devices and GET /plans against devices and plans loaded from a module in the same way as the server loads them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Assisted-by: Claude Fable 5.2 via Claude Code
|
|
||
| `Movable` with no type arguments accepts any `Movable` device. | ||
|
|
||
| `Any` is not a way to switch the check off. `Movable[Any]` says that the plan may set the device to anything, so it accepts only a device that can be set to anything: one declared as `Movable[Any]` or `Movable[object]`, or one that declares no type argument. For a type that only produces values the opposite holds, so `Readable[Any]` accepts any `Readable` device. |
There was a problem hiding this comment.
Anyis not a way to switch the check off.Movable[Any]says that the plan may set the device to anything, so it accepts only a device that can be set to anything: one declared asMovable[Any]orMovable[object], or one that declares no type argument
Is this true? I was under the impression this was how it worked (see 2116 in bluesky) but I don't think it does any more. Passing a Movable[int] to a function that accepts a Movable[Any] will type check as valid but fail at runtime.
There was a problem hiding this comment.
Definitely with the changes, Movable[int] into Movable[Any] should not go: your function could have
def f(m: Movable[Any]) -> MsgGenerator:
m.set(1)
m.set("banana")
m.set({})There was a problem hiding this comment.
But run that through a type checker and it passes?
There was a problem hiding this comment.
It passes type checking because Any returns True for any check, but for purposes of a typevar in the method signature, it shouldn't and blueapi should be more strict than that. If you want Any Movable, the type should be Movable. Movable[object] or Movable[Any] is only for a Movable that can be set to anything.
| Whether a value of type sub can be used where one of type sup is expected, as | ||
| far as that can be told at run time. | ||
|
|
||
| Any and type variables are compatible with everything, and everything can be |
There was a problem hiding this comment.
type variables are compatible with everything
If we ignore bounds and constraints?
There was a problem hiding this comment.
Lemme add a test and find out
Fixes how blueapi decides whether a device satisfies a plan parameter annotated with a parameterised device type such as
Movable[int], and fixesGET /devicesandGET /plansfor the same cases. There is no linked issue. This targets the fork for review and has not been proposed upstream.Why
is_compatible_argsincore/context.pycompared every type argument withissubclass(declared, annotated), whatever variance the annotated type declares. bluesky declaresMovable(andCheckable) as contravariant in the value it is given, so for those the check was backwards: aMovable[int]parameter accepted aMovable[bool]device and rejected aMovable[object]one.Fixing that exposed four related defects on the same path:
TypeErrororAttributeErrorinstead of being compared.Anyin an annotation was compared as if it were an ordinary class.GET /devicesfail for every device.GET /plansfail for every plan.Behaviour before and after
Measured, not inferred, by loading
tests/unit_tests/core/fake_generic_module.pythrough config sources on each side. Rows are what the device's class declares. Columns are the annotation of the plan parameter.Upstream
main(1e761f9)m: Movablem: Movable[Any]m: Movable[int]m: Movable[int] | Movable[str]m: Movable[int | str]Movable[Any]Movable[object]AttributeErrorMovable[int | str]TypeErrorTypeErrorTypeErrorMovable[int]Movable[str]Movable[bool]With that module loaded,
GET /devicesandGET /plansboth raisedAttributeError: 'types.UnionType' object has no attribute '__name__'.This branch
m: Movablem: Movable[Any]m: Movable[int]m: Movable[int] | Movable[str]m: Movable[int | str]Movable[Any]Movable[object]Movable[int | str]Movable[int]Movable[str]Movable[bool]Both endpoints return normally.
This can reject requests that used to be accepted
Reviewers should decide whether this needs
fix!:. The cells that went from accept to reject are:Movable[bool]for aMovable[int]parameter, including throughMovable[int] | Movable[str]. The realistic form is a plan annotatedMovable[SomeEnumBase]given a device that only takes one enum subclass.Movable[int],Movable[str]orMovable[bool]for aMovable[int | str]parameter.The cells that went from reject or error to accept are:
Movable[object]for everyMovable[...]parameter. The realistic form is aMovable[str]device for a plan annotatedMovable[SomeStrEnum].Movable[int | str], forMovable[int]and forMovable[int] | Movable[str].The rules as implemented
__parameters__of the annotated type.annotatedmust be a subtype ofdeclared.declaredmust be the same as, or a subtype of,annotated. This is the old rule.Anymeans different things on each side.object.Movable[Any]accepts only a device that can be set to anything.Readable[Any]accepts anyReadable.issubclassare compatible only if equal.list[int]satisfieslist[int]and notlist[bool]. This used to raiseTypeError.Movable[int] | Movable[str]is a union of two annotations, validated by pydantic one member at a time, so each member follows the rules above.Error message
When the device is the right kind but declares the wrong type arguments, the message now says what it declares:
On
mainthis case was accepted. Other mismatches involving a union reportedissubclass() arg 1 must be a class, or escaped asAttributeError. For a union of annotations pydantic reports one such error per member.Endpoints
GET /devices:ProtocolInfo.typesusedarg.__name__. A union has none on Python 3.11 to 3.13, so one device declaringMovable[int | None]broke the listing. On 3.14 it has one, which is justUnion. Unions are now named member by member, for example"int | str". Anything else without a name falls back tostr().GET /plans, two changes:typesas"int | str".enumof device names for a parameter used to list every device of the right kind, ignoring type arguments, so it offered devices the plan would then reject. It now lists only devices the parameter accepts. A test asserts that the offered set equals the accepted set for every plan.REST_API_VERSIONis not bumped. Regenerating it produces no diff.Edge cases and decisions
Each of these was checked by running it, unless it says otherwise.
Readable,ConfigurableandSubscribableinvariant, althoughReadableonly produces values. Strict invariance would rejectReadable[bool]forReadable[int], which is harmless, so the old leniency is kept and a test pins it.Movable[object]is accepted forMovable[Any]. A device declared that way can be set to anything, and it already satisfiesMovable[int | str]. This goes slightly beyond "onlyMovable[Any]orMovable". It is one condition to remove if unwanted.Readable[Any]now accepts anyReadable. Onmainit rejected a device declaringReadable[bool], by the same rule that rejected theMovable[Any]cases. That exact case was not run onmain.__class_getitem__. The extra arguments are compared covariantly, as before, so nothing is silently skipped.Converter[T_contra, T_co]is checked contravariantly in the first argument and covariantly in the second.class LaterMover(Movable["Later"]):Movableand rejected byMovable[int]andMovable[Any], because they cannot be compared.ForwardRef('Later')in the error message and inGET /devices. That is accurate but not pretty.str()fallback. Without it, building the new error message would itself raiseAttributeError.int | stris atypes.UnionType, while a union of two parameterised types such asMovable[int] | Movable[str]is atypingunion. On 3.14 both aretyping.Union.is_unionrecognises all of them. The matching and naming helpers were run on 3.11, 3.12, 3.13 and 3.14. The full suite was run on 3.13 only.is_compatible_argsno longer raises for arguments that are not classes. It returnsFalse, which becomes the normal "is not of type" rejection.GET /devices. A device with an unbound type argument still reports the variable's name, andAnyreports"Any".Known limitations that this does not change
SimMotorand aSignalRW[int]both declare nothing forMovabledirectly.Movable[str]parameter accepts both of them.class Shutter(StandardReadable, Movable[ShutterState]).Movable[float]whereMovable[int]is asked for, because anintis acceptable as afloat.issubclass(int, float)is false, so blueapi rejects it, as it did onmain.class Foo[T], Python 3.12 or later) have inferred variance, which is not visible on the type parameter. Their arguments are compared covariantly.Movable[list[int]]is shown asMovable[list], as before.Changes by file
src/blueapi/core/context.pyis_compatible_argsapplies the rules above, using new private helpers_contravariant_parametersand_is_subtype.is_union, shared withservice/model.py.qualified_namenames unions and falls back tostr()for anything without a name.enumusesis_compatible.src/blueapi/service/model.py:_type_namereplacesarg.__name__.docs/explanations/type_validators.md: new section "Type Arguments of Devices", covering variance,Any, the two union forms, the error message and the schema.tests/unit_tests/core/fake_generic_module.py: seven ophyd-async devices built by a dodalDeviceManager, and five plans.tests/unit_tests/core/test_context.py,tests/unit_tests/service/test_rest_api.py, newtests/unit_tests/service/test_model.py.Testing
126 test cases are added. Before this work
is_compatible_argshad no direct tests.is_compatible_argswith stand-in generic classes: covariant, contravariant and invariant parameters against a subtype, the same type, a supertype and an unrelated type, plus unions,Anyon each side, mixed variance, uncomparable types and missing type arguments.Movablestructurally: each annotation against each device, including the exact error text.DeviceManagerSourceandPlanSource: all 35 plan and device pairs, and a check that each plan's schema offers exactly the devices it accepts.GET /devicesandGET /plans, asserting the full type and device lists.Run locally on Python 3.13:
helm.context.pyandmodel.pyis executed.Not run: the helm tests, the system tests, and the helm and lock-file pre-commit hooks.
Related
AGENTS.mdwhose generics section, and whose description, describe the old behaviour ofis_compatible_args. Both need updating if this is merged upstream.type_validators.mdand notplans.md, to avoid conflicting with docs: Update docs to match code DiamondLightSource/blueapi#1773, which rewritesplans.md. Its two-line example is an inline code block like the rest of that page, not a tested example file.Instructions to reviewer on how to test:
pytest tests/unit_tests/core/test_context.py tests/unit_tests/service/test_rest_api.py tests/unit_tests/service/test_model.py.ACCEPTED_MOVERSintests/unit_tests/core/test_context.py, which is what the loaded-device tests assert.tox -e docsand read "Type Arguments of Devices" inexplanations/type_validators.html.Movable[object]forMovable[Any], and the narrowerenumin plan schemas.Checks for reviewer
🤖 Generated with Claude Code
Assisted-by: Claude Fable 5.2 via Claude Code