[Feature Request] Raise explanatory exceptions when confronted with unusable unsubscripted multiprocessing.(Simple)?Queue "type hints"  #397

Activity

1. leycec commented on Jun 14, 2024

   

   leycec

   on Jun 14, 2024

   Member

   More actions

   ...heh. More non-standard mostly undocumented type hints, huh? No problem. @beartype chews up non-standard mostly undocumented type hints for breakfast. It turns out everyone else is confused about multiprocessing.Queue[...] type hints, too. Indeed, it looks like this probably effects a constellation of interrelated non-standard mostly undocumented type hints that scare everybody – including:

   • multiprocessing.Queue[...] type hints.
   • multiprocessing.SimpleQueue[...] type hints.
   • queue.Queue[...] type hints.
   • queue.SimpleQueue[...] type hints.

   It seems likely that the multiprocessing variants are "simple" aliases of their corresponding original queue types. Moreover, if I'm reading the official queue documentation correctly, then @beartype has no means of deeply type-checking queue types.

   Kinda unfortunate, but what can you do? The most @beartype can do here is shallowly type-check that queue types are of the expected classes; @beartype can't actually peek inside the items queued in a queue type without destroying that queue. This is probably by design to ensure that queues are safeguarded against multithreading race conditions. Still...

   Kinda unfortunate. Let's do our best!

   @beartype: We do our best even when that doesn't amount to much.

   👍React with 👍1Reacted by Moosems
2. leycec

   changed the title [-]Additional context seems to break beartype in certain cases[/-] [+][Feature Request] `(multiprocessing|queue).(Simple)?Queue[...]` type hint support[/+] on Jun 14, 2024
3. leycec commented on Jun 15, 2024

   

   leycec

   on Jun 15, 2024

   Last edited by leycec

   Member

   More actions

   Hoh, boy. I just spelunked into this. I have good news and I have bad news.

   But first... the tl;dr! Ain't nobody got the time to read all the madness below. So, this is what you want to do:

   from beartype import beartype
   from multiprocessing import Queue
   from multiprocessing.queues import Queue as GenericQueue
   from typing import TypedDict
   
   class SpecialDict(TypedDict):
       pass
   
   class Example:
       @beartype
       def __init__(self, list1: GenericQueue) -> None:
           self.list1: GenericQueue[SpecialDict] = list1
   
   q1: GenericQueue[SpecialDict] = Queue()
   Example(q1)

   ...which behaves as expected under the current stable @beartype 0.18.5 release. Indeed, the above is the only usable way of typing what you want to type. The initial example you gave is improperly typed – but it's not your fault. Why? Because...

   Tell @Moosems What He's Won, Folks! ...yeah, it's bad news.

   The bad news is that the multiprocessing.Queue() and multiprocessing.SimpleQueue() attributes are balls-crazy methods. Even though the official documentation states that both are classes, the official documentation lies. They're not classes. They have nothing to do with classes. They're standard pure-Python bound methods. Clearly, methods:

   • Are not valid type hints.
   • Cannot be subscripted.

   Sadly, this means that @beartype is actually right about everything. From the perspective of runtime type-checking (...so, @beartype), PEP-compliance means "valid as type hints." Since multiprocessing.Queue() and multiprocessing.SimpleQueue() are invalid as type hints, they're PEP-noncompliant.

   Moreover, since neither multiprocessing.Queue() nor multiprocessing.SimpleQueue() are subscriptable, no code anywhere should ever try doing stuff like multiprocessing.Queue[SpecialDict]. Yeah, sure, technically that "works." ^{...for certain specious definitions of "works."} Pragmatically, though, that really doesn't work at all. Trying to subscript something that can't be subscripted is syntactically invalid code. Right? Those "type hints" don't actually run at runtime; they raise exceptions at runtime.

   If you were to automatically apply the @beartype decorator to the above code with an import hook like beartype.claw.beartype_this_package(), you'd immediately receive the same fatal exception as you've exhibited above:

   TypeError: 'method' object is not subscriptable

   Why? Because @beartype import hooks necessarily evaluate type hints annotating global variable assignments (e.g., the Queue[SpecialDict] in q1: Queue[SpecialDict] = Queue()) at module importation runtime. Since @beartype import hooks are really the standard mode of operation for @beartype in 2024, this means that the module you've defined above is basically unimportable by most @beartype users in 2024.

   But Wait! There's Good News.

   The good news is twofold:

   1. Never use either the multiprocessing.Queue() or multiprocessing.SimpleQueue() methods as type hints. Not even once. They're busted. Friends never let friends use busted.
   2. Instead, only use the equivalent multiprocessing.queues.Queue and multiprocessing.queues.SimpleQueue types as type hints. Unlike multiprocessing.Queue() and multiprocessing.SimpleQueue(), multiprocessing.queues.Queue and multiprocessing.queues.SimpleQueue are both valid type hints and subscriptable in the standard way.

   Consider refactoring your initial example for the good of well-typed APIs and your long-suffering coworker who is now ripping out his last hair:

   from beartype import beartype
   from multiprocessing import Queue
   from multiprocessing.queues import Queue as GenericQueue
   from typing import TypedDict
   
   class SpecialDict(TypedDict):
       pass
   
   class Example:
       @beartype
       def __init__(self, list1: GenericQueue) -> None:
           self.list1: GenericQueue[SpecialDict] = list1
   
   q1: GenericQueue[SpecialDict] = Queue()
   Example(q1)

   Flex it, polar @beartype. 💪 🐻‍❄️

   But Wait! There Is Something @beartype Should Do Here.

   @beartype can't fix the fact that multiprocessing.Queue() and multiprocessing.SimpleQueue() are balls-crazy methods (rather than the balls-normal types they should be). That's CPython's fault, unfortunately. They should have resolved this obvious issue decades ago. They didn't. My emoji weeps. 😭

   @beartype can fix the exception it raises when confronted with either multiprocessing.Queue() or multiprocessing.SimpleQueue() as type hints. The current exception engenders confusion. Instead, @beartype should clarify this madness by explicitly informing funny guys with great personalities who are amazing devs such as yourself, @Moosems, that only multiprocessing.queues.Queue and multiprocessing.queues.SimpleQueue should be used as type hints.

   Let's edit this feature request title for great justice.

   ❤️React with ❤️1Reacted by Moosems
4. leycec

   changed the title [-][Feature Request] `(multiprocessing|queue).(Simple)?Queue[...]` type hint support[/-] [+][Feature Request] Raise explanatory exceptions when confronted with unusable `multiprocessing.(Simple)?Queue[...]` "type hints" :rage4: [/+] on Jun 15, 2024
5. leycec

   changed the title [-][Feature Request] Raise explanatory exceptions when confronted with unusable `multiprocessing.(Simple)?Queue[...]` "type hints" :rage4: [/-] [+][Feature Request] Raise explanatory exceptions when confronted with unusable unsubscripted `multiprocessing.(Simple)?Queue` "type hints" :rage4: [/+] on Jun 15, 2024
6. Moosems commented on Jun 15, 2024

   

   Moosems

   on Jun 15, 2024

   Last edited by Moosems

   Author

   More actions

   @leycec Thank you so so much for looking into that! Despite your claim "Ain't nobody got the time to read all the madness below" I actually found your explanation to be quite interesting and even a very enjoyable read 😄.
7. leycec commented on Jun 18, 2024

   

   leycec

   on Jun 18, 2024

   Last edited by leycec

   Member

   More actions

   ...heh. multiprocessing.Queue internals, huh? Now we're really getting into the guts of the beast. To quote core CPython dev @JelleZijlstra at CPython PR python/cpython#99511:

   multiprocessing scares me...

   Preach, brother @JelleZijlstra. multiprocessing scares us all.

   The First Rule of Type Subscriptability Is...

   Wait. What the heck is "type subscriptability," anyway? Okay. Let's back up.

   Back when we were all younger and had muscles and smiled blissfully (albeit ignorantly), Python 3.9 implemented PEP 585. PEP 585 allowed core types (like list and dict) to be subscripted with child type hints (like the typing.Union[int, str] child type hint subscripting list[typing.Union[int, str]]). That's type subscriptability. So far so good, right?

   Clearly, not all types are subscriptable. Only types explicitly designed to be subscripted are subscriptable. The usual way this happens is with a crazy one-liner like __class_getitem__ = classmethod(types.GenericAlias). It's best not to ask questions here. But the point I'm trying but failing to make is that type subscriptability doesn't happen automatically. It's hard, keyboard-breaking work. Type subscriptability only happens when someone who is not us makes it happen. So far so good, right?

   Clearly, this implies that most types are not subscriptable. Although the general expectation is that more and more standard CPython types will become subscriptable over time, this is a gradual and frankly boring process that takes years and a great deal of pounding sand over CPython PRs.

   In this case, the multiprocessing.queues.Queue class only became subscriptable with Python 3.12. This is why your GitHub Actions-based continuous integration workflow is failing. That workflow is running Python 3.11, which lacks support for subscripting multiprocessing.queues.Queue. Consider this output that is beyond boring:

   $ python3.12
   >>> from multiprocessing.queues import Queue as GenericQueue
   >>> print(GenericQueue[str])
   multiprocessing.queues.Queue[str]  # <-- good. this is good, albeit boring.
   
   $ python3.12
   >>> from multiprocessing.queues import Queue as GenericQueue
   >>> print(GenericQueue[str])
   TypeError: type 'Queue' is not subscriptable  # <-- OMFG!!!!

   You Have Two Equally Awful Solutions Here

   1. Solution 1: Drop support for Python < 3.12. Obviously, nobody's doing that. Next!
   2. Solution 2: Conditionally support Python < 3.12 with a type alias resembling:

   from sys import version
   from multiprocessing.queues import Queue as GenericQueue
   from typing import TYPE_CHECKING
   
   class SpecialDict(TypedDict):
       pass
   
   # If either static type-checking *OR* this is CPython >= 3.12,
   # subscript "GenericQueue" because we can. Just do it already!
   if TYPE_CHECKING:
       GenericQueueOfSpecialDicts = GenericQueue[SpecialDict]
   # Else, this is CPython < 3.12. We are now in the No Man's Land
   # of Typing. In this case, avoid subscripting "GenericQueue". Ugh.
   else:
       GenericQueueOfSpecialDicts = GenericQueue
   
   class Example:
       @beartype
       def __init__(self, list1: GenericQueue) -> None:
           self.list1: GenericQueueOfSpecialDicts = list1
   
   q1: GenericQueueOfSpecialDicts = Queue()
   Example(q1)

   Of course, I have tested nothing. Let's pretend that works.

   Oh – and thanks so much for the kind words! It's a pleasure to dissect the CPython standard library on a workday that I should be dutifully working instead. Why, oh why, did no work get done yet again!? Oh, well. Let's just listen to music and pretend I did something. 🎶 🎧 🎶

   ❤️React with ❤️1Reacted by Moosems
8. Moosems commented on Jun 18, 2024

   

   Moosems

   on Jun 18, 2024

   Author

   More actions

   It's quite unfortunate how messy multiprocessing is 😄.

   😄React with 😄1Reacted by Cecil Curry