-
-
Notifications
You must be signed in to change notification settings - Fork 30.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration #121723
Comments
@vsajip I suspect this is caused by the fact that you instantiate a queue to get its runtime type. Maybe a protocol approach for a queue would be better? I don't know if the queue you expect should be any queue or if whatever is a queue-like API is fine. |
A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging? The fundamental problem here is that a proxy is used from which you can't easily infer what type the proxy is to. Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried. |
Yes. Don't really care about the type itself, just checking that methods of certain names exist, not even their signatures. And if something goes wrong, there would be a TypeError when you want to use the queue I think. You could reduce issues by having a well-documented section on what kind of queue API you expect, and that should be enough I think. |
IIRC, the proxy is a local class and it does not necessarily require to have a class. It's a bit hard to explain here but if you look at |
Right, I had a quick look when this issue came up before and it didn't seem that straightforward, which is why I went with the approach of instantiating the queue to get its type 😞 |
So... protocol approach it will be? (I can make a PR tomorrow or in two days) |
…lers.QueueHandler`. (pythonGH-122154) (cherry picked from commit fb864c7) Co-authored-by: Bénédikt Tran <[email protected]>
…lers.QueueHandler`. (pythonGH-122154) (cherry picked from commit fb864c7) Co-authored-by: Bénédikt Tran <[email protected]>
…lers.QueueHandler`. (pythonGH-122154)
…xt in emulated JIT CI (pythonGH-122969) (cherry picked from commit 7b8328b) Co-authored-by: Bénédikt Tran <[email protected]>
…lers.QueueHandler`. (pythonGH-122154)
…xt in emulated JIT CI (python#122969)
…ext in emulated JIT CI (GH-122991) (cherry picked from commit 7b8328b) Co-authored-by: Bénédikt Tran <[email protected]>
Bug report
Bug description:
multiprocessing.Pool with spawn method breaks logging.handlers.QueueHandler configuration
got exception (when q = manager.Queue() or q = multiprocessing.Queue())
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Linked PRs
logging.handlers.QueueHandler
. #122154logging.handlers.QueueHandler
. (GH-122154) #122603logging.handlers.QueueHandler
. (GH-122154) #122604test_config_queue_handler_multiprocessing_context
in emulated JIT CI #122969The text was updated successfully, but these errors were encountered: