Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

spacemanspiff2007 · 2024-09-27T05:56:20Z

Bug report

Bug description:

I think this is related to #119819

import logging.config
from queue import SimpleQueue

q = SimpleQueue()

config = {
    'version': 1,
    'handlers': {
        'sink': {
            'class': 'logging.handlers.QueueHandler',
            'queue': q,
        },
    },
    'root': {
        'handlers': ['sink'],
    },
}
logging.config.dictConfig(config)

ValueError: Unable to configure handler 'sink'

SimpleQueue does not implement the full Queue interfaces thus both isinstance(obj, queue.Queue) and the queue interface check fails.
Since this has been working on 3.8 - <3.12 I think the queue interface check is checking for methods that are not used at all and should be adjusted accordingly.

cpython/Lib/logging/config.py

Lines 500 to 525 in 2518918

    
           def _is_queue_like_object(obj): 
        
               """Check that *obj* implements the Queue API.""" 
        
               if isinstance(obj, queue.Queue): 
        
                   return True 
        
               # defer importing multiprocessing as much as possible 
        
               from multiprocessing.queues import Queue as MPQueue 
        
               if isinstance(obj, MPQueue): 
        
                   return True 
        
               # Depending on the multiprocessing start context, we cannot create 
        
               # a multiprocessing.managers.BaseManager instance 'mm' to get the 
        
               # runtime type of mm.Queue() or mm.JoinableQueue() (see gh-119819). 
        
               # 
        
               # Since we only need an object implementing the Queue API, we only 
        
               # do a protocol check, but we do not use typing.runtime_checkable() 
        
               # and typing.Protocol to reduce import time (see gh-121723). 
        
               # 
        
               # Ideally, we would have wanted to simply use strict type checking 
        
               # instead of a protocol-based type checking since the latter does 
        
               # not check the method signatures. 
        
               queue_interface = [ 
        
                   'empty', 'full', 'get', 'get_nowait', 
        
                   'put', 'put_nowait', 'join', 'qsize', 
        
                   'task_done', 
        
               ] 
        
               return all(callable(getattr(obj, method, None)) 
        
                          for method in queue_interface)

Tested with 3.12.6

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

The text was updated successfully, but these errors were encountered:

YvesDup · 2024-10-01T16:06:37Z

Except if I miss something, SimpleQueue has never had a put_nowait method, the one used in the QueueHandler and QueueListener classes (Just see here) . So using this class in the logging QueueHandler always failed.

spacemanspiff2007 · 2024-10-01T17:11:11Z

As I've written this has been working on 3.8 - 3.11. I'm not sure exactly when it broke.
SimpleQueue actually implements put_nowait.

I skimmed over the file you linked and it seems the only required interface methods are put_nowait and get.
The interface check should reflect that.
Additionally get has to raise queue.Empty if the queue is empty but there is no way to check that ...

YvesDup · 2024-10-01T20:13:32Z

My apologies for my mistaken: I was thinking you talked about multiprocessing.SimpleQueue not about queue.SimpleQueue. So first of all, I suggess to modify title issue (prefixing SimpleQueuewith the queue module).

And then, I agree with you the queue_interface list of the _is_queue_like_object method is too restrictive. At least join and task_done methods must be removed. full and emptymethods (or exceptions) have to exam too.

Perhars it will be nice to ask for a python core dev advice.

YvesDup · 2024-10-02T12:42:10Z

@picnixz as GH-122154 author, and introducing is_queue_like_object in file logging/config.py, could you take a look to this issue please ?

picnixz · 2024-10-02T12:53:24Z

Urgh, this issue will haunt me all my life :') According to the docs, it required a "Queue API", but I didn't consider that SimpleQueue was... too simple enough and did not entirely conform to what Queue did. Ok, So I think we should only care about two methods that are those given in SimpleQueue.

YvesDup · 2024-10-02T13:27:52Z

So I think we should only care about two methods that are those given in SimpleQueue.

What do you think about adding queue.simpleQueue at L502, as below:

if isinstance(obj, (queue.Queue, queue.SimpleQueue)): 
         return True

picnixz · 2024-10-02T13:31:43Z

It's correct (and that's what I also did), but since we really don't know whether people could use objects or not I just relaxed the interface (and I think it's fine like this).

YvesDup · 2024-10-02T15:18:58Z

@spacemanspiff2007 good catch !!

spacemanspiff2007 · 2024-10-02T17:31:09Z

@YvesDup @picnixz Thank you both for your help and implementation

picnixz · 2024-10-02T17:38:23Z

I should be the one thanking you since this was my code :') sorry for the inconvenience!

…H-124897)

…dlers (pythonGH-124897) (cherry picked from commit 7ffe94f) Co-authored-by: Bénédikt Tran <[email protected]>

…ndlers (GH-124897) (GH-125059) (cherry picked from commit 7ffe94f)

…ndlers (GH-124897) (GH-125060) (cherry picked from commit 7ffe94f)

spacemanspiff2007 added the type-bug An unexpected behavior, bug, or error label Sep 27, 2024

AlexWaygood added the stdlib Python modules in the Lib dir label Sep 27, 2024

AlexWaygood added this to Logging issues 🪵 Sep 27, 2024

spacemanspiff2007 changed the title ~~Python 3.12+ breaks backwards compatibility for logging QueueHandler with SimpleQueue~~ Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue Oct 2, 2024

picnixz self-assigned this Oct 2, 2024

picnixz moved this to In Progress in Logging issues 🪵 Oct 2, 2024

bedevere-app bot mentioned this issue Oct 2, 2024

gh-124653: relax (again) detection of queue API for logging handlers #124897

Merged

vsajip pushed a commit that referenced this issue Oct 7, 2024

gh-124653: Relax (again) detection of queue API for logging handlers (G…

7ffe94f

…H-124897)

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 7, 2024

pythongh-124653: Relax (again) detection of queue API for logging han…

27684ab

…dlers (pythonGH-124897) (cherry picked from commit 7ffe94f) Co-authored-by: Bénédikt Tran <[email protected]>

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 7, 2024

pythongh-124653: Relax (again) detection of queue API for logging han…

e92ca5b

…dlers (pythonGH-124897) (cherry picked from commit 7ffe94f) Co-authored-by: Bénédikt Tran <[email protected]>

This was referenced Oct 7, 2024

[3.13] gh-124653: Relax (again) detection of queue API for logging handlers (GH-124897) #125059

Merged

[3.12] gh-124653: Relax (again) detection of queue API for logging handlers (GH-124897) #125060

Merged

vsajip pushed a commit that referenced this issue Oct 8, 2024

[3.13] gh-124653: Relax (again) detection of queue API for logging ha…

1e820e6

…ndlers (GH-124897) (GH-125059) (cherry picked from commit 7ffe94f)

vsajip pushed a commit that referenced this issue Oct 8, 2024

[3.12] gh-124653: Relax (again) detection of queue API for logging ha…

bc237ed

…ndlers (GH-124897) (GH-125060) (cherry picked from commit 7ffe94f)

vsajip closed this as completed Oct 8, 2024

github-project-automation bot moved this from In Progress to Done in Logging issues 🪵 Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

spacemanspiff2007 commented Sep 27, 2024 •

edited by bedevere-app bot

Loading

YvesDup commented Oct 1, 2024

spacemanspiff2007 commented Oct 1, 2024 •

edited

Loading

YvesDup commented Oct 1, 2024

YvesDup commented Oct 2, 2024

picnixz commented Oct 2, 2024 •

edited

Loading

YvesDup commented Oct 2, 2024

picnixz commented Oct 2, 2024

YvesDup commented Oct 2, 2024

spacemanspiff2007 commented Oct 2, 2024

picnixz commented Oct 2, 2024

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

Comments

spacemanspiff2007 commented Sep 27, 2024 • edited by bedevere-app bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

YvesDup commented Oct 1, 2024

spacemanspiff2007 commented Oct 1, 2024 • edited Loading

YvesDup commented Oct 1, 2024

YvesDup commented Oct 2, 2024

picnixz commented Oct 2, 2024 • edited Loading

YvesDup commented Oct 2, 2024

picnixz commented Oct 2, 2024

YvesDup commented Oct 2, 2024

spacemanspiff2007 commented Oct 2, 2024

picnixz commented Oct 2, 2024

spacemanspiff2007 commented Sep 27, 2024 •

edited by bedevere-app bot

Loading

spacemanspiff2007 commented Oct 1, 2024 •

edited

Loading

picnixz commented Oct 2, 2024 •

edited

Loading