Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

asyncio.TaskGroup does not close unawaited coroutines #115957

Closed
arthur-tacca opened this issue Feb 26, 2024 · 3 comments
Closed

asyncio.TaskGroup does not close unawaited coroutines #115957

arthur-tacca opened this issue Feb 26, 2024 · 3 comments
Labels
topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@arthur-tacca
Copy link

arthur-tacca commented Feb 26, 2024 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

Consider the following code:

import asyncio

async def wait_and_raise():
    await asyncio.sleep(0.5)
    raise RuntimeError(1)

async def wait_and_start(tg):
    try:
        await asyncio.sleep(1)
    finally:
        try:
            tg.create_task(asyncio.sleep(1))
        except RuntimeError as e:
            print(f"wait_and_start() caught {e!r}")

async def main():
    try:
        async with asyncio.TaskGroup() as tg:
            tg.create_task(wait_and_start(tg))
            tg.create_task(wait_and_raise())
    except Exception as e:
        print(f"main() caught {e!r}")

    try:
        tg.create_task(asyncio.sleep(1))
    except RuntimeError as e:
        print(f"main() caught {e!r}")

asyncio.run(main())

This gives the following output

wait_and_start() caught RuntimeError('TaskGroup <TaskGroup tasks=1 errors=1 cancelling> is shutting down')
C:\code\taskgrouptest.py:16: RuntimeWarning: coroutine 'sleep' was never awaited
  print(f"wait_and_start() caught {e!r}")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
main() caught ExceptionGroup('unhandled errors in a TaskGroup', [RuntimeError(1)])
main() caught RuntimeError('TaskGroup <TaskGroup cancelling> is finished')
C:\code\taskgrouptest.py:29: RuntimeWarning: coroutine 'sleep' was never awaited
  print(f"main() caught {e!r}")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Arguably, when you call tg.create_task() on a task group that is shutting down or has finished, the calling code "knows" about the error because it gets a RuntimeError exception (as you can see above), so there is no need to get a warning about a coroutine that was not awaited. So, when a TaskGroup encounters this situation, it should close the coroutine before raising the error.

The other argument would be that this still represents a design mistake so should still get the warning. I can see both points of view but I'm raising this issue so a conscious decision can be made.

For comparison, when you do this on a Trio Nursery or AnyIO TaskGroup that has already closed, a coroutine never even gets created in the first place, because you use a different syntax (nursery.start_soon(foo, 1, 2) rather than tg.create_task(foo(1, 2))), so it's a lot like if asyncio were to close the coroutine. The situation is a bit different for a nursery that is shutting down: then it runs till the first (unshielded) await and is cancelled at that point, which is possible because they use level-based cancellation rather than edge-based cancellation.

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs
@arthur-tacca arthur-tacca added the type-bug An unexpected behavior, bug, or error label Feb 26, 2024
@github-project-automation github-project-automation bot moved this to Todo in asyncio Feb 26, 2024
@arthur-tacca
Copy link
Author

I'm told loop.create_task() has the same problem if the loop is closed.

@gvanrossum
Copy link
Member

I see. Reproduced and understood. The issue is in this code at the top of TaskGroup.create_task():

cpython/Lib/asyncio/taskgroups.py

Lines 156 to 161 in 6087315

if not self._entered:
raise RuntimeError(f"TaskGroup {self!r} has not been entered")
if self._exiting and not self._tasks:
raise RuntimeError(f"TaskGroup {self!r} is finished")
if self._aborting:
raise RuntimeError(f"TaskGroup {self!r} is shutting down")

If we inserted coro.close() before each of the three raise statements there we would no longer get the warning. When using the common pattern, the coroutine would be closed without executing a single line of code.

If the coroutine argument is something else, who knows, but the create_task() argument is expected to conform to the complete protocol of coroutines, so if someone is duck typing this, they should already be prepared for this.

It's also the case that if someone squirrels away a reference to the coroutine object, they can tell that it hasn't been closed (it's tricky, you basically have to try to call its send() or throw() method), but that also feels like a very remote possibility where someone is already breaking the contract for create_task().

So I think this is a reasonable fix, but I wouldn't want to backport it to 3.12 or before.

Would you be interested in submitting a PR? The fix itself is trivial, but it would be nice if there was a test for the new behavior (this can be adapted from your example, using coro.send(None) to test whether the coroutine has already been closed). A "versionchanged" directive in the docs for TaskGroup.create_task() would also be appreciated, as well as a brief mention in "What's new in 3.13". Let me know if you need more pointers.

Regarding loop.create_task(), the logic there is a bit trickier, and the use case feels weaker, so I'd leave that alone for now.

Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Feb 27, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup has been closed
5ba5ec7
@Jason-Y-Z Jason-Y-Z mentioned this issue Feb 27, 2024
Merged
@Jason-Y-Z
Copy link
Contributor

Jason-Y-Z commented Feb 27, 2024 •
edited
Loading

Sorry for jumping in, but this looks quite relevant to what I'm working on, so I had a go at implementing the simple change, in #116009, and modified the tests accordingly.

Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Feb 27, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup has been closed
e4ab02f
Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Feb 27, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup has been closed
6a0265d
@arthur-tacca arthur-tacca mentioned this issue Feb 28, 2024
Open
Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Feb 28, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup is inactive
8e9866e
Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Mar 2, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup is inactive
30ce83c
Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Mar 2, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup is inactive
e7c3a5e
Jason-Y-Z added a commit to Jason-Y-Z/cpython that referenced this issue Mar 5, 2024
@Jason-Y-Z
pythongh-115957: Close coroutine if the TaskGroup is inactive
5bcc3d5
gvanrossum pushed a commit that referenced this issue Mar 6, 2024
@Jason-Y-Z
gh-115957: Close coroutine if TaskGroup.create_task() raises an error (…
ce0ae1d 
…#116009)
@github-project-automation github-project-automation bot moved this from Todo to Done in asyncio Mar 6, 2024
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@Jason-Y-Z @adorilson
pythongh-115957: Close coroutine if TaskGroup.create_task() raises an…
258c243 
… error (python#116009)
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@Jason-Y-Z @diegorusso
pythongh-115957: Close coroutine if TaskGroup.create_task() raises an…
d5be5e3 
… error (python#116009)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-asyncio type-bug An unexpected behavior, bug, or error
Projects
asyncio
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gvanrossum @Jason-Y-Z @arthur-tacca @Eclips4