Multiple PyFrameObjects can be created for a single _PyInterpreterFrame
#97002
Assignees
Labels
3.11
only security fixes
3.12
bugs and security fixes
release-blocker
sprint
type-crash
A hard crash of the interpreter, possibly with a core dump
Comments
davfsa
added a commit
to davfsa/hikari
that referenced
this issue
Sep 21, 2022
2 tasks
|
@pablogsal, this looks a lot like #95818. Possibly related to #96975? I'll dig into it further. |
|
@davfsa, does this crash pretty consistently? I'm unable to reproduce on Do you have concrete (ideally, minimal) steps that I can follow to reproduce it? Or maybe a core dump? |
I'll check when I get home, but before opening this issue I ran our test suite several times and was quite consistent. The segfault happens only on our CI, so might be a bit hard to get a core dump from there. I'll try to get one locally. So far the underlying issue I found happens when aiohttp leaves an open websocket, leading to pytest to wrap it and raise a warning, but we have it setup to error on any warnings. I'll try go get a minimally repo as soon as I can, but the issue can be found when running our test suite (https://github.com/hikari-py/hikari) using |
|
I am able to reproduce this intermittently now, partially minimized. What I'm doing is running: $ pytest -v tests/hikari/impl/test_rest_bot.py --covSometimes we get those attribute errors in the tracebacks where frames/code/etc. are being replaced with seemingly random objects, and sometimes it just segfaults. So this does look like a use-after-free somewhere, similar to #95818. Worth noting is that this only seems to happen when |
|
With a debug build, I'm (sometimes) able to observe the following assertion failure during Objects/frameobject.c:1304: PyFrame_GetCode: Assertion `code != NULL' failed.
Fatal Python error: Aborted
Current thread 0x00007f86102d0280 (most recent call first):
File "/home/brandtbucher/cpython/Lib/traceback.py", line 353 in _walk_tb_with_full_positions
File "/home/brandtbucher/cpython/Lib/traceback.py", line 416 in _extract_from_extended_frame_gen
File "/home/brandtbucher/cpython/Lib/traceback.py", line 688 in __init__
File "/home/brandtbucher/cpython/Lib/traceback.py", line 139 in format_exception
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 83 in pytest_runtest_setup
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda>
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 124 in runtestprotocol
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main
File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main
File "/home/brandtbucher/env311/bin/pytest", line 8 in <module>Full dump: #0 raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x000055878e6ede65 in faulthandler_fatal_error (signum=6) at ./Modules/faulthandler.c:373
#2 <signal handler called>
#3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#4 0x00007fdbd477a859 in __GI_abort () at abort.c:79
#5 0x00007fdbd477a729 in __assert_fail_base (fmt=0x7fdbd4910588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55878e7bb321 "code != NULL", file=0x55878e7a2af4 "Objects/frameobject.c", line=1304, function=<optimized out>) at assert.c:92
#6 0x00007fdbd478bf36 in __GI___assert_fail (assertion=assertion@entry=0x55878e7bb321 "code != NULL", file=file@entry=0x55878e7a2af4 "Objects/frameobject.c", line=line@entry=1304, function=function@entry=0x55878e7a3640 <__PRETTY_FUNCTION__.18739> "PyFrame_GetCode") at assert.c:101
#7 0x000055878e55e344 in PyFrame_GetCode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6df0>: frame=frame@entry=) at Objects/frameobject.c:1306
#8 0x000055878e55e37a in frame_getcode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6670>: f=, closure=<optimized out>) at Objects/frameobject.c:97
#9 0x000055878e54223b in getset_get (descr=descr@entry=0x7fdbd4385850, Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=obj@entry=, type=<optimized out>) at Objects/descrobject.c:202
#10 0x000055878e599bf9 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=, name='f_code', dict=dict@entry=0x0, suppress=suppress@entry=0) at ./Include/object.h:133
#11 0x000055878e59a520 in PyObject_GenericGetAttr (obj=<optimized out>, name=<optimized out>) at Objects/object.c:1367
#12 0x000055878e5998b2 in PyObject_GetAttr (Python Exception <class 'gdb.error'> There is no member named f_frame.: v=v@entry=, name='f_code') at Objects/object.c:916
#13 0x000055878e66121b in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0a3d2a0, throwflag=throwflag@entry=0) at Python/ceval.c:3466
#14 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0a3d2a0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#15 gen_send_ex2 (gen=0x7fdbd0a3d250, arg=arg@entry=0x0, presult=presult@entry=0x7ffdc69a9b10, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#16 0x000055878e55491f in gen_iternext (gen=<optimized out>) at Objects/genobject.c:594
#17 0x000055878e665e67 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6310, frame@entry=0x7fdbd42d61b0, throwflag=throwflag@entry=0) at ./Include/object.h:133
#18 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d61b0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#19 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#20 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#21 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, args=args@entry=0x7ffdc69a9d30, nargsf=nargsf@entry=4, kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:152
#22 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, obj=obj@entry=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=args@entry=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:482
#23 0x000055878e5bcda1 in slot_tp_init (self=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:7861
#24 0x000055878e5b83e1 in type_call (type=0x55878f938370, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:1112
#25 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, keywords=keywords@entry=('limit', 'compact')) at Objects/call.c:214
#26 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('limit', 'compact'), nargsf=9223372036854775811, args=0x7fdbd42d6188, callable=<type at remote 0x55878f938370>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#27 PyObject_Vectorcall (callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargsf=<optimized out>, nargsf@entry=9223372036854775811, kwnames=kwnames@entry=('limit', 'compact')) at Objects/call.c:299
#28 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, kwnames=('limit', 'compact')) at Python/ceval.c:7305
#29 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6100, frame@entry=0x7fdbd0967c40, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#30 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0967c40, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#31 gen_send_ex2 (gen=gen@entry=0x7fdbd0967bf0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa090, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#32 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0967bf0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287
#33 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310
#34 0x000055878e545401 in method_vectorcall_O (func=<method_descriptor at remote 0x7fdbd4387110>, args=0x7ffdc69aa230, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:481
#35 0x000055878e536905 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ffdc69aa230, callable=<method_descriptor at remote 0x7fdbd4387110>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#36 PyObject_VectorcallMethod (name=<optimized out>, args=args@entry=0x7ffdc69aa230, nargsf=2, nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at Objects/call.c:855
#37 0x000055878e65ca0a in PyObject_CallMethodOneArg (arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, name=<optimized out>, self=<generator at remote 0x7fdbd0967bf0>) at ./Include/cpython/abstract.h:116
#38 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0b4f710, throwflag=throwflag@entry=0) at Python/ceval.c:2598
#39 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0b4f710, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#40 gen_send_ex2 (gen=gen@entry=0x7fdbd0b4f6c0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa2f0, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219
#41 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0b4f6c0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287
#42 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310
#43 0x000055878e593230 in cfunction_vectorcall_O (func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=0x7fdbd42d60e0, nargsf=<optimized out>, kwnames=<optimized out>) at ./Include/cpython/methodobject.h:52
#44 0x000055878e53610a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fdbd42d60e0, callable=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#45 PyObject_Vectorcall (callable=callable@entry=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=args@entry=0x7fdbd42d60e0, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:299
#46 0x000055878e657908 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, func@entry=<method_descriptor at remote 0x7fdbd4387110>, args=args@entry=0x7fdbd42d60d8, nargs=2, kwnames=0x0) at Python/ceval.c:7299
#47 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6010, frame@entry=0x7fdbd42d5ec8, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#48 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ec8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#49 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#50 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#51 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aa5d0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:152
#52 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:482
#53 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/typeobject.c:7630
#54 0x000055878e5352c9 in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:343
#55 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:355
#56 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), callargs=callargs@entry=(), kwdict=kwdict@entry={'item': <Function at remote 0x7fdbd0a83f60>}, use_tracing=255) at Python/ceval.c:7347
#57 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5e40, frame@entry=0x7fdbd42d5cc0, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#58 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5cc0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#59 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#60 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#61 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at ./Include/object.h:772
#62 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:328
#63 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:355
#64 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635ff0>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwdict=kwdict@entry={}, use_tracing=255) at Python/ceval.c:7347
#65 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5c18, frame@entry=0x7fdbd42d5ae0, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#66 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ae0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#67 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#68 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#69 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635910>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#70 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#71 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#72 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635910>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#73 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d59f0, frame@entry=0x7fdbd42d58a8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#74 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d58a8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#75 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#76 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#77 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aadd0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/call.c:152
#78 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/call.c:482
#79 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/typeobject.c:7630
#80 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, keywords=keywords@entry=('item', 'nextitem')) at Objects/call.c:214
#81 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('item', 'nextitem'), nargsf=9223372036854775808, args=0x7fdbd42d5890, callable=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#82 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('item', 'nextitem')) at Objects/call.c:299
#83 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, kwnames=('item', 'nextitem')) at Python/ceval.c:7305
#84 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd42d5810, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#85 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5810, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#86 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#87 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#88 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3637120>, tuple=tuple@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#89 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#90 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#91 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3637120>, callargs=callargs@entry=(<Session at remote 0x7fdbd25c6610>,), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#92 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5720, frame@entry=0x7fdbd42d55d8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#93 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d55d8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#94 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#95 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#96 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab3a0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:152
#97 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=(), kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:482
#98 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=(), kwds={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/typeobject.c:7630
#99 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, keywords=keywords@entry=('session',)) at Objects/call.c:214
#100 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('session',), nargsf=9223372036854775808, args=0x7fdbd42d55d0, callable=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#101 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('session',)) at Objects/call.c:299
#102 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, kwnames=('session',)) at Python/ceval.c:7305
#103 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5568, frame@entry=0x7fdbd42d5430, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#104 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5430, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#105 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#106 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#107 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, tuple=tuple@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772
#108 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328
#109 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355
#110 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3636f10>, callargs=callargs@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347
#111 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5340, frame@entry=0x7fdbd42d51f8, throwflag=throwflag@entry=0) at Python/ceval.c:5377
#112 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d51f8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#113 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#114 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393
#115 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab970, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:152
#116 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=(), kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:482
#117 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=(), kwds={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/typeobject.c:7630
#118 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, keywords=keywords@entry=('config',)) at Objects/call.c:214
#119 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('config',), nargsf=9223372036854775808, args=0x7fdbd42d51d0, callable=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#120 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('config',)) at Objects/call.c:299
#121 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, kwnames=('config',)) at Python/ceval.c:7305
#122 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5120, frame@entry=0x7fdbd42d5020, throwflag=throwflag@entry=0) at Python/ceval.c:4767
#123 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5020, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#124 _PyEval_Vector (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=0x7fdbd41fa620, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6428
#125 0x000055878e66d81b in PyEval_EvalCode (co=co@entry=<code at remote 0x7fdbd437cf40>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/ceval.c:1154
#126 0x000055878e6c088b in run_eval_code_obj (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, co=co@entry=0x7fdbd437cf40, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/pythonrun.c:1714
#127 0x000055878e6c1028 in run_mod (mod=mod@entry=0x55878f87d5b0, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, flags=flags@entry=0x7ffdc69abe28, arena=arena@entry=0x7fdbd42731c0) at Python/pythonrun.c:1735
#128 0x000055878e6c1134 in pyrun_file (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, closeit=closeit@entry=1, flags=0x7ffdc69abe28) at Python/pythonrun.c:1630
#129 0x000055878e6c4642 in _PyRun_SimpleFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:440
#130 0x000055878e6c488e in _PyRun_AnyFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:79
#131 0x000055878e6e808b in pymain_run_file_obj (program_name=program_name@entry='/home/brandtbucher/env311/bin/python', filename=filename@entry='/home/brandtbucher/env311/bin/pytest', skip_source_first_line=0) at Modules/main.c:360
#132 0x000055878e6e83e3 in pymain_run_file (config=config@entry=0x55878e9f2fc0 <_PyRuntime+59904>) at Modules/main.c:379
#133 0x000055878e6e94b8 in pymain_run_python (exitcode=exitcode@entry=0x7ffdc69abf84) at Modules/main.c:601
#134 0x000055878e6e9522 in Py_RunMain () at Modules/main.c:680
#135 0x000055878e6e959c in pymain_main (args=args@entry=0x7ffdc69abfe0) at Modules/main.c:710
#136 0x000055878e6e9668 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#137 0x000055878e496d06 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15 |
|
Confirmed that it's not def trace(frame, event, arg):
return trace
sys.settrace(trace) |
|
Not sure if it's a symptom or the bug, but I'm fairly confident that the specialization machinery isn't responsible for the changing opcode (I disabled specialization of |
|
Another puzzle piece: the segfault only happens if the file is loaded from a cached PYC. Forcing recompilation with |
|
Again, these could just be action at a distance from the VM getting into some bad state earlier in the program, or changing when and how GC/tracing events happen. But they're interesting, nonetheless. |
|
Sorry for not being able to send this yesterday, was exhausted. But this morning i started playing around a bit with this to try and get a core dump. Instead of a core dump, I got this rather interesting error: Full tracebackThe first part of the traceback is normal. It is possible aiohttp (a library we use) to leave unclosed sockets if not closed correctly. Which we dont mind for the sake of testing. But the interesting part is the last part of the traceback, where somehow Thank you @brandtbucher for having a look at it yesterday! EDIT: Dont want to spam your inbox, so just appending this here, but I have managed to find another case of something seemingly random being injected as Full traceback |
Can confirm this (not 100% of the time, but ill say 9/10 times), when running with already available pyc files, a segfault occurs (see bellow) in the same place all the time, but when I remove it and run the test suite again, a segfault happens in a different part of the run. Not modifying anything and running it again, the segfault doesnt happen at all. But two or three more runs later and the initial segfault can be found again the same place. This seems to be really random and inconsistent for the most part. The tracebacks show that they are usually caused during the cleanup of an unclosed socket, which is controlled by GC and I dont think it can run in a predictable way outside of the current setup I have. I was having a go at trying to get a minimally reproducible project to make it easier to debug, but no luck. If you are curious, I am getting the segfaults weirdness that I mentioned above by running the following: (We use a random-seed plugin to randomize the order of the tests in every run. So I tried to make it as consistent as possible. Also, Core dump#0 PyObject_GetAttr (v=v@entry=0x7f1084d07ab0, name=name@entry=0x7f10840bcb80) at ../Objects/object.c:916
#1 0x000055d50a427c0d in _PyObject_GetMethod (obj=obj@entry=0x7f1084d07ab0, name=0x7f10840bcb80, method=method@entry=0x7ffcd8fcc100) at ../Objects/object.c:1163
#2 0x000055d50a4f30b6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d075c8, frame@entry=0x7f107e9a7aa0, throwflag=throwflag@entry=0) at ../Python/ceval.c:4490
#3 0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107e9a7aa0, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#4 gen_send_ex2 (gen=0x7f107e9a7a50, arg=arg@entry=0x0, presult=presult@entry=0x7ffcd8fcc200, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#5 0x000055d50a3e2d2d in gen_iternext (gen=<optimized out>) at ../Objects/genobject.c:594
#6 0x000055d50a4f23c1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07498, frame@entry=0x7f1084d07338, throwflag=throwflag@entry=0) at ../Python/ceval.c:4339
#7 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d07338, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#8 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#9 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#10 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083dccc00, args=args@entry=0x7ffcd8fcc420, nargsf=nargsf@entry=4,
kwargs=kwargs@entry=0x7f107f4d8530) at ../Objects/call.c:152
#11 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083dccc00, obj=obj@entry=0x7f107df320e0, args=args@entry=0x7f107e1de3f0,
kwargs=kwargs@entry=0x7f107f4d8530) at ../Objects/call.c:482
#12 0x000055d50a44ab4f in slot_tp_init (self=0x7f107df320e0, args=0x7f107e1de3f0, kwds=0x7f107f4d8530) at ../Objects/typeobject.c:7861
#13 0x000055d50a447ae9 in type_call (type=<optimized out>, args=0x7f107e1de3f0, kwds=0x7f107f4d8530) at ../Objects/typeobject.c:1112
#14 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargs=<optimized out>,
keywords=keywords@entry=0x7f1083d84820) at ../Objects/call.c:214
#15 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x55d50b381af0, args=0x7f1084d07310, nargsf=<optimized out>, kwnames=0x7f1083d84820) at ../Include/internal/pycore_call.h:90
#16 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargsf=nargsf@entry=9223372036854775811, kwnames=kwnames@entry=0x7f1083d84820) at ../Objects/call.c:299
#17 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x55d50b381af0, args=args@entry=0x7f1084d07310, nargs=<optimized out>, kwnames=0x7f1083d84820)
at ../Python/ceval.c:7309
#18 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07288, frame@entry=0x7f107f1dfd60, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#19 0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107f1dfd60, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#20 gen_send_ex2 (gen=gen@entry=0x7f107f1dfd10, arg=0x7f107f4bff80, presult=presult@entry=0x7ffcd8fcc780, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#21 0x000055d50a3e2659 in gen_send_ex (gen=0x7f107f1dfd10, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:287
#22 0x000055d50a3e2cc6 in gen_send (gen=<optimized out>, arg=<optimized out>) at ../Objects/genobject.c:310
#23 0x000055d50a3d31fc in method_vectorcall_O (func=0x7f10840d7110, args=0x7ffcd8fcc950, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/descrobject.c:481
#24 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f10840d7110, args=0x7ffcd8fcc950, nargsf=2, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#25 0x000055d50a3c310f in PyObject_VectorcallMethod (name=<optimized out>, args=args@entry=0x7ffcd8fcc950, nargsf=<optimized out>, nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at ../Objects/call.c:855
#26 0x000055d50a4e8cc7 in PyObject_CallMethodOneArg (arg=0x7f107f4bff80, name=<optimized out>, self=0x7f107f1dfd10) at ../Include/cpython/abstract.h:116
#27 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=frame@entry=0x7f107e56f8b0, throwflag=throwflag@entry=0) at ../Python/ceval.c:2598
#28 0x000055d50a3e2278 in _PyEval_EvalFrame (throwflag=0, frame=0x7f107e56f8b0, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#29 gen_send_ex2 (gen=gen@entry=0x7f107e56f860, arg=0x7f107f4bff80, presult=presult@entry=0x7ffcd8fcca10, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:219
#30 0x000055d50a3e2659 in gen_send_ex (gen=0x7f107e56f860, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at ../Objects/genobject.c:287
#31 0x000055d50a3e2cc6 in gen_send (gen=<optimized out>, arg=<optimized out>) at ../Objects/genobject.c:310
#32 0x000055d50a420109 in cfunction_vectorcall_O (func=0x7f107dfbb950, args=0x7f1084d07268, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/methodobject.c:514
#33 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f107dfbb950, args=0x7f1084d07268, nargsf=1, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#34 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f107dfbb950, args=args@entry=0x7f1084d07268, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ../Objects/call.c:299
#35 0x000055d50a4e3a1c in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x7f107dfbb950, func@entry=0x7f10840d7110, args=args@entry=0x7f1084d07260, nargs=2, kwnames=0x0) at ../Python/ceval.c:7303
#36 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d07198, frame@entry=0x7f1084d07050, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#37 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d07050, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#38 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#39 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#40 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fccd10, nargsf=nargsf@entry=1,
kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:152
#41 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>,
kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:482
#42 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083181960, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f107f4d8b30) at ../Objects/typeobject.c:7630
#43 0x000055d50a3c2914 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f107f4d8b30)
at ../Objects/call.c:343
#44 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f1083181960, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f107f4d8b30) at ../Objects/call.c:355
#45 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083181960, callargs=callargs@entry=0x55d50a87b698 <_PyRuntime+58904>, kwdict=kwdict@entry=0x7f107f4d8b30,
use_tracing=255) at ../Python/ceval.c:7351
#46 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06fc8, frame@entry=0x7f1084d06e48, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#47 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06e48, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#48 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#49 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#50 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338e360, tuple=tuple@entry=0x7f107e054190,
kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:245
#51 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338e360, args=args@entry=0x7f107e054190, kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:328
#52 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338e360, args=args@entry=0x7f107e054190, kwargs=kwargs@entry=0x7f107f4ba450) at ../Objects/call.c:355
#53 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338e360, callargs=callargs@entry=0x7f107e054190, kwdict=kwdict@entry=0x7f107f4ba450, use_tracing=255)
at ../Python/ceval.c:7351
#54 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06da0, frame@entry=0x7f1084d06c68, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#55 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06c68, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#56 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#57 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#58 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338dc80, tuple=tuple@entry=0x7f107ef72850,
kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#59 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338dc80, args=args@entry=0x7f107ef72850, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#60 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338dc80, args=args@entry=0x7f107ef72850, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#61 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338dc80, callargs=callargs@entry=0x7f107ef72850, kwdict=kwdict@entry=0x0, use_tracing=255)
at ../Python/ceval.c:7351
#62 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d06b78, frame@entry=0x7f1084d06a30, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#63 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06a30, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#64 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#65 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#66 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fcd510, nargsf=nargsf@entry=1,
kwargs=kwargs@entry=0x7f107f4ba6f0) at ../Objects/call.c:152
#67 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083182090, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>,
kwargs=kwargs@entry=0x7f107f4ba6f0) at ../Objects/call.c:482
#68 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083182090, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f107f4ba6f0) at ../Objects/typeobject.c:7630
#69 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargs=<optimized out>,
keywords=keywords@entry=0x7f10834999f0) at ../Objects/call.c:214
#70 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f1083182090, args=0x7f1084d06a18, nargsf=<optimized out>, kwnames=0x7f10834999f0) at ../Include/internal/pycore_call.h:90
#71 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargsf=nargsf@entry=9223372036854775808, kwnames=kwnames@entry=0x7f10834999f0) at ../Objects/call.c:299
#72 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083182090, args=args@entry=0x7f1084d06a18, nargs=<optimized out>, kwnames=0x7f10834999f0)
at ../Python/ceval.c:7309
#73 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=frame@entry=0x7f1084d06998, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#74 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06998, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#75 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#76 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#77 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338f490, tuple=tuple@entry=0x7f1080698be0,
kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#78 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338f490, args=args@entry=0x7f1080698be0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#79 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338f490, args=args@entry=0x7f1080698be0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#80 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338f490, callargs=callargs@entry=0x7f1080698be0, kwdict=kwdict@entry=0x0, use_tracing=255)
at ../Python/ceval.c:7351
#81 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d068a8, frame@entry=0x7f1084d06760, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#82 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06760, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#83 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#84 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#85 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fcdaf0, nargsf=nargsf@entry=1,
kwargs=kwargs@entry=0x7f1082304b90) at ../Objects/call.c:152
#86 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f10831824f0, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>,
kwargs=kwargs@entry=0x7f1082304b90) at ../Objects/call.c:482
#87 0x000055d50a44ad13 in slot_tp_call (self=0x7f10831824f0, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f1082304b90) at ../Objects/typeobject.c:7630
#88 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargs=<optimized out>,
keywords=keywords@entry=0x7f10835f6030) at ../Objects/call.c:214
#89 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f10831824f0, args=0x7f1084d06758, nargsf=<optimized out>, kwnames=0x7f10835f6030) at ../Include/internal/pycore_call.h:90
#90 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargsf=nargsf@entry=9223372036854775808, kwnames=kwnames@entry=0x7f10835f6030) at ../Objects/call.c:299
#91 0x000055d50a4e37b6 in trace_call_function (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f10831824f0, args=args@entry=0x7f1084d06758, nargs=<optimized out>, kwnames=0x7f10835f6030)
at ../Python/ceval.c:7309
#92 0x000055d50a4f42fc in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d066f0, frame@entry=0x7f1084d065b8, throwflag=throwflag@entry=0) at ../Python/ceval.c:4766
#93 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d065b8, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#94 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#95 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#96 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f108338f280, tuple=tuple@entry=0x7f10822711d0,
kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#97 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f108338f280, args=args@entry=0x7f10822711d0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#98 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f108338f280, args=args@entry=0x7f10822711d0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#99 0x000055d50a4e3a90 in do_call_core (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f108338f280, callargs=callargs@entry=0x7f10822711d0, kwdict=kwdict@entry=0x0, use_tracing=0)
at ../Python/ceval.c:7351
#100 0x000055d50a4f7fe9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d064c8, frame@entry=0x7f1084d06380, throwflag=throwflag@entry=0) at ../Python/ceval.c:5376
#101 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06380, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#102 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#103 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#104 0x000055d50a3c3403 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, args=args@entry=0x7ffcd8fce0d0, nargsf=nargsf@entry=1,
kwargs=kwargs@entry=0x7f10822f25d0) at ../Objects/call.c:152
#105 0x000055d50a3c35d3 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083b49230, obj=obj@entry=0x7f1083166ef0, args=args@entry=0x55d50a87b698 <_PyRuntime+58904>,
kwargs=kwargs@entry=0x7f10822f25d0) at ../Objects/call.c:482
#106 0x000055d50a44ad13 in slot_tp_call (self=0x7f1083166ef0, args=0x55d50a87b698 <_PyRuntime+58904>, kwds=0x7f10822f25d0) at ../Objects/typeobject.c:7630
#107 0x000055d50a3c2c93 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083166ef0, args=args@entry=0x7f1084d06358, nargs=<optimized out>,
keywords=keywords@entry=0x7f10835f6a30) at ../Objects/call.c:214
#108 0x000055d50a3c304f in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f1083166ef0, args=0x7f1084d06358, nargsf=<optimized out>, kwnames=0x7f10835f6a30) at ../Include/internal/pycore_call.h:90
#109 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f1083166ef0, args=args@entry=0x7f1084d06358, nargsf=<optimized out>, kwnames=kwnames@entry=0x7f10835f6a30) at ../Objects/call.c:299
#110 0x000055d50a4f4558 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d062a8, frame@entry=0x7f1084d061b8, throwflag=throwflag@entry=0) at ../Python/ceval.c:4771
#111 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d061b8, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#112 _PyEval_Vector (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=func@entry=0x7f1083fb9bd0, locals=locals@entry=0x7f1083f5b2f0, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0)
at ../Python/ceval.c:6429
#113 0x000055d50a4fa195 in PyEval_EvalCode (co=co@entry=0x7f10832cb0a0, globals=globals@entry=0x7f1083f5b2f0, locals=locals@entry=0x7f1083f5b2f0) at ../Python/ceval.c:1154
#114 0x000055d50a4d86a3 in builtin_exec_impl (module=module@entry=0x7f108412b770, source=0x7f10832cb0a0, globals=0x7f1083f5b2f0, locals=0x7f1083f5b2f0, closure=0x0) at ../Python/bltinmodule.c:1075
#115 0x000055d50a4d8807 in builtin_exec (module=0x7f108412b770, args=<optimized out>, nargs=2, kwnames=<optimized out>) at ../Python/clinic/bltinmodule.c.h:465
#116 0x000055d50a41fade in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f108412be90, args=0x7f1084d06180, nargsf=<optimized out>, kwnames=0x0) at ../Objects/methodobject.c:443
#117 0x000055d50a3c2f5e in _PyObject_VectorcallTstate (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=0x7f108412be90, args=0x7f1084d06180, nargsf=9223372036854775810, kwnames=0x0) at ../Include/internal/pycore_call.h:92
#118 0x000055d50a3c3077 in PyObject_Vectorcall (callable=callable@entry=0x7f108412be90, args=args@entry=0x7f1084d06180, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/call.c:299
#119 0x000055d50a4f4558 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, frame=0x7f1084d060d8, frame@entry=0x7f1084d06020, throwflag=throwflag@entry=0) at ../Python/ceval.c:4771
#120 0x000055d50a4fa0b2 in _PyEval_EvalFrame (throwflag=0, frame=0x7f1084d06020, tstate=0x55d50a895a38 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#121 _PyEval_Vector (tstate=0x55d50a895a38 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:6429
#122 0x000055d50a3c2ac0 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#123 0x000055d50a3c2521 in _PyVectorcall_Call (tstate=tstate@entry=0x55d50a895a38 <_PyRuntime+166328>, func=0x55d50a3c2a6c <_PyFunction_Vectorcall>, callable=callable@entry=0x7f1083e63c20, tuple=tuple@entry=0x7f1083f977a0,
kwargs=kwargs@entry=0x0) at ../Objects/call.c:245
#124 0x000055d50a3c29d5 in _PyObject_Call (tstate=0x55d50a895a38 <_PyRuntime+166328>, callable=callable@entry=0x7f1083e63c20, args=args@entry=0x7f1083f977a0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:328
#125 0x000055d50a3c2a42 in PyObject_Call (callable=callable@entry=0x7f1083e63c20, args=args@entry=0x7f1083f977a0, kwargs=kwargs@entry=0x0) at ../Objects/call.c:355
#126 0x000055d50a57332f in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at ../Modules/main.c:300
#127 0x000055d50a5738b7 in pymain_run_python (exitcode=exitcode@entry=0x7ffcd8fce814) at ../Modules/main.c:595
#128 0x000055d50a57395b in Py_RunMain () at ../Modules/main.c:680
#129 0x000055d50a5739d5 in pymain_main (args=args@entry=0x7ffcd8fce870) at ../Modules/main.c:710
#130 0x000055d50a573aa3 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:734
#131 0x000055d50a323e06 in main (argc=<optimized out>, argv=<optimized out>) at ../Programs/python.c:15 |
Yeah, in general this error is manifesting itself as "code objects are turning into something else". Sometimes that errors in Python land, sometimes it segfaults in C land. I'm 99% sure that what's happening is some sort of use-after-free. Basically, the refcount of a code object is wrong, so it gets deallocated and its memory is later used for some other, new, object (I've seen sentinels, functions, mocks, and other things). It's tough to identify the root cause, though: it could just be bad refcounting somewhere, or it could be yet another symptom of broken internal state. And yeah, the common things seem to be that this is caused by resource warnings during GC, while tracing, with generators/coroutines on the stack somewhere above. One really helpful thing that you could do (since you're way more familiar with the library than I am) would be to try to reproduce this without any (or at least as few as possible) third-party C extensions loaded. It looks like when I'm running |
|
Also, for my own notes: this also happens with |
That is what I have been trying to do this morning, as the underlying issue for a ResourceWarning is an open socket.socket. But every time the error gets handled nicely and I can't seem to cause a crash. I'll have a try again later today :) Also, all those modules are not pytest, they are loaded by us. I was trying to find out exactly where the GCed object comes from to find the test that seems to cause this. I'll try figuring that out too |
|
Disabling the event loop warning in |
|
So we need a warning or error to happen in |
The stack for the core dump seems to indicate that it happens on a Not sure if the BaseEventLoop has much place here. Just wanted to make a comment on that, I am most probably wrong |
|
Let's start by working on the same reproducer. I'm running def trace(frame, event, arg):
return trace
# sys.settrace(trace)Running this succeeds (and prints a Uncommenting the I'd like to have this same file fail (either segfaults or weird Python errors) with as little third-party code as possible (especially C code). |
|
I'm currently working on bisecting the errors to a specific CPython revision. |
|
It's not the code objects that are being replaced, it's the frames. I just saw this failure: AttributeError: 'bool' object has no attribute 'co_positions'It's not possible for a It could even be our stack of (This would also explain the |
|
Got some more information, not sure how relevant this would be, but posting it just in case. Simpler reproduction codeThis has been tricky, as it seems it is very inconsistent on when it occurs (probably due to gc). Removing/adding code can lead to very different results. Because of this, I havent attempted much to make it smaller, but this is what I am currently using: Commit: hikari-py/hikari@68ee392 Patchdiff --git a/tests/hikari/impl/test_rest_bot.py b/tests/hikari/impl/test_rest_bot.py
index ab2120c0..2b6a7b89 100644
--- a/tests/hikari/impl/test_rest_bot.py
+++ b/tests/hikari/impl/test_rest_bot.py
@@ -38,6 +38,17 @@ from hikari.internal import ux
from tests.hikari import hikari_test_helpers
+def trace(frame, event, arg):
+ return trace
+
+
+sys.settrace(trace)
+import gc
+
+v = 10
+gc.set_threshold(v, v, v)
+
+
class TestRESTBot:
@pytest.fixture()
def mock_interaction_server(self):
@@ -231,64 +242,19 @@ class TestRESTBot:
assert mock_rest_bot.is_alive is expected
def test_print_banner(self, mock_rest_bot):
- with mock.patch.object(ux, "print_banner") as print_banner:
- mock_rest_bot.print_banner("okokok", True, False, {"test_key": "test_value"})
-
- print_banner.assert_called_once_with("okokok", True, False, extra_args={"test_key": "test_value"})
+ pass
@pytest.mark.asyncio()
async def test_close(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
- mock_rest_bot._close_event = close_event = mock.Mock()
- mock_interaction_server.close = mock.AsyncMock()
- mock_rest_bot._is_closing = False
-
- await mock_rest_bot.close()
-
- mock_interaction_server.close.assert_awaited_once()
- mock_rest_client.close.assert_awaited_once()
- close_event.set.assert_called_once()
- assert mock_rest_bot._is_closing is True
+ pass
@pytest.mark.asyncio()
async def test_close_when_is_closing(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
- mock_rest_bot._close_event = mock.Mock()
- mock_interaction_server.close = mock.AsyncMock()
- mock_rest_bot._is_closing = True
- mock_rest_bot.join = mock.AsyncMock()
-
- await mock_rest_bot.close()
-
- mock_interaction_server.close.assert_not_called()
- mock_rest_client.close.assert_not_called()
- mock_rest_bot._close_event.set.assert_not_called()
- mock_rest_bot.join.assert_awaited_once()
+ pass
@pytest.mark.asyncio()
async def test_close_when_inactive(self, mock_rest_bot):
- with pytest.raises(errors.ComponentStateConflictError):
- await mock_rest_bot.close()
-
- @pytest.mark.asyncio()
- async def test_join(self, mock_rest_bot):
- mock_rest_bot._close_event = mock.AsyncMock()
-
- await mock_rest_bot.join()
-
- mock_rest_bot._close_event.wait.assert_awaited_once()
-
- @pytest.mark.asyncio()
- async def test_join_when_not_alive(self, mock_rest_bot):
- with pytest.raises(errors.ComponentStateConflictError):
- await mock_rest_bot.join()
-
- @pytest.mark.asyncio()
- async def test_on_interaction(self, mock_rest_bot, mock_interaction_server):
- mock_interaction_server.on_interaction = mock.AsyncMock()
-
- result = await mock_rest_bot.on_interaction(b"1", b"2", b"3")
-
- assert result is mock_interaction_server.on_interaction.return_value
- mock_interaction_server.on_interaction.assert_awaited_once_with(b"1", b"2", b"3")
+ pass
def test_run(self, mock_rest_bot):
# Dependent on test-order the current event loop may be pre-set and closed without pytest.mark.asyncio
@@ -345,164 +311,3 @@ class TestRESTBot:
mock_rest_bot.run(asyncio_debug=True)
get_or_make_loop.return_value.set_debug.assert_called_once_with(True)
-
- def test_run_with_coroutine_tracking_depth(self, mock_rest_bot):
- mock_rest_bot.start = mock.Mock()
- mock_rest_bot.join = mock.Mock()
-
- with mock.patch.object(aio, "get_or_make_loop"):
- with mock.patch.object(
- sys, "set_coroutine_origin_tracking_depth", side_effect=AttributeError, create=True
- ) as set_tracking_depth:
- mock_rest_bot.run(coroutine_tracking_depth=42)
-
- set_tracking_depth.assert_called_once_with(42)
-
- def test_run_when_already_running(self, mock_rest_bot):
- mock_rest_bot._close_event = object()
-
- with pytest.raises(errors.ComponentStateConflictError):
- mock_rest_bot.run()
-
- def test_run_closes_executor_when_present(self, mock_rest_bot, mock_executor):
- mock_rest_bot.join = mock.AsyncMock()
- # Dependent on test-order the current event loop may be pre-set and closed without pytest.mark.asyncio
- # therefore we need to ensure there's no pre-set event loop.
- asyncio.set_event_loop(None)
- mock_rest_bot.run(
- asyncio_debug=False,
- backlog=321,
- check_for_updates=False,
- close_loop=False,
- close_passed_executor=True,
- coroutine_tracking_depth=32123,
- enable_signal_handlers=True,
- host="192.168.1.102",
- path="pathathath",
- port=4554,
- reuse_address=True,
- reuse_port=False,
- shutdown_timeout=534.534,
- socket=object(),
- ssl_context=object(),
- )
-
- mock_executor.shutdown.assert_called_once_with(wait=True)
- assert mock_rest_bot.executor is None
-
- def test_run_ignores_close_executor_when_not_present(self, mock_rest_bot):
- # Dependent on test-order the current event loop may be pre-filled and closed without pytest.mark.asyncio
- # therefore we need to ensure there's no pre-set event loop.
- asyncio.set_event_loop(None)
- mock_rest_bot.join = mock.AsyncMock()
- mock_rest_bot._executor = None
-
- mock_rest_bot.run(
- asyncio_debug=False,
- backlog=321,
- check_for_updates=False,
- close_loop=False,
- close_passed_executor=True,
- coroutine_tracking_depth=32123,
- enable_signal_handlers=True,
- host="192.168.1.102",
- path="pathathath",
- port=4554,
- reuse_address=True,
- reuse_port=False,
- shutdown_timeout=534.534,
- socket=object(),
- ssl_context=object(),
- )
-
- assert mock_rest_bot.executor is None
-
- @pytest.mark.asyncio()
- async def test_start(self, mock_rest_bot, mock_interaction_server, mock_rest_client):
- mock_socket = object()
- mock_ssl_context = object()
- mock_rest_bot._is_closing = True
-
- with mock.patch.object(ux, "check_for_updates"):
- await mock_rest_bot.start(
- backlog=34123,
- check_for_updates=False,
- enable_signal_handlers=False,
- host="hostostosot",
- port=123123123,
- path="patpatpapt",
- reuse_address=True,
- reuse_port=False,
- socket=mock_socket,
- shutdown_timeout=4312312.3132132,
- ssl_context=mock_ssl_context,
- )
-
- ux.check_for_updates.assert_not_called()
-
- mock_interaction_server.start.assert_awaited_once_with(
- backlog=34123,
- enable_signal_handlers=False,
- host="hostostosot",
- port=123123123,
- path="patpatpapt",
- reuse_address=True,
- reuse_port=False,
- socket=mock_socket,
- shutdown_timeout=4312312.3132132,
- ssl_context=mock_ssl_context,
- )
- mock_rest_client.start.assert_called_once_with()
- assert mock_rest_bot._is_closing is False
-
- @pytest.mark.asyncio()
- async def test_start_checks_for_update(self, mock_rest_bot, mock_http_settings, mock_proxy_settings):
- stack = contextlib.ExitStack()
- stack.enter_context(mock.patch.object(asyncio, "create_task"))
- stack.enter_context(mock.patch.object(ux, "check_for_updates", new=mock.Mock()))
-
- with stack:
- await mock_rest_bot.start(
- backlog=34123,
- check_for_updates=True,
- enable_signal_handlers=False,
- host="hostostosot",
- port=123123123,
- path="patpatpapt",
- reuse_address=True,
- reuse_port=False,
- socket=object(),
- shutdown_timeout=4312312.3132132,
- ssl_context=object(),
- )
-
- asyncio.create_task.assert_called_once_with(
- ux.check_for_updates.return_value, name="check for package updates"
- )
- ux.check_for_updates.assert_called_once_with(mock_http_settings, mock_proxy_settings)
-
- @pytest.mark.asyncio()
- async def test_start_when_is_alive(self, mock_rest_bot):
- mock_rest_bot._close_event = object()
-
- with mock.patch.object(ux, "check_for_updates", new=mock.Mock()) as check_for_updates:
- with pytest.raises(errors.ComponentStateConflictError):
- await mock_rest_bot.start()
-
- check_for_updates.assert_not_called()
-
- def test_get_listener(self, mock_rest_bot, mock_interaction_server):
- mock_type = object()
-
- result = mock_rest_bot.get_listener(mock_type)
-
- assert result is mock_interaction_server.get_listener.return_value
- mock_interaction_server.get_listener.assert_called_once_with(mock_type)
-
- def test_set_listener(self, mock_rest_bot, mock_interaction_server):
- mock_type = object()
- mock_listener = object()
-
- mock_rest_bot.set_listener(mock_type, mock_listener, replace=True)
-
- mock_interaction_server.set_listener.assert_called_once_with(mock_type, mock_listener, replace=True)Removing any more code makes the crash disappear. I havent attempted to lower it even more beyond this. It is also worth noting that I am unable to get 100% crash rate. Sometimes it will just error our with a traceback originated from I have unfortunately not been able to get rid of the C extensions, as doing so (just like removing/adding code) makes the crash not happen. I do not think this is due to the C extensions causing the crash, but rather the sensitivity of the crash happening. Crash upon printing or accessing attributes in tb.tb_frameAfter modifying def _walk_tb_with_full_positions(tb):
# Internal version of walk_tb that yields full code positions including
# end line and column information.
while tb is not None:
- positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
+ try:
+ positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
+ except:
+ breakpoint()
# Yield tb_lineno when co_positions does not have a line number to
# maintain behavior with walk_tb.
if positions[0] is None:
yield tb.tb_frame, (tb.tb_lineno, ) + positions[1:]
else:
yield tb.tb_frame, positions
tb = tb.tb_nextExample 1 (attemting to print it)Example 2 (accessing attributes on it)Incomplete framesWhile looking a the issues that have been mentioned I noticed that they all seem to have incomplete frames as a common factor. I added a simple assertion in PyCodeObject *
PyFrame_GetCode(PyFrameObject *frame)
{
assert(frame != NULL);
- PyCodeObject *code = frame->f_frame->f_code;
+ _PyInterpreterFrame *int_frame = frame->f_frame;
+ assert(!_PyFrame_IsIncomplete(int_frame));
+ PyCodeObject *code = int_frame->f_code;
assert(code != NULL);
Py_INCREF(code);
return code;
}LogsThe CPython version I used here is self-built off 29f17a4 with the only diff being the changes specified in each section) |
|
I've been working on this for a few days, and have been able to minimize it quite a bit. Put the following file ( import asyncio
import gc
import sys
def trace(frame, event, arg):
return trace
gc.set_threshold(1, 0, 0)
def test_crasher():
asyncio.new_event_loop()
# Allocate bunch of GC-tracked objects. This has the effect of pushing the
# new event loop (and its soon-to-be-unclosed socket) to the oldest
# generation and forcing a GC collection shortly after this test completes.
# The socket will be finalized. When this happens, it raises a warning at a
# very specific point in pytest's runner machinery, which puts the VM in a
# bad state and starts breaking stuff:
[[] for _ in range(6941)]
# This is needed, perhaps because this bug requires creation of a Python
# frame object (in this case, as an argument to the trace function)?
sys.settrace(trace)
# Old event loop will be collected during the next GC run:
asyncio.set_event_loop(None)
# See you on the other side!Then, from that directory, run: $ pytest crash.py --verbose --capture no --pythonwarnings error --pythonwarnings ignore::pytest.PytestUnraisableExceptionWarningAll of the options are required to reproduce. It will hang for a while (about a minute on my machine... the low GC threshold is a real performance-killer), then print the following traceback: ===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0 -- /home/brandtbucher/env311/bin/python
cachedir: .pytest_cache
rootdir: /home/brandtbucher/cpython
collected 1 item
crash.py::test_crasher FAILED
========================================================================================== FAILURES ===========================================================================================
________________________________________________________________________________________ test_crasher _________________________________________________________________________________________
exc = <class 'ResourceWarning'>, value = ResourceWarning('unclosed <socket.socket fd=7, family=1, type=1, proto=0>'), tb = <traceback object at 0x7f5f4f349590>, limit = None, chain = True
def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
chain=True):
"""Format a stack trace and the exception information.
The arguments have the same meaning as the corresponding arguments
to print_exception(). The return value is a list of strings, each
ending in a newline and some containing internal newlines. When
these lines are concatenated and printed, exactly the same text is
printed as does print_exception().
"""
value, tb = _parse_value_tb(exc, value, tb)
> te = TracebackException(type(value), value, tb, limit=limit, compact=True)
../Lib/traceback.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../Lib/traceback.py:688: in __init__
self.stack = StackSummary._extract_from_extended_frame_gen(
../Lib/traceback.py:416: in _extract_from_extended_frame_gen
for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
../Lib/traceback.py:353: in _walk_tb_with_full_positions
positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
code = True, instruction_index = 0
def _get_code_position(code, instruction_index):
if instruction_index < 0:
return (None, None, None, None)
> positions_gen = code.co_positions()
E AttributeError: 'bool' object has no attribute 'co_positions'
../Lib/traceback.py:366: AttributeError
=================================================================================== short test summary info ===================================================================================
FAILED crash.py::test_crasher - AttributeError: 'bool' object has no attribute 'co_positions'
===================================================================================== 1 failed in 49.22s =====================================================================================I think The real pain here is pytest itself. There are a lot of very magic frames up above the problematic warning, and at least one of them is giving us trouble.
Only the
Yeah, that's odd. No frame object should have an incomplete |
|
Thanks a lot for having a look at this! Now, I sadly got some bad news. I am unable to reproduce the crash with the code you provided, the test just passes after 1s, but with a small change, I can make it crash in the same way it did for you (takes 33s). import asyncio
import gc
import sys
def trace(frame, event, arg):
return trace
gc.set_threshold(1, 0, 0)
def test_crasher():
asyncio.new_event_loop()
# Allocate bunch of GC-tracked objects. This has the effect of pushing the
# new event loop (and its soon-to-be-unclosed socket) to the oldest
# generation and forcing a GC collection shortly after this test completes.
# The socket will be finalized. When this happens, it raises a warning at a
# very specific point in pytest's runner machinery, which puts the VM in a
# bad state and starts breaking stuff:
- [[] for _ in range(6941)]
+ a = [[] for _ in range(6941)]
# This is needed, perhaps because this bug requires creation of a Python
# frame object (in this case, as an argument to the trace function)?
sys.settrace(trace)
# Old event loop will be collected during the next GC run:
asyncio.set_event_loop(None)
# See you on the other side!LogsWhat i find funny is the initial passing and then failing of the test. @brandtbucher would you mind testing out my diff and seeing if that makes it crash for you too? Then we will both finally have a consistent reproduction (yay!). Again, thank you so much for making it this easy to reproduce, I was unable to go any further. I am going to make a little script to go back in the git history and try to find the first commit where this doesnt crash, which will hopefully narrow this down even further. |
I understand your frustration. That's the nature of these sorts of bugs that rely on precise GC collections: tiny, tiny changes in allocation behavior can make all the difference. Something as simple as new environment variables, import paths, PYC files, etc. can change everything... especially with all the gunk that Pytest runs before the test even begins, and after it ends. I've found that recompiling the same commit multiple times can sometimes require tweaking the magic That value is super fragile, so it probably doesn't lend itself to bisecting to a dirty commit very well. I'm going to try digging into this by figuring out where in the code those incomplete frames are coming from. |
|
Ok, that makes a lot more sense. I had the false sense that you could consistently reproduce it and I was so confused on why I couldn't. Tysm for the insights and how go get the magic number. I'll try and keep reproducing the error today 😄 I don't mind how tedious this bug is, it's something I was well aware when trying to help out, it's more what I described above that I didn't really understand. Thanks for the explanation and your work! |
|
Here's one with no other Python code (besides, of course, pytest): # pytest crash.py --verbose --capture no --pythonwarnings error --pythonwarnings ignore::pytest.PytestUnraisableExceptionWarning
import gc
import _socket
import sys
def trace(frame, event, arg):
return trace
gc.set_threshold(1, 0, 0)
class Socket(_socket.socket):
pass
class Crasher:
def __init__(self):
self.resource = Socket()
self.cycle = self
def test_crasher():
Crasher()
[[] for _ in range(6267)]
sys.settrace(trace)
# print("", *gc.get_stats(), gc.get_count(), sep="\n--> ")Next steps are figuring out what sort of frames pytest is running above us. They're doing something that we aren't handling correctly, and I want to know what. |
|
My initial suspicion is that their pattern of hooks-that-are-context-managers-that-are-really-generators could be a big part of this. Pytest's extensive use of this pattern seems like a pretty good stress test of our Their |
|
Thanks for the second repo! I was having a look through pytests code and their I haven't had enough time to have a proper look, but might be useful for you :) |
|
Yep, that caught my eye too! Looking at it now. |
|
False alarm, still crashes. :( |
_get_code_positionPyFrameObjects can be created for a single _PyInterpreterFrame
|
Update: We've been working on this at the CPython core dev sprint at Google this week. I finally found the root issue, which is that a poorly-timed GC collection just before tracing the resumption of a generator can cause two duplicate frame objects to be created, which corrupts the VM's internal state in all sorts of fun ways. Thanks for your help reporting and debugging this! |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this issue
Oct 6, 2022
… one `PyFrameObject` (pythonGH-97996) (cherry picked from commit 21a2d9f) Co-authored-by: Brandt Bucher <[email protected]>
|
Confirmed that this fixes the segfaults in the Hikari test suite. This fix will ship in 3.11.0! |
brandtbucher
pushed a commit
that referenced
this issue
Oct 7, 2022
Thank you for finding the root cause and fixing it! 🥳 |
|
@brandtbucher Since gh-98002 is merged, can this issue be closed or have its |
|
I'd prefer to leave it open, and let @pablogsal close it once the cherry-pick to 3.11.0 is finished. |
carljm
added a commit
to carljm/cpython
that referenced
this issue
Oct 8, 2022
* main: pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996) pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984) fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965) pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990) pythongh-97850: Remove the open issues section from the import reference (python#97935) Docs: pin sphinx-lint (pythonGH-97992) pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591) Add Pynche's move to the What's new in 3.11 (python#97974) pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785) pythongh-86482: Document assignment expression need for ()s (python#23291) pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
… one `PyFrameObject` (pythonGH-97996)
Repository owner
moved this from Todo
to Done
in Release and Deferred blockers 🚫
Oct 22, 2022
Repository owner
moved this from Todo
to Done
in Sprint 2024
Oct 22, 2022
pablogsal
pushed a commit
that referenced
this issue
Oct 22, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Crash report
I am not really sure under what circumstances this can occur, as I have not been able to isolate the specific piece of code that triggers this. Thankfully, when trying to re-create the segfault to capture a core dump locally, I have found that I instead got a full traceback, which you can find bellow
Error messages
Segfault
Traceback
Your environment
The text was updated successfully, but these errors were encountered: