Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_regrtest: test_huntrleaks() fails randomly: leaked [9, 1, 1] references #109739

Closed
vstinner opened this issue Sep 22, 2023 · 8 comments
Closed

test_regrtest: test_huntrleaks() fails randomly: leaked [9, 1, 1] references #109739

vstinner opened this issue Sep 22, 2023 · 8 comments
Labels
OS-windows tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

vstinner commented Sep 22, 2023 •
edited by bedevere-app bot
Loading

Failure on GHA Windows x86 CI:

FAIL: test_huntrleaks (test.test_regrtest.ArgsTestCase.test_huntrleaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_regrtest.py", line 1068, in test_huntrleaks
    self.check_huntrleaks(run_workers=False)
  File "D:\a\cpython\cpython\Lib\test\test_regrtest.py", line 1065, in check_huntrleaks
    self.check_leak(code, 'references', run_workers=run_workers)
  File "D:\a\cpython\cpython\Lib\test\support\__init__.py", line 2564, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\test_regrtest.py", line 1047, in check_leak
    self.assertIn(line2, output)
AssertionError:

'test_regrtest_huntrleaks leaked [1, 1, 1] references, sum=3\n' 

not found in

'0:00:00 Run 1 test sequentially\n
0:00:00 [1/1] test_regrtest_huntrleaks\n
beginning 6 repetitions\n
123456\n
......\n
test_regrtest_huntrleaks leaked [9, 1, 1] references, sum=11\n
test_regrtest_huntrleaks leaked [1, 1, 1] memory blocks, sum=3\n
test_regrtest_huntrleaks failed (reference leak)\n
\n
== Tests result: FAILURE ==\n
\n
1 test failed:\n
    test_regrtest_huntrleaks\n
\n
Total duration: 292 ms\n
Total tests: run=1\n
Total test files: run=1/1 failed=1\n
Result: FAILURE\n'

build: https://github.com/python/cpython/actions/runs/6274556366/job/17041487300?pr=109727

Linked PRs
@vstinner vstinner added the tests Tests in the Lib/test dir label Sep 22, 2023
@chgnrdv chgnrdv mentioned this issue Sep 24, 2023
Merged
@sobolevn sobolevn mentioned this issue Sep 25, 2023
Merged
@vstinner vstinner mentioned this issue Sep 25, 2023
Merged
@vstinner
Copy link
Member Author

Windows x64: https://github.com/python/cpython/actions/runs/6299520741/job/17100438819?pr=109831

@vstinner
Copy link
Member Author

I cannot reproduce the issue on Linux. I stopped the stress test after 19 minutes:

vstinner@mona$ ./python -m test -v test_regrtest -m 'test_huntrleaks*' -v --forever -j10 -r
(...)
0:19:19 load avg: 11.84 [1770] test_regrtest passed

@vstinner
Copy link
Member Author

I failed to reproduce the issue on Windows with this command:

vstinner@WIN C:\victor\python\main>python -m test -v test_regrtest -m test.test_regrtest.ArgsTestCase.test_huntrleaks -v --forever
(...)
0:17:19 load avg: 0.23 [1371] test_regrtest
test_huntrleaks (test.test_regrtest.ArgsTestCase.test_huntrleaks) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.676s

OK

^C

I interrupted the test after 17 minutes.

@vstinner
Copy link
Member Author

To reproduce the issue on Windows, create the file Lib\test\test_leak.py:

import unittest

GLOBAL_LIST = []

class RefLeakTest(unittest.TestCase):
    def test_leak(self):
        GLOBAL_LIST.append(object())

And run the script leak.py:

import sys
import subprocess

while True:
    sys.stdout.write(".")
    sys.stdout.flush()

    cmd = [sys.executable, "-m", "test", "-R", "3:3", "test_leak"]
    proc = subprocess.run(cmd,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.STDOUT,
                          text=True)
    if "test_leak leaked [1, 1, 1] references" in proc.stdout:
        continue

    print(proc.stdout, end="")
    if proc.returncode:
        print(f"FAILED: exit code {proc.returncode}")
        break

@vstinner
Copy link
Member Author

vstinner commented Sep 25, 2023 •
edited
Loading

And run the script leak.py:

And more the system less determistic by running the following command in a different terminal:

python -m test -j1 -r --forever

Example of the bug on Windows:

vstinner@WIN C:\victor\python\main>python leak.py
Running Debug|x64 interpreter...
........................0:00:00 Run 1 test sequentially
0:00:00 [1/1] test_leak
beginning 6 repetitions
123456
......
test_leak leaked [9, 1, 1] references, sum=11
test_leak leaked [1, 1, 1] memory blocks, sum=3
test_leak failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_leak

Total duration: 237 ms
Total tests: run=1
Total test files: run=1/1 failed=1
Result: FAILURE
FAILED: exit code 2

@vstinner
Copy link
Member Author

I can still reproduce the leak even if I disable the bytecode specialization.

diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h
index a77fa11baf..7de6384574 100644
--- a/Include/internal/pycore_code.h
+++ b/Include/internal/pycore_code.h
@@ -236,7 +236,7 @@ extern int _PyLineTable_PreviousAddressRange(PyCodeAddressRange *range);
 /** API for executors */
 extern void _PyCode_Clear_Executors(PyCodeObject *code);

-#define ENABLE_SPECIALIZATION 1
+#define ENABLE_SPECIALIZATION 0

 /* Specialization functions */

@vstinner
Copy link
Member Author

"before" variables when the bug occurs:

test_leak leaked [9, 1, 1] references, sum=11
test_leak leaked [1, 1, 1] memory blocks, sum=3
interned_before=12109
alloc_before=61013
rc_before=94336
fd_before=3

"before" variables when the bug does not occur:

interned_before=12109
alloc_before=61013
rc_before=94333
fd_before=3
test_leak leaked [1, 1, 1] references, sum=3

Only rc_before is different: 94 333 (ok) => 94 336 (bug).

@vstinner
Copy link
Member Author

Oooooooh, the issue comes from regrtest logger: from WindowsLoadTracker thread.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 25, 2023
@vstinner
pythongh-109739: regrtest disables load tracker if refleak
c9ccb80 
Disable the Windwos load tracker when hunting reference leak.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 25, 2023
@vstinner
pythongh-109739: regrtest disables load tracker if refleak
10b0d91 
Disable the Windwos load tracker when hunting reference leak.
@bedevere-app bedevere-app bot mentioned this issue Sep 25, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 25, 2023
@vstinner
pythongh-109739: regrtest disables load tracker if refleak
9fdae73 
Disable the Windwos load tracker when hunting reference leak.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109739: regrtest disables load tracker if refleak
d3a0ac9 
Disable the Windwos load tracker when hunting reference leak.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109739: regrtest disables load tracker if refleak
582f4bd 
regrtest: Fix reference leak check on Windows. Disable the load
tracker on Windows in the reference leak check mode (-R option).
vstinner added a commit that referenced this issue Sep 26, 2023
@vstinner
gh-109739: regrtest disables load tracker if refleak (#109871)
4091deb 
regrtest: Fix reference leak check on Windows. Disable the load
tracker on Windows in the reference leak check mode (-R option).
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109739: regrtest disables load tracker if refleak (python#10…
8e7d778 
…9871)

regrtest: Fix reference leak check on Windows. Disable the load
tracker on Windows in the reference leak check mode (-R option).
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109739: regrtest disables load tracker if refleak (python#10…
470adc5 
…9871)

regrtest: Fix reference leak check on Windows. Disable the load
tracker on Windows in the reference leak check mode (-R option).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OS-windows tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @sobolevn