Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regrtest should re-run failed tests in subprocesses #108834

Closed
vstinner opened this issue Sep 2, 2023 · 0 comments
Closed

regrtest should re-run failed tests in subprocesses #108834

vstinner opened this issue Sep 2, 2023 · 0 comments
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

vstinner commented Sep 2, 2023

Feature or enhancement

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Proposal:

Buildbots and GitHub Actions run the Python test suite with --verbose2 option to re-run failed tests in verbose mode. The current regrtest implementation is the re-run tests in the main process. It is not reliable: if a test crash or timeout, the main process cannot get back the control since the process is killed. Usually, there is no final report after the test is re-run, and the output is just "truncated".

I propose to re-run failed tests in subprocesses to make them more reliable:

  • Each test file is run in a fresh process and so does not inherit the state of previously executed tests. It runs in a known and reproducible state: threads, signals, etc.
  • The main process can kill the worker process if it hangs.
  • If faulthandler kills a worker process on a timeout, the main control detects it and continues its regular work.
  • Same if a worker process is killed because of a fatal error: Python Fatal Error, segmentation fault, etc.
  • Regular regrtest features can be re-used, like detection of leaking temporary files.

I'm working on an implementation.

Linked PRs

@vstinner vstinner added the type-feature A feature request or enhancement label Sep 2, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Add TestResult.get_rerun_match_tests() method
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* run_tests_worker() does also parses the JSON.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* Add tests on normalize_test_name() function.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Fix statistics in regrtest summary.

Changes:

* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
@AlexWaygood AlexWaygood added the tests Tests in the Lib/test dir label Sep 3, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
vstinner added a commit that referenced this issue Sep 3, 2023
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic
namespace which was used to get and set 'random' attributes. Instead,
each class has its own attributes, and some classes are even
read-only (frozen dataclass), like RunTests.

Reorganize files:

* Rename runtest.py to single.py.
* Rename runtest_mp.py to mp_runner.py.
* Create findtests.py, result.py, results.py and worker.py.

Reorganize classes:

* Copy command line options ('ns' namespace) to Regrtest
  and RunTests attributes: add many attributes.
* Add Results class with methods: get_state(), display_result(),
  get_exitcode(), JUnit methods, etc.
* Add Logger class: log(), display_progress(), system load tracker.
* Remove WorkerJob class: the worker now gets a RunTests instance.

Move function and methods:

* Convert Regrtest.list_cases() to a function.
* Convert display_header(), display_sanitizers(), set_temp_dir() and
  create_temp_dir() methods of Regrtest to functions in utils.py.
  Rename set_temp_dir() to select_temp_dir(). Rename
  create_temp_dir() to make_temp_dir().
* Move abs_module_name() and normalize_test_name() to utils.py.

cmdline.py changes:

* Rename internal --worker-args command line option to --worker-json.
* Rename ns.trace to ns.coverage.
* No longer gets the number of CPUs: it's now done by Regrtest class.
* Add missing attributes to Namespace: coverage, threshold, wait.

Misc:

* Add test_parse_memlimit() and test_set_memlimit() to test_support.
* Add some type annotations.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2023
When the --fail-rerun option is used a test failed and then passed,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2023
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner added a commit that referenced this issue Sep 5, 2023
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner added a commit to vstinner/cpython that referenced this issue Sep 5, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic
namespace which was used to get and set 'random' attributes. Instead,
each class has its own attributes, and some classes are even
read-only (frozen dataclass), like RunTests.

Reorganize files:

* Rename runtest.py to single.py.
* Rename runtest_mp.py to mp_runner.py.
* Create findtests.py, result.py, results.py and worker.py.

Reorganize classes:

* Copy command line options ('ns' namespace) to Regrtest
  and RunTests attributes: add many attributes.
* Add Results class with methods: get_state(), display_result(),
  get_exitcode(), JUnit methods, etc.
* Add Logger class: log(), display_progress(), system load tracker.
* Remove WorkerJob class: the worker now gets a RunTests instance.

Move function and methods:

* Convert Regrtest.list_cases() to a function.
* Convert display_header(), display_sanitizers(), set_temp_dir() and
  create_temp_dir() methods of Regrtest to functions in utils.py.
  Rename set_temp_dir() to select_temp_dir(). Rename
  create_temp_dir() to make_temp_dir().
* Move abs_module_name() and normalize_test_name() to utils.py.
* Move capture_output() to utils.py.
* Rename dash_R() to runtest_refleak()
* Merge display_sanitizers() code into display_header().

cmdline.py changes:

* Rename internal --worker-args command line option to --worker-json.
* Rename ns.trace to ns.coverage.
* No longer gets the number of CPUs: it's now done by Regrtest class.
* Add missing attributes to Namespace: coverage, threshold, wait.

Misc:

* Add test_parse_memlimit() and test_set_memlimit() to test_support.
* Add some type annotations.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 5, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic
namespace which was used to get and set 'random' attributes. Instead,
each class has its own attributes, and some classes are even
read-only (frozen dataclass), like RunTests.

Reorganize files:

* Rename runtest.py to single.py.
* Rename runtest_mp.py to mp_runner.py.
* Create findtests.py, result.py, results.py and worker.py.

Reorganize classes:

* Copy command line options ('ns' namespace) to Regrtest
  and RunTests attributes: add many attributes.
* Add Results class with methods: get_state(), display_result(),
  get_exitcode(), JUnit methods, etc.
* Add Logger class: log(), display_progress(), system load tracker.
* Remove WorkerJob class: the worker now gets a RunTests instance.

Move function and methods:

* Convert Regrtest.list_cases() to a function.
* Convert display_header(), display_sanitizers(), set_temp_dir() and
  create_temp_dir() methods of Regrtest to functions in utils.py.
  Rename set_temp_dir() to select_temp_dir(). Rename
  create_temp_dir() to make_temp_dir().
* Move abs_module_name() and normalize_test_name() to utils.py.
* Move capture_output() to utils.py.
* Rename dash_R() to runtest_refleak()
* Merge display_sanitizers() code into display_header().

cmdline.py changes:

* Rename internal --worker-args command line option to --worker-json.
* Rename ns.trace to ns.coverage.
* No longer gets the number of CPUs: it's now done by Regrtest class.
* Add missing attributes to Namespace: coverage, threshold, wait.

Misc:

* Add test_parse_memlimit() and test_set_memlimit() to test_support.
* Add some type annotations.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 5, 2023
…#108839)

When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.

(cherry picked from commit 31c2945)
vstinner added a commit to vstinner/cpython that referenced this issue Sep 5, 2023
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").

(cherry picked from commit 1170d5a)
vstinner added a commit to vstinner/cpython that referenced this issue Sep 8, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic
namespace which was used to get and set 'random' attributes. Instead,
each class has its own attributes, and some classes are even
read-only (frozen dataclass), like RunTests.

Reorganize files:

* Rename runtest.py to single.py.
* Rename runtest_mp.py to mp_runner.py.
* Create findtests.py, result.py, results.py and worker.py.

Reorganize classes:

* Copy command line options ('ns' namespace) to Regrtest
  and RunTests attributes: add many attributes.
* Add Results class with methods: get_state(), display_result(),
  get_exitcode(), JUnit methods, etc.
* Add Logger class: log(), display_progress(), system load tracker.
* Remove WorkerJob class: the worker now gets a RunTests instance.

Move function and methods:

* Convert Regrtest.list_cases() to a function.
* Convert display_header(), display_sanitizers(), set_temp_dir() and
  create_temp_dir() methods of Regrtest to functions in utils.py.
  Rename set_temp_dir() to select_temp_dir(). Rename
  create_temp_dir() to make_temp_dir().
* Move abs_module_name() and normalize_test_name() to utils.py.
* Move capture_output() to utils.py.
* Rename dash_R() to runtest_refleak()
* Merge display_sanitizers() code into display_header().

cmdline.py changes:

* Rename internal --worker-args command line option to --worker-json.
* Rename ns.trace to ns.coverage.
* No longer gets the number of CPUs: it's now done by Regrtest class.
* Add missing attributes to Namespace: coverage, threshold, wait.

Misc:

* Add test_parse_memlimit() and test_set_memlimit() to test_support.
* Add some type annotations.
Sep 8, 2023
* gh-108834: regrtest reruns failed tests in subprocesses (#108839)

When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.

(cherry picked from commit 31c2945)

* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)

When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").

(cherry picked from commit 1170d5a)

* gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480)

(cherry picked from commit 1e0d627)

* Manually sync Lib/test/libregrtest/ from main

---------

Co-authored-by: Serhiy Storchaka <[email protected]>
vstinner added a commit to vstinner/cpython that referenced this issue Sep 8, 2023
* main() now calls _parse_args() and pass 'ns' to Regrtest
  constructor.  Remove kwargs argument from Regrtest.main().
* _parse_args() checks ns.huntrleaks.
* set_temp_dir() is now responsible to call expanduser().
* Regrtest.main() sets self.tests earlier.
* Add TestsList type.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 8, 2023
* main() now calls _parse_args() and pass 'ns' to Regrtest
  constructor.  Remove kwargs argument from Regrtest.main().
* _parse_args() checks ns.huntrleaks.
* set_temp_dir() is now responsible to call expanduser().
* Regrtest.main() sets self.tests earlier.
* Add TestTuple and TestList types.
* Rename MatchTests to MatchTestList and rename MatchTestsDict
  to MatchTestDict.
* RunTests.tests type becomes TestTuple.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

2 participants