-
-
Notifications
You must be signed in to change notification settings - Fork 30.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
regrtest should re-run failed tests in subprocesses #108834
Labels
Comments
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
* Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Add TestResult.get_rerun_match_tests() method * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * run_tests_worker() does also parses the JSON. * If failed test names cannot be parsed, log a warning and don't filter tests. * Add tests on normalize_test_name() function.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Fix statistics in regrtest summary. Changes: * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 3, 2023
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Fix and enhance statistics in regrtest summary. Add "(filtered)" when --match and/or --ignore options are used. * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function.
vstinner
added a commit
that referenced
this issue
Sep 3, 2023
When using --rerun option, regrtest now re-runs failed tests in verbose mode in fresh worker processes to have more deterministic behavior. So it can write its final report even if a test killed a worker progress. Add --fail-rerun option to regrtest: exit with non-zero exit code if a test failed pass passed when re-run in verbose mode (in a fresh process). That's now more useful since tests can pass when re-run in a fresh worker progress, whereas they failed when run after other tests when tests are run sequentially. Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Fix and enhance statistics in regrtest summary. Add "(filtered)" when --match and/or --ignore options are used. * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function. * Add test_success() and test_skip() tests to test_regrtest.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 4, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic namespace which was used to get and set 'random' attributes. Instead, each class has its own attributes, and some classes are even read-only (frozen dataclass), like RunTests. Reorganize files: * Rename runtest.py to single.py. * Rename runtest_mp.py to mp_runner.py. * Create findtests.py, result.py, results.py and worker.py. Reorganize classes: * Copy command line options ('ns' namespace) to Regrtest and RunTests attributes: add many attributes. * Add Results class with methods: get_state(), display_result(), get_exitcode(), JUnit methods, etc. * Add Logger class: log(), display_progress(), system load tracker. * Remove WorkerJob class: the worker now gets a RunTests instance. Move function and methods: * Convert Regrtest.list_cases() to a function. * Convert display_header(), display_sanitizers(), set_temp_dir() and create_temp_dir() methods of Regrtest to functions in utils.py. Rename set_temp_dir() to select_temp_dir(). Rename create_temp_dir() to make_temp_dir(). * Move abs_module_name() and normalize_test_name() to utils.py. cmdline.py changes: * Rename internal --worker-args command line option to --worker-json. * Rename ns.trace to ns.coverage. * No longer gets the number of CPUs: it's now done by Regrtest class. * Add missing attributes to Namespace: coverage, threshold, wait. Misc: * Add test_parse_memlimit() and test_set_memlimit() to test_support. * Add some type annotations.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 4, 2023
When the --fail-rerun option is used a test failed and then passed, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 4, 2023
When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner
added a commit
that referenced
this issue
Sep 5, 2023
When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 5, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic namespace which was used to get and set 'random' attributes. Instead, each class has its own attributes, and some classes are even read-only (frozen dataclass), like RunTests. Reorganize files: * Rename runtest.py to single.py. * Rename runtest_mp.py to mp_runner.py. * Create findtests.py, result.py, results.py and worker.py. Reorganize classes: * Copy command line options ('ns' namespace) to Regrtest and RunTests attributes: add many attributes. * Add Results class with methods: get_state(), display_result(), get_exitcode(), JUnit methods, etc. * Add Logger class: log(), display_progress(), system load tracker. * Remove WorkerJob class: the worker now gets a RunTests instance. Move function and methods: * Convert Regrtest.list_cases() to a function. * Convert display_header(), display_sanitizers(), set_temp_dir() and create_temp_dir() methods of Regrtest to functions in utils.py. Rename set_temp_dir() to select_temp_dir(). Rename create_temp_dir() to make_temp_dir(). * Move abs_module_name() and normalize_test_name() to utils.py. * Move capture_output() to utils.py. * Rename dash_R() to runtest_refleak() * Merge display_sanitizers() code into display_header(). cmdline.py changes: * Rename internal --worker-args command line option to --worker-json. * Rename ns.trace to ns.coverage. * No longer gets the number of CPUs: it's now done by Regrtest class. * Add missing attributes to Namespace: coverage, threshold, wait. Misc: * Add test_parse_memlimit() and test_set_memlimit() to test_support. * Add some type annotations.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 5, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic namespace which was used to get and set 'random' attributes. Instead, each class has its own attributes, and some classes are even read-only (frozen dataclass), like RunTests. Reorganize files: * Rename runtest.py to single.py. * Rename runtest_mp.py to mp_runner.py. * Create findtests.py, result.py, results.py and worker.py. Reorganize classes: * Copy command line options ('ns' namespace) to Regrtest and RunTests attributes: add many attributes. * Add Results class with methods: get_state(), display_result(), get_exitcode(), JUnit methods, etc. * Add Logger class: log(), display_progress(), system load tracker. * Remove WorkerJob class: the worker now gets a RunTests instance. Move function and methods: * Convert Regrtest.list_cases() to a function. * Convert display_header(), display_sanitizers(), set_temp_dir() and create_temp_dir() methods of Regrtest to functions in utils.py. Rename set_temp_dir() to select_temp_dir(). Rename create_temp_dir() to make_temp_dir(). * Move abs_module_name() and normalize_test_name() to utils.py. * Move capture_output() to utils.py. * Rename dash_R() to runtest_refleak() * Merge display_sanitizers() code into display_header(). cmdline.py changes: * Rename internal --worker-args command line option to --worker-json. * Rename ns.trace to ns.coverage. * No longer gets the number of CPUs: it's now done by Regrtest class. * Add missing attributes to Namespace: coverage, threshold, wait. Misc: * Add test_parse_memlimit() and test_set_memlimit() to test_support. * Add some type annotations.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 5, 2023
…#108839) When using --rerun option, regrtest now re-runs failed tests in verbose mode in fresh worker processes to have more deterministic behavior. So it can write its final report even if a test killed a worker progress. Add --fail-rerun option to regrtest: exit with non-zero exit code if a test failed pass passed when re-run in verbose mode (in a fresh process). That's now more useful since tests can pass when re-run in a fresh worker progress, whereas they failed when run after other tests when tests are run sequentially. Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Fix and enhance statistics in regrtest summary. Add "(filtered)" when --match and/or --ignore options are used. * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function. * Add test_success() and test_skip() tests to test_regrtest. (cherry picked from commit 31c2945)
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 5, 2023
When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test"). (cherry picked from commit 1170d5a)
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
Remove "global variables" from libregrtest: no longer pass 'ns' magic namespace which was used to get and set 'random' attributes. Instead, each class has its own attributes, and some classes are even read-only (frozen dataclass), like RunTests. Reorganize files: * Rename runtest.py to single.py. * Rename runtest_mp.py to mp_runner.py. * Create findtests.py, result.py, results.py and worker.py. Reorganize classes: * Copy command line options ('ns' namespace) to Regrtest and RunTests attributes: add many attributes. * Add Results class with methods: get_state(), display_result(), get_exitcode(), JUnit methods, etc. * Add Logger class: log(), display_progress(), system load tracker. * Remove WorkerJob class: the worker now gets a RunTests instance. Move function and methods: * Convert Regrtest.list_cases() to a function. * Convert display_header(), display_sanitizers(), set_temp_dir() and create_temp_dir() methods of Regrtest to functions in utils.py. Rename set_temp_dir() to select_temp_dir(). Rename create_temp_dir() to make_temp_dir(). * Move abs_module_name() and normalize_test_name() to utils.py. * Move capture_output() to utils.py. * Rename dash_R() to runtest_refleak() * Merge display_sanitizers() code into display_header(). cmdline.py changes: * Rename internal --worker-args command line option to --worker-json. * Rename ns.trace to ns.coverage. * No longer gets the number of CPUs: it's now done by Regrtest class. * Add missing attributes to Namespace: coverage, threshold, wait. Misc: * Add test_parse_memlimit() and test_set_memlimit() to test_support. * Add some type annotations.
* gh-108834: regrtest reruns failed tests in subprocesses (#108839) When using --rerun option, regrtest now re-runs failed tests in verbose mode in fresh worker processes to have more deterministic behavior. So it can write its final report even if a test killed a worker progress. Add --fail-rerun option to regrtest: exit with non-zero exit code if a test failed pass passed when re-run in verbose mode (in a fresh process). That's now more useful since tests can pass when re-run in a fresh worker progress, whereas they failed when run after other tests when tests are run sequentially. Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Fix and enhance statistics in regrtest summary. Add "(filtered)" when --match and/or --ignore options are used. * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function. * Add test_success() and test_skip() tests to test_regrtest. (cherry picked from commit 31c2945) * gh-108834: regrtest --fail-rerun exits with code 5 (#108896) When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test"). (cherry picked from commit 1170d5a) * gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480) (cherry picked from commit 1e0d627) * Manually sync Lib/test/libregrtest/ from main --------- Co-authored-by: Serhiy Storchaka <[email protected]>
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
* main() now calls _parse_args() and pass 'ns' to Regrtest constructor. Remove kwargs argument from Regrtest.main(). * _parse_args() checks ns.huntrleaks. * set_temp_dir() is now responsible to call expanduser(). * Regrtest.main() sets self.tests earlier. * Add TestsList type.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
* main() now calls _parse_args() and pass 'ns' to Regrtest constructor. Remove kwargs argument from Regrtest.main(). * _parse_args() checks ns.huntrleaks. * set_temp_dir() is now responsible to call expanduser(). * Regrtest.main() sets self.tests earlier. * Add TestTuple and TestList types. * Rename MatchTests to MatchTestList and rename MatchTestsDict to MatchTestDict. * RunTests.tests type becomes TestTuple.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature or enhancement
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Proposal:
Buildbots and GitHub Actions run the Python test suite with
--verbose2
option to re-run failed tests in verbose mode. The current regrtest implementation is the re-run tests in the main process. It is not reliable: if a test crash or timeout, the main process cannot get back the control since the process is killed. Usually, there is no final report after the test is re-run, and the output is just "truncated".I propose to re-run failed tests in subprocesses to make them more reliable:
I'm working on an implementation.
Linked PRs
The text was updated successfully, but these errors were encountered: