Use higher precision time source in retry decorator/tests and avoid flaky failures #12839
Conversation
|
If time resolution is important, why not switch to On Windows: >>> mono = [time.monotonic() for x in range(10)]
>>> [mono[-1] - m for m in mono]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
>>> perf = [time.perf_counter() for x in range(10)]
>>> [perf[-1] - p for p in perf]
[3.4999102354049683e-06, 8.996576070785522e-07, 8.00006091594696e-07, 5.997717380523682e-07, 3.995373845100403e-07, 2.998858690261841e-07, 1.993030309677124e-07, 1.993030309677124e-07, 9.96515154838562e-08, 0.0] |
|
I mean, we could do that. I used Time resolution is not that important in practice, it's just that poor resolution becomes a pain when trying to write unit tests for the retry decorator. |
My understanding is Looking at PEP 418 the difference seems to be different API calls (GetTickCount64 vs. QueryPerformanceCounter), I do not know enough about Windows APIs to know much about the difference beyond googling them. But my personal experience is there is no noticable downside to
Yeah, this was a drive by suggestion, I didn't quite grok what was the issue here, just from my windows experience |
|
Yea, let's switch to |
|
FYI: This appears to be a bug in the python interpreter, |
Thanks, I just read through python/cpython#88494 and the tl;dr is Python 3.13 now uses precise timers on Windows for However, I don’t see any effort to backport so for now it seems pip must use |
578510a to
98cfa14
Compare
|
OK, I think this should actually be good now. I took a look at the other Windows CI failures and discovered that Sorry all. |
|
OK, so Footnotes
|
98cfa14 to
f5a6fd0
Compare
|
🙃 alright, let's just skip these tests entirely on Windows |
…laky failures
I've replaced time.monotonic() with time.perf_counter() in the
retry utility and test suite since it's effectively monotonic[^1] while
providing much higher resolution (esp. on Windows).
In addition, to avoid flaky failures (originally on Windows) I've added
a margin of error to the timed tests. I've also outright disabled the
timed tests on Windows as further testing revealed that they were simply
too unreliable to be useful (I once observed a deviation of 30%).
[^1]: It's not guaranteed to be monotonic in the Python docs, but it is
_indeed_ monotonic on all platforms we care about.
f5a6fd0 to
2bec84f
Compare
Despite my best efforts to account for Windows' coarse timer precision, the
test_retry_waitandtest_retry_time_limittests are still randomly failing as it isn't sleeping for long enough by a small margin:Example 1: 472.171 - 472.156 >= 0.015 (~1% too low)
Example 2: 554.406 - 554.265 >= 0.15 (~7% too low)
(call_time - test_start_time >= expected sleep duration between retries)
To avoid these random failures, I've adjusted the required sleep duration to be 10% less which should be enough.
I've also replaced
time.monotonic()withtime.perf_counter()in the retry utility and test suite since it's effectively monotonic1 while providing much higher resolution (esp. on Windows).@pradyunsg this should fix the CI failures on main. Sorry for the trouble!
Footnotes
It's not guaranteed to be monotonic in the Python docs, but it is
indeed monotonic on all platforms we care about. ↩