-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix!: Fix NaN
ordering to make NaNs compare greater than any other float, and equal to themselves
#12721
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few minor comments on the Python side!
NaN
ordering to make NaNs compare greater than any other float, and equal to themselves
can I treat NaN as inf when use polars? because when you do rolling_mean, there maybe a lot of NaN |
If you want to treat NaN as inf you could use |
Does this not explicity break the IEEE 754 NaN comparison rules? What's the thought process behind doing this? |
It does. See explanation: |
This PR makes floats have a total ordering in their comparison operator, where NaNs compare equal to each other (regardless of sign or payload) and greater than any other floating point value. This matches the order used in e.g. duckdb and allows unambiguous sorting, hashing and equality. Ironically,
sorting/hashing is not yet included in this PR, just the explicit comparison operators, one step at a time.Also included in this PR is the new
polars-compute
crate that is intended to contain pure computational functions that do not requirepolars-core
(but may usepolars-arrow
). We can start moving other things (such as the efficient summation routine) here after this PR.Example
Before:
After:
>>> s == s shape: (3,) Series: '' [bool] [ true true true ]