-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Redesign logical types #1489
Redesign logical types #1489
Conversation
aea523e
to
294c539
Compare
be47a97
to
f9d0ee0
Compare
Looks solid, great work! One general comment is that this approach (the previous also) may make it difficult to support timestamps with timezones (more generally, logical types whose number of variants is very large and can't be enumerated). To address this in arrow2, what I did was to not rely on a trait to map logical information. For example:
if we use something like
the In other words, for timezones, we will need to implement something like
and there won't be any
may not make sense when two timezones are not equal (i.e. when I think we can figure out this later, just pointing out that this was one of the things that I went back and forth in arrow2. |
Agreed, the runtime
Yes, there are probably a lot of edge cases, once timezones come in to play. Maybe its a good idea to cast the Thanks for the review, good to know we are on the good path for this. |
f9d0ee0
to
ae7d5e6
Compare
This sets up a new architecture for logical types. Allowing for more data type support without increasing compile times. The basic ideas, is that polars only uses the primitive types and converts to the logical arrays upon the boundary between polars - arrow.