Thanks! Would it make sense to create an alias for that function called transpose, which also creates a new dataframe from it? If not it might be a good idea to add the keyword "transpose" in the description somewhere of rows so searching it yields it.

Yes, will add the alias. 👍

@ritchie46 I'm wondering whether it would make sense to introduce a lazy transpose - skipping the extra allocation (turning it into an iterator) if there is a subsequent operation. Edit: this might fit ndarray more, eg it could optimize DF * DF.transpose()

Now that I think of it. I will support this natively, as going to python rows is super expensive.

@ritchie46 I'm wondering whether it would make sense to introduce a lazy transpose - skipping the extra allocation (turning it into an iterator) if there is a subsequent operation. Edit: this might fit ndarray more, eg it could optimize DF * DF.transpose()

@alippai Currently these operation sadly cannot be done in lazy. I need to know the schema of every node in the query plan. An operation like pivot, and transpose create the schema based on the data (which is unknown at that point).

Makes sense, I really appreciate the implementation detail!

fwiw, spark supports it (lazily), but it is a shotgun shot to the foot, as it performs two queries, one of them to compute the distincts during planning.

Just to get the complexity of the task: an n x m sized 2D single type (int/float) specialization of the DF type would be needed for this, right?

fwiw, spark supports it (lazily), but it is a shotgun shot to the foot, as it performs two queries, one of them to compute the distincts during planning.

Ouch.. that's definitely a shotgun.

In such I cases, I'd rather have a user doing something like this, and document why'd they want this:

temp = long_query().transpose().collect()

(temp.lazy()
 .select(..)   # continue from here.
 )

Just to get the complexity of the task: an n x m sized 2D single type (int/float) specialization of the DF type would be needed for this, right?

For max performance I guess something like that. I am planning to use the AnyValue enums (these are enums around all possible dtypes). Then we can turn the DataFrame into rows. Infer the schema per row and read the rows as columns.

Added in 2fa53db