Use Polars to read and write rather than Pandas

[d-j-hatton]

Using polars CSV read and write functionality rather than the equivalents in pandas can lead to significant speed ups. There is the added benefit that everything can be kept in polars for additional speed ups in dataframe manipulation later. To maintain the current behaviour polars dataframes are converted to pandas before returning from starfile.read by default. The additional keyword argument polars=True can be specified to return a polars.DataFrame. starfile.write will accept data blocks that are either pandas or polars dataframes.

As polars will only accept a single character separator arbitrary whitespace has to be parsed when the star file is read. Some modifications have therefore been made to the line by line parsing for efficiency.

Attached are some very rough read and write benchmarks from an M1 MacBook Pro on particle star files of different sizes. For read the bars are split into the time taken to perform read_csv and the rest. "Polars to pandas" and "Pure polars" refer to code modified as in this PR, "Pure pandas" is the existing implementation.

Notes:

• This adds polars as a dependency
• I couldn't get the pre-commits to pass because of existing code but some of them ran and changed formatting which I can try and change back if you want

[alisterburt]

What an awesome PR to wake up to! Will take a proper look once I'm up and running ☺️ thanks @d-j-hatton !

[jojoelfe]

Hey @d-j-hatton, are you still interested in this? I missed this, but think its a great idea. I've always been interested in polars and from looking at the documentation maybe there is even a way to make the quotation code a bit saner.

I've thought a bit about whether we should be cautious to add new dependencies, since this is a helper package used in quite a few other projects and this always increases the risk of collisions. Is there an easy way to make this an optional dependency?

[d-j-hatton]

Yes, it will be possible to have it as an optional dependency by adding an optional dependency group so you can pip install starfile[polars] and using some pattern like

try:
    import polars
except ImportError:
    polars = None

to check for the presence of the package in the code and default to pandas if not present. I can make some changes for that and fix the conflicts that have cropped up

[jojoelfe]

Thank you, that sounds great! Also happy to help, but won't touch anything unless you ask me to.