Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Polars to read and write rather than Pandas #56

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Use Polars to read and write rather than Pandas #56

wants to merge 13 commits into from

Commits on Mar 16, 2024

  1. Use polars to parse star file as a CSV 

    Polars requires a single character delimeter so first replace extended whitespace with a single whitespace character
    @d-j-hatton
    d-j-hatton committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    d92de11 View commit details
    Browse the repository at this point in the history

  2. Return Pandas DataFrame by default

    @d-j-hatton
    d-j-hatton committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    2927ef4 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. Fixes for empty strings and return Pandas DataFrames by default

    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    1df4d28 View commit details
    Browse the repository at this point in the history

  2. Use Polars to write DataFrames to star files. Currently only accpets … 

    …Pandas DataFrames and just converts to Polars
    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    f7f1255 View commit details
    Browse the repository at this point in the history

  3. DataBlocks may not contain Polars DataFrames rather than Pandas

    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    cea36b5 View commit details
    Browse the repository at this point in the history

  4. Allow write to take Polars DataFrames

    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    8cac0d8 View commit details
    Browse the repository at this point in the history

  5. Deal with string quotes in polars

    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    2690f61 View commit details
    Browse the repository at this point in the history

  6. Add tests for parsing into Polars DataFrames without converting to Pa… 

    …ndas
    @d-j-hatton
    d-j-hatton committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    3edefec View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. Perform whitespace shrinking as file is read on a per line basis and … 

    …slight efficiency improvements for that
    @d-j-hatton
    d-j-hatton committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    e56286d View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Add <NA> to null values list

    @d-j-hatton
    d-j-hatton committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    f59b55c View commit details
    Browse the repository at this point in the history

  2. Catch Polars as well as Pandas DataFrame

    @d-j-hatton
    d-j-hatton committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    81eda46 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Add polars dependency and make compatible with python versions before… 

    … 3.10
    @d-j-hatton
    d-j-hatton committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    1bee801 View commit details
    Browse the repository at this point in the history

  2. Small addition to docs

    @d-j-hatton
    d-j-hatton committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    0bb0f3e View commit details
    Browse the repository at this point in the history