Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON serialize and deserialize polars datatypes #35

Closed
wants to merge 21 commits into from
Closed

JSON serialize and deserialize polars datatypes #35

wants to merge 21 commits into from

Conversation

thomasaarholt
Copy link
Collaborator

  • Downgrade Sphinx to compatible version
  • very rough draft for pydantic v2
  • fix: DataFrameValidationError mirrors pydantic v1 ValidationError
  • wip: subclass FieldInfo, extend with patito field attributes
  • fix: update LDF.collect() for polars==0.19.8
  • wip: restructure model reconstitution routines, all tests passing
  • chore: lockfile and formatting
  • fix: handle multiple annotations when dtype is present in Field
  • check: annotated dtypes match those specified in Field.dtype
  • chore: misc typing improvements
  • fix: search for nested field constraints on validation, ignore nulls
  • fix: better inference for column nullability
  • wip: better type mismatch messaging, move Representation to internal
  • wip: onto classproperty formulation in lieu of metaclass
  • wip: custom classproperty (from polars)
  • wip: robustify array dtype inference, add pt custom fields to Field()
  • feature: custom Field/FieldInfo constructors for better extensibility
  • fix: make _PatitoFieldInfo private, ensure ext fields -> _schema_props
  • wip: serialization
  • wip: modularize annotation -> dtype and dtype validation
  • Example serializing and deserializing polars datatypes

JakobGM and others added 21 commits October 26, 2023 12:42
@JakobGM @brendancooley
Downgrade Sphinx to compatible version
a5ddd94
@brendancooley
very rough draft for pydantic v2
2287188
@brendancooley
fix: DataFrameValidationError mirrors pydantic v1 ValidationError
42299db 
test_validators passes with exception of tests containing `json_schema_extra`
@brendancooley
wip: subclass FieldInfo, extend with patito field attributes
4b98da9 
all `test_validators` now succeed, all but `test_dataframe_get_method` now pass in `test_polars`
@brendancooley
fix: update LDF.collect() for polars==0.19.8
eef74fe 
fix: switch to *args/**kwargs
@brendancooley
wip: restructure model reconstitution routines, all tests passing
87d57bd
@brendancooley
chore: lockfile and formatting
ffe343d 
chore: cleanup init

chore: cleanup init

chore: more misc cleanup

cleanup json_schema_extras
@brendancooley
fix: handle multiple annotations when dtype is present in Field
e2bf0d7
@brendancooley
check: annotated dtypes match those specified in Field.dtype
017c59b
@brendancooley
chore: misc typing improvements
161300b
@brendancooley
fix: search for nested field constraints on validation, ignore nulls
9e132bc
@brendancooley
fix: better inference for column nullability
93e171e
@brendancooley
wip: better type mismatch messaging, move Representation to internal
e9dbca6
@brendancooley
wip: onto classproperty formulation in lieu of metaclass
399673e 
not supported python<3.9, will break in python>=3.13
at that point, we can roll our own following polars: https://github.com/pola-rs/polars/blob/8d29d3cebec713363db4ad5d782c74047e24314d/py-polars/polars/datatypes/classes.py#L25C12-L25C12
@brendancooley
wip: custom classproperty (from polars)
a63db3f
@brendancooley
wip: robustify array dtype inference, add pt custom fields to Field()
db44aa3
@brendancooley
feature: custom Field/FieldInfo constructors for better extensibility
438974c
@brendancooley
fix: make _PatitoFieldInfo private, ensure ext fields -> _schema_props
386ba83
@brendancooley
wip: serialization
ee63672 
wip: ColumnInfo API

cleanup
@brendancooley
wip: modularize annotation -> dtype and dtype validation
daf8f59 
working to minimize references to _schema_properties and work with more structured and customizable objects. should be easier to add new supported dtypes with this framework and debug in future

wip: tests -> new dtype inference engine

wip: dtype docstrings
@thomasaarholt
Example serializing and deserializing polars datatypes
f572b43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@thomasaarholt @brendancooley @ion-elgreco @JakobGM