Discussion: how to implement vectorized validations on struct columns?

[dkapitan]

We are looking for a solution for performance validation of healthcare data in the FHIR format. Details are described here:
beda-software/FHIRPathMappingLanguage#18

Patito looks very interesting and I was wondering whether you could say something about vectorized validation of struct columns with polars and patito?

Basic idea

We want a performant solution for tabular legacy --> FHIR mappings like this (note: this code doesn't work):

import pandera as pa
from pandera.engines.pandas_engine import PydanticModel # PydanticModel only available in pandas_engine
import polars as pl
from resources import Patient

class PatientSchema(pa.DataFrameModel):
    """Pandera schema using the pydantic model."""

    class Config:
        """Config with dataframe-level data type."""

        dtype = PydanticModel(Patient)
        coerce = True  # this is required, otherwise a SchemaInitError is raised

patient = pl.read_json("general-person-example.json")
PatientSchema.validate(patient)

Current issues

• PydanticModel still needs to be implemented in polars_engine, see Add PydanticModel to pandera.engines.polars_engine (polars engine support for pydantic models) unionai-oss/pandera#1874
• Even when this is implemented, it does row-wise validation and doesn't use the vectorization of polars. To leverage that, we need to extend the FHIR types with column-level checks in polars

tagging @ir4y and @yannick-vinkesteijn