Schema version field prevents validating older versioned data

[dbirman]

Because we encode the version explicitly in the dumped models it's impossible to validate old data (that might actually be valid) with the current schema directly, unless you remove the field. Would be great to get a solution built into the schema for this, possibly annotating the version field with https://docs.pydantic.dev/2.0/api/functional_validators/#pydantic.functional_validators.SkipValidation

[dbirman]

Actually it's impossible to validate even with the field removed, I think it has to get explicitly set to the correct value, I guess pulled from the main schema maybe?

[bruno-f-cruz]

Check here for a possible solution:
Implementation: https://github.com/AllenNeuralDynamics/Aind.Behavior.Services/blob/9629b92a08878c8adf3b58de3240017dc43196b9/src/DataSchemas/aind_behavior_services/session/__init__.py#L27

Test:
https://github.com/AllenNeuralDynamics/Aind.Behavior.Services/blob/main/tests/test_schema_version_coercion.py

The strategy is to assume that the only incompatibility is a field that is coercable. The value will be updated and the rest of the schema runs as is. If it crashes one assumes it is not compatible if it doesn't we assume it is. Once deserialized without throwing, you assume that the deserialized object is from the latest library version.