Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first attempt at an adjacently tagged union for consideration #94

Draft
wants to merge 60 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
921e8e4
first attempt at an adjacently tagged union for consideration
altendky Mar 12, 2020
4f98346
black
altendky Mar 12, 2020
14c7273
raise on extra keys when deserializing adjacently tagged
altendky Mar 12, 2020
cd0f7b6
remove trailing comma on single line, thanks black
altendky Mar 12, 2020
08c6e0c
correct TypeTagField.field type hint
altendky Mar 12, 2020
ddb5df4
add str and decimal.Decimal examples
altendky Mar 12, 2020
2492519
break by adding a couple of different list examples
altendky Mar 12, 2020
1ce7e67
remove explicit registry collision check
altendky Mar 13, 2020
cac94de
correct tests, better demonstrate failures, prep for additional regis…
altendky Mar 13, 2020
f2cffbe
black...
altendky Mar 13, 2020
84cd5ed
maybe interesting
altendky May 30, 2020
e76a271
use typeguard instead, for now at least
altendky May 31, 2020
18e78d3
black
altendky May 31, 2020
b233cf4
cleanup
altendky May 31, 2020
7a44b41
no protocol for now
altendky May 31, 2020
7731c00
remove duplicate code
altendky May 31, 2020
b30170e
pop a little
altendky May 31, 2020
21c9e1b
extract adjacent functions and generalize to TaggedUnion
altendky May 31, 2020
6a4bae8
group adjacently tagged functions
altendky May 31, 2020
8b97cf6
add externally tagged support
altendky May 31, 2020
f63f290
add internally tagged support
altendky May 31, 2020
1b41217
make tagged type and value keys configurable
altendky May 31, 2020
74494eb
complain if other than exactly one hint matches
altendky Jun 1, 2020
d5d7183
isinstance() for str vs. typing.Sequence[str] handling
altendky Jun 1, 2020
39104a7
Merge branch 'master' into 36-altendky-first_attempt
altendky Jul 29, 2021
f29ef21
black
altendky Jul 29, 2021
8633a8e
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 5, 2021
d76ec24
additional heuristics for List vs. Sequence, 3.7+ only
altendky Aug 5, 2021
4b4892f
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 5, 2021
39caf6f
make test examples frozen
altendky Aug 5, 2021
e615380
mypy
altendky Aug 5, 2021
3090a14
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 6, 2021
0e9c3a9
actually use typing-extensions
altendky Aug 6, 2021
419fd30
black
altendky Aug 6, 2021
7034ea2
check
altendky Aug 6, 2021
fc889f6
coverage
altendky Aug 6, 2021
408b21f
create specific exceptions
altendky Aug 6, 2021
cc9279f
check
altendky Aug 6, 2021
a6dfccd
specify field registry protocol and check it
altendky Aug 6, 2021
c7a385e
add test that looks like real user code
altendky Aug 6, 2021
d610e5e
just use Any for now
altendky Aug 6, 2021
47197f6
drop some args/kwargs to be more explicit
altendky Aug 6, 2021
c2cf8a6
docstrings and some more
altendky Aug 9, 2021
6722760
use typing_inspect.get_origin() instead of .__origin__
altendky Aug 17, 2021
47861f9
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 18, 2021
76bc4da
type hinting catchup
altendky Aug 19, 2021
918f2ab
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 19, 2021
aa85230
tagged union documentation
altendky Aug 23, 2021
64a20d9
add missing snippets
altendky Aug 23, 2021
c0f1ef6
Merge branch 'main' into 36-altendky-first_attempt
altendky Aug 24, 2021
82f92be
black
altendky Aug 24, 2021
2b8ca42
doc warning tidy
altendky Aug 24, 2021
c256d80
docutils < 0.17 for circular include error
altendky Aug 24, 2021
dc5d731
avoid trailing blank lines in example in docs
altendky Aug 24, 2021
bd64aa0
xfail for working-.__origin__-requiring tests on < 3.7
altendky Aug 24, 2021
f8a377f
parametrize against *_tagged_union[_from_registry] for coverage
altendky Aug 25, 2021
694000f
actually test tagged union examples
altendky Aug 30, 2021
d5acb63
add new example resources
altendky Aug 30, 2021
f6d2314
isort
altendky Aug 30, 2021
ea773b7
CatsAndDogs corrected to CatOrDog
altendky Sep 8, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ omit = *migrations*
exclude_lines =
# Lines matching these regexes don't need to be covered
# https://coverage.readthedocs.io/en/coverage-5.5/excluding.html?highlight=exclude_lines#advanced-exclusion

# this is the default but must be explicitly specified since
# we are overriding exclude_lines
pragma: no cover
Expand Down
8 changes: 7 additions & 1 deletion dev-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,9 @@ importlib-metadata==4.6.1 \
importlib-resources==5.2.2 \
--hash=sha256:2480d8e07d1890056cb53c96e3de44fead9c62f2ba949b0f2e4c4345f4afa977 \
--hash=sha256:a65882a4d0fe5fbf702273456ba2ce74fe44892c25e42e057aca526b702a6d4b
# via virtualenv
# via
# -r test-requirements.in
# virtualenv
incremental==21.3.0 \
--hash=sha256:02f5de5aff48f6b9f665d99d48bfc7ec03b6e3943210de7cfc88856d755d6f57 \
--hash=sha256:92014aebc6a20b78a8084cdd5645eeaa7f74b8933f70fa3ada2cfbd1e3b54321
Expand Down Expand Up @@ -600,6 +602,10 @@ typed-ast==1.4.3 \
# astroid
# black
# mypy
typeguard==2.12.1 \
--hash=sha256:c2af8b9bdd7657f4bd27b45336e7930171aead796711bc4cfc99b4731bb9d051 \
--hash=sha256:cc15ef2704c9909ef9c80e19c62fb8468c01f75aad12f651922acf4dbe822e02
# via -r requirements.in
typing-extensions==3.10.0.0 \
--hash=sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497 \
--hash=sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342 \
Expand Down
124 changes: 124 additions & 0 deletions docs/reference/fields.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
..
TODO: figure out proper location for this stuff including making it public
if in desert


desert._fields module
=====================

Tagged Unions
-------------

Serializing and deserializing data of uncertain type can be tricky.
Cases where the type is hinted with :class:`typing.Union` create such cases.
Some solutions have a list of possible types and use the first one that works.
This can be used in some cases where the data for different types is sufficiently unique as to only work with a single type.
In more general cases you have difficulties where data of one type ends up getting processed by another type that is similar, but not the same.

In an effort to reduce the heuristics involved in serialization and deserialization an explicit tag can be added to identify the type.
That is the basic feature of this group of utilities related to tagged unions.

A tag indicating the object's type can be applied in various ways.
Presently three forms are implemented: adjacently tagged, internally tagged, and externally tagged.
Adjacently tagged is the most explicit form and is the recommended default.
You can write your own helper functions to implement your own tagging form if needed and still make use of the rest of the mechanisms implemented here.
A code example follows the forms below and provides related reference.

- A class definition and bare serialized object for reference

.. literalinclude:: ../../tests/test_fields.py
:language: python
:start-after: # start cat_class_example
:end-before: # end cat_class_example

.. literalinclude:: ../../tests/example/untagged.json
:language: json


- Adjacently tagged

.. literalinclude:: ../../tests/example/adjacent.json
:language: json

- Internally tagged

.. literalinclude:: ../../tests/example/internal.json
:language: json

- Externally tagged

.. literalinclude:: ../../tests/example/external.json
:language: json

The code below is an actual test from the Desert test suite that provides an example usage of the tools that will be covered in detail below.

.. literalinclude:: ../../tests/test_fields.py
:language: python
:start-after: # start tagged_union_example
:end-before: # end tagged_union_example


Fields
......

A :class:`marshmallow.fields.Field` is needed to describe the serialization.
This role is filled by :class:`desert._fields.TaggedUnionField`.
Several helpers at different levels are included to generate field instances that support each of the tagging schemes shown above.
:ref:`Registries <tagged_union_registries>` are used to collect and hold the information needed to make the choices the field needs.
The helpers below create :class:`desert._fields.TaggedUnionField` instances that are backed by the passed registry.

.. autofunction:: desert._fields.adjacently_tagged_union_from_registry
.. autofunction:: desert._fields.internally_tagged_union_from_registry
.. autofunction:: desert._fields.externally_tagged_union_from_registry

.. autoclass:: desert._fields.TaggedUnionField
:members:
:undoc-members:
:show-inheritance:

The fields can be created from :class:`desert._fields.FromObjectProtocol` and :class:`desert._fields.FromTagProtocol` instead of registries, if need.

.. autofunction:: desert._fields.adjacently_tagged_union
.. autofunction:: desert._fields.internally_tagged_union
.. autofunction:: desert._fields.externally_tagged_union

.. autoclass:: desert._fields.FromObjectProtocol
:members: __call__
:undoc-members:
:show-inheritance:

.. autoclass:: desert._fields.FromTagProtocol
:members: __call__
:undoc-members:
:show-inheritance:


.. _tagged_union_registries:

Registries
..........

Since unions are inherently about handling multiple types, fields that handle unions must be able to make decisions about multiple types.
Registries are not required to leverage other pieces of union support if you are developing their logic yourself.
If you are using the builtin mechanisms then a registry will be needed to define the relationships between tags, fields, and object types.

..
TODO: sure seems like the user shouldn't need to call Nested() themselves

The registry's :meth:`desert._fields.FieldRegistryProtocol.register` method will primarily be used.
As an example, you might register a custom class ``Cat`` by providing a hint of ``Cat``, a tag of ``"cat"``, and a field such as ``marshmallow.fields.Nested(desert.schema(Cat))``.

.. autoclass:: desert._fields.FieldRegistryProtocol
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: desert._fields.TypeAndHintFieldRegistry
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: desert._fields.HintTagField
:members:
:undoc-members:
:show-inheritance:
1 change: 1 addition & 0 deletions docs/reference/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ Reference
:glob:

desert*
fields
2 changes: 2 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
sphinx~=4.1
# >= 1.0.0rc1 for https://github.com/readthedocs/sphinx_rtd_theme/issues/1115
sphinx-rtd-theme >= 1.0.0rc1
# < 0.17 for /home/docs/checkouts/readthedocs.org/user_builds/desert/checkouts/94/src/desert/_fields.py:docstring of desert._fields.externally_tagged_union_from_registry:4: WARNING: circular inclusion in "include" directive: snippets/tag_forms/external.rst < snippets/tag_forms/internal.rst < snippets/tag_forms/adjacent.rst < snippets/tag_forms/external.rst < reference/fields.rst
docutils < 0.17
sphinx-autodoc-typehints
-e .[dev]
2 changes: 2 additions & 0 deletions requirements.in
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ marshmallow>=3.0
attrs
typing_inspect
dataclasses; python_version < "3.7"
#https://github.com/Stewori/pytypes/archive/b7271ec654d3553894febc6e0d8ad1b0e1ac570a.zip
typeguard
4 changes: 4 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ mypy-extensions==0.4.3 \
--hash=sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d \
--hash=sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8
# via typing-inspect
typeguard==2.12.1 \
--hash=sha256:c2af8b9bdd7657f4bd27b45336e7930171aead796711bc4cfc99b4731bb9d051 \
--hash=sha256:cc15ef2704c9909ef9c80e19c62fb8468c01f75aad12f651922acf4dbe822e02
# via -r requirements.in
typing-extensions==3.10.0.0 \
--hash=sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497 \
--hash=sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342 \
Expand Down
Loading