keep_none option to skip Field._serialize() on None values

[PhilMarsh]

I have optional attrs in some of my app-level objects.
My app-level objects are fully explicit: missing optional fields are explicitly set to None instead of being left unset.
I want my Schema to be similarly explicit on dump (ie: never skip fields), while being permissive on load (ie: allow missing fields as None).

So the behavior I want for my Fields is:
A) on load, missing or None is deserialized to None.
B) on dump, None is serialized to None.

Requirement A is easily satisfied by missing=None.
Requirement B requires more work. For custom types, I must add an is None check to the override of Field._serialize().

This asymmetry of load and dump behavior is strange to me for a couple reasons:

1. Load-time optionality is controlled dynamically via Field parameters, which are stated in the Schema.
   But dump-time optionality is controlled statically by the Field implementation, forcing all Schemas to accept the same behavior (or use some sort of Field factory to customize behavior). I think it is common to have types which are optional in some Schemas and required in other Schemas.
2. None may be a valid attr value on the parent object/Schema, but None is not a valid value for my custom Field's data type. For example, given:

   Foo.bar = Bar()
   

   Assuming .bar is optional, Foo.bar = None is a valid assignment. But None is not a valid Bar object. So a BarField should not be responsible for serializing a None. I would actually expect BarField to raise some TypeError-like if it was given a None to serialize.

My proposal is to add a keep_none parameter to Field to make this dump behavior configurable.
Behavior is:

• During serialization, if an attr is found to be None and keep_none is True, then return None as the serialized value and do not call Field._serialize().

This seems similar to #229, except I don't want to skip the None Fields; I want to keep them as None in the serialized output.

EDIT 1: Requirement A is satisfied by missing=None (which implies allow_none=True), not by allow_none=True alone.

[deckar01]

Does setting default to None work for your use case?

https://marshmallow.readthedocs.io/en/3.0/api_reference.html#marshmallow.fields.Field

from marshmallow import fields, Schema

class Test(Schema):
    foo = fields.Str(missing=None, default=None)

schema = Test()
obj = schema.load({})
# {'foo': None}
schema.dump(obj)
# {'foo': None}

[PhilMarsh]

@deckar01 Sadly no, for 2 reasons:

• default is only triggered when the attr is missing from the object (mine exist as None)
• default is a pre-serialization value which is then passed to Field._serialize(). Field._serialize() is only skipped when default=marshmallow.missing, but then that causes the entire field to be missing from the serialized output (I want a None in the output).

From: https://github.com/marshmallow-code/marshmallow/blob/3.0/marshmallow/fields.py#L245

EDIT 1: link to 3.0 code, not dev

[deckar01]

Oh, missed that it was an issue for custom fields. Thanks. Ya, the if value is None: return None pattern is repeated all throughout fields.py. It is also shown in the custom fields example.

https://marshmallow.readthedocs.io/en/3.0/custom_fields.html#creating-a-field-class

[PhilMarsh]

I didn't even realize it was in most/all of the native Fields. That's actually a bit concerning for me. It means my field serialization is missing some constraints I would like to have.

[sloria]

@PhilMarsh so is your issue solved by updating your custom fields' _serialize methods to serialize None to None?

[sloria]

On a related note: #1381 implements a missing_values parameter that defines missing values for deserialization. See here for what the usage would look like.

It seems like you're looking for an analagous API for serialization, like the default_values proposed here. But I'm wondering if it's necessary for your specific case since all fields already serialize None to None.

[sloria]

Closing for now. Please comment if this is still an issue