Clean way of "Postelizing" callbacks

[thorwhalen]

Described here is a design proposal to balance explicitness and UX of callbacks (more precisely, callable arguments that parametrize the behavior of python objects). The scope of this design goes beyond dol, but we'll use dol examples, since this is where we want to solve the problem first.

Context

At the time of writing this, wrap_kvs has some design problems. One of its syndromes is mentioned in the wrap_kvs inconsistencies (due to incorrect signature-based conditioning) issue, and is probably the cause of some other problems.

The source of this is some not-well-thought-out "postelization" of the input "transformation" functions. The code looks at the signature of the functions to try to decide on how to apply it.

Why is it that (see this issue), as arguments of kv_wraps, obj_of_data=lambda x: bytes.decode(x) and obj_of_data=bytes.decode lead to different behaviors (the second one raises an error). In fact, obj_of_data=lambda self: bytes.decode(self) would lead to that same error too. All of these definitions of obj_of_data are functionally equivalent in that you get the same outputs for the same inputs, so we really shouldn't have a divergence of behaviors here.

The problem is that wrap_kvs has this horrible thing where it looks at the signature of obj_of_data and applies the function differently in both cases. More specifically, the default behavior is to do obj = obj_of_data(data), but if obj_of_data has at least two required arguments (i.e. have no default), or the first argument name is "self", is will instead use it as obj = obj_of_data(self, data)!

Why does it do that? Because though the first behavior seems to be the most common, sometimes we need obj_of_data to have access to the store instance (self) to do it's job (for example, seeing self.rootdir to use that in it's transformation.

A better solution (and it's problems)

kv_walk on the other hand has a good "engineering" design. The signature is:

def kv_walk(
    v: Mapping,
    yield_func: Callable[[PT, KT, VT], Any] = asis,
    walk_filt: Callable[[PT, KT, VT], bool] = val_is_mapping,
    pkv_to_pv: Callable[[PT, KT, VT], Tuple[PT, VT]] = tuple_keypath_and_val,
    p: PT = (),
) -> Iterator[Any]:
   ...

Check out the code and see how simple it is.
And yet, kv_walk can be parametrized to do pretty much any kind of nested-structure processing you can imagine.
It is no doubt the best design as far as engineering goes. But what about the UX?

No doubt, such functions as kv_walk are destined to be used in specialized functions that are more user friendly, but still, see that this raw interface obliges the user to express pkv_to_pv "fully".
Even if all that is needed is a function foo from, say, p to v, the user would have express it to kv_walk as pkv_to_pv=lambda p, k, v: (p, foo(p)), or walk_filt=lambda p, k, v: foo(p)

This is the problem that wrap_kvs tried to solve (badly) with it's conditional acrobatics.
It tried to make it easy and natural for the user to use, for the most common use cases.

It's a noble cause to strive for a good development UX.
And it's not only about making it easy for the user: It has positive effects on the behavior of the system as well.

I wrote more about general problem of UX vs explicit compromise here, so won't repeat myself.
I will, how ever look into the particular problem of input functions/callbacks here.

Let's now see how we can make our better solution even better: Getting a better UX without sacrificing too much (if any) design robustness.

call forgivingly? Nah!

One solution would be to use i2.call_forgivingly.
This function allows you to extract the inputs a function needs via their variable names.

That is, doing obj = call_forgivingly(obj_of_data, self=self, data=data) in the code of wrap_kvs would enable any obj_of_data to have access to self and data, extracting only what it needs from it.

Still, this isn't the best from a UX point of view: The user's callbacks' argument names need to comply to the expected convention (have argument names self and/or data)!

It's also not great from a robustness point of view (what if a function used the argument name "data", but didn't mean that data (in the context of the call_forgivingly call).

wrap in a binder

What if the user had at their disposal a class to wrap their function to give it the required single-stable interface (like the one kv_walk has).

This resembles the bind part of a FuncNode or the InnerMapIngress (which we might actually want to use here).

Essentially, we'd be able to do things like (but this is not my final interface for this!)

obj_of_data = Binder('self data')(bytes.decode, self=0)
walk_filt = Binder('p k v')(foo, p=0)

That's a worse UX than what we use in kv_walk, and probably worse in robustness as well (more layers, less simplicity). But the set up opens the possibility of some desirable abilities.

Things like having an object that encapsulates the concern of mapping external functions to the expected form for a callback.

Things like doing:

apply_to_p = functools.partial(Binder('p k v'), p=0)

and then offer the apply_to_p helper function to the user to use as apply_to_p(foo) instead of having to do the lambda p, k, v: foo(p) (which seems convoluted, especially if names are long, and is not picklable) every time.

Still, I'm not convinced it's worth it...