Crazy idea: serde support

[birkenfeld]

A few times now, I had to take in complex-ish Python data and convert it into an equivalent in Rust. FromPyObject already does the basic types, but I'd also like to convert e.g. nested dicts with certain keys into structs.

Do you think it would make sense to let serde handle this? It already has all the options you'd like, e.g. flattening of sub-structure and renaming of things. For structs, it should support both dicts (taking keys) and other objects (taking attributes).

The "ser" direction could also be supported, although I don't need it, and it would probably only construct dicts for structs.

[ohadravid]

The "ser" direction could also be supported, although I don't need it, and it would probably only construct dicts for structs.

We actually wanted something like this, mostly when you want to return something which is 'data', but don't want (or can't) create a dedicated python wrapper for each struct.

[pganssle]

@birkenfeld I think that this sort of thing is a good idea, but is it really something that belongs in PyO3?

I think it makes sense to leave the core C API wrappers in PyO3 and break out useful additional functionality into their own crates.

[jmrgibson]

I took a stab at something like this, by implementing the custom serde traits, but then ran into lifetime issues with all the python data since everything relies on the interpreter lifetime. Not that it probably isn't possible, but It'll take someone with more experience than me.

I ended up just dumping everything to a raw json string and passing it over, which is a trivial amount of code on both python and rust sides, but not an elegant solution.

[gperinazzo]

I've made a crate to help deal with this. It let's you derive the necessary traits for PyO3 to convert a dict into your struct, as long as it knows how to convert every field.

I plan on having serde-like field attributes in the future, but if you're not using them the base functionality should beat passing things around as json.

[m3047]

I would encourage you all not to overthink this. So allow me to step back and do exactly that at a more abstract level I guess... :-/

Who is your audience? Is it people who are primarily writing in Rust or Python?

What are they struggling with? Are they struggling with learning one language or the other, or learning how to call something from Python using e.g. ctypes or have they stumbled on pyo3 and are they wondering how that works?

What are they trying to accomplish? Are they trying to translate from one language to the other? Do they feel the "need for speed" and contemplating trying something other than C? Are they primarily Rust programmers looking for a more transparent "glue" for some larger problem (see the popularity of embedded Lua for various tasks)?

I have a brick in the wall to offer as someone primarily writing in Python who had an optimization problem and stumbled across pyo3 and thought "well maybe I'll trying doing it in Rust rather than C": m3047/pyo3-deserializer

(A friend of mine who write primarily in Rust brought this issue to my attention.)

[davidhewitt]

One for the folks watching this: I've made a WIP crate which targets a full serde implementation for PyO3. See https://github.com/davidhewitt/pythonize

The idea is that it should be pretty much equivalent to serde_json but generates Python objects instead of string data.

Note that unlike dict-derive this means that it doesn't integrate very nicely with types that already implement IntoPy / FromPyObject - it's got to be serde all the way down. Maybe there's a way to solve that.

At the moment only the to-Python direction is implemented, but I'd be interested in taking feedback / help finishing this off.

[leplatrem]

I had a similar use-case, convert a PyObject into a serde_json::json::Value. It might be slightly off-topic here, but I thought it could be useful for someone looking for something similar and landing here :)
https://github.com/mozilla-services/python-canonicaljson-rs/blob/62599b246055a1c8a78e5777acdfe0fd594be3d8/src/lib.rs#L87-L167
(it was mostly inspired by Matthias Endler's hyperjson)

[kivo360]

I had a similar use-case, convert a PyObject into a serde_json::json::Value. It might be slightly off-topic here, but I thought it could be useful for someone looking for something similar and landing here :)
https://github.com/mozilla-services/python-canonicaljson-rs/blob/62599b246055a1c8a78e5777acdfe0fd594be3d8/src/lib.rs#L87-L167
(it was mostly inspired by Matthias Endler's hyperjson)

This is exactly what I needed.

[davidhewitt]

@kivo360 @leplatrem pythonize can also do similar (and more) using the depythonize API. It can convert PyObject into any type which implements Deserialize.

An interesting question I was wondering this week is whether we should provide serde wrappers for types like Py<T>.

[McSpidey]

Is there any chance pythonize would become a native part of PyO3?

[davidhewitt]

I haven't seen a need for it to become part of the main crate; what's the advantage that you see from doing so?

[McSpidey]

I'm new to PyO3 but even from just porting one simple app it seemed logical a native serialiser/deserialiser would be a fundamental core feature for efficient workflows, and serde is seemingly the defacto rust standard for this.

[davidhewitt]

That's a reasonable opinion. There's no performance or API disadvantages by having pythonize separate, and it reduces compile time for those who don't need it. For now I'm inclined to keep them separate, if there's a strong motivation to merge it in we can revisit.

[McSpidey]

Is there still a compile time disadvantage if it's implemented as a 'feature'?

[davidhewitt]

Probably not a huge difference for users, however the PyO3 CI already takes a long time, and I count that as a compile time disadvantage too!

Also worth bearing in mind that I hoped that anything which merged into PyO3 would have a solution for davidhewitt/pythonize#1 so that there would only be one integrated way to convert Rust -> Python objects, rather than two different mechanisms in the same crate.

I really do hold the opinion that pythonize is solving a different enough problem from PyO3 core that there's no need to make it part of the main API right now.

[fzyzcjy]

Hi, is there any chance to make pythonize work automatically? Or, what is the best practice when returning a complex nested struct? Thanks!

For example, I hope to use sth like:

#[derive(Debug, Serialize, Deserialize, PartialEq)]
struct Sample {
    foo: String,
    bar: Option<usize>
}

#[pymethods]
impl MyClass {
  fn f() -> Sample { ... }
}

... and it auto works (instead of manually calling pythonize).

[adamreichold]

I am not sure if we can make this work transparently as the trait bound T: Serialize would be quite broad. I guess, we could use a wrapper type like Pythonize(T) which would automatically call into pythonize behind the scenes (similar to e.g. the extractor and response wrappers provided by Axum), gated behind an optional integration feature as we use for other third-party crates.