Howto return a PyBytes with the new Bound interface of pyo3

[brmmm3]

I'm trying to return a PyBytes object with the new Bound interface, but the Rust compiler complains about a lifetime problem:

fn to_vec(&self, py: Python) -> PyResult<Bound<PyBytes>> {
        match self.instance.statistics.to_vec() {
            Ok(v) => {
                Ok(
                    PyBytes::new_bound_with(py, v.len(), |b| {
                        b.copy_from_slice(&v);
                        Ok(())
                    })?
                )
            }
            Err(e) => Err(PyException::new_err(e.to_string())),
        }
    }

to_vec returns a Vec<u8>. I'm confused about the lifetime problem and don't know how to solve it.

The compiler error is:

error: lifetime may not live long enough
   --> pyscandir/src/count.rs:126:17
    |
123 |       fn to_vec(&self, py: Python) -> PyResult<Bound<PyBytes>> {
    |                 -      -- has type `pyo3::Python<'1>`
    |                 |
    |                 let's call the lifetime of this reference `'2`
...
126 | /                 Ok(
127 | |                     PyBytes::new_bound_with(py, v.len(), |b| {
128 | |                         b.copy_from_slice(&v);
129 | |                         Ok(())
130 | |                     })?
131 | |                 )
    | |_________________^ method was supposed to return data with lifetime `'2` but it is returning data with lifetime `'1`

[brmmm3]

I've found the solution:

fn to_vec(&self, py: Python) -> PyResult<Py<PyBytes>> {
        match self.instance.statistics.to_vec() {
            Ok(v) => {
                Ok(
                    PyBytes::new_bound_with(py, v.len(), |b| {
                        b.copy_from_slice(&v);
                        Ok(())
                    })?.into()
                )
            }
            Err(e) => Err(PyException::new_err(e.to_string())),
        }
    }

[birkenfeld]

The solution with Bound here is to mark explicitly that the lifetime of the Bound reference is the one of the Python marker:

fn to_vec<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyBytes>> {

because it's otherwise inferred to be the one of &self.

@davidhewitt are these lifetime inference problems new with Bound? Do/Should we have a chapter in the guide about it?

[brmmm3]

Is my solution with Py also correct or is it wrong?

[adamreichold]

Is my solution with Py also correct or is it wrong?

It is not incorrect, but it should not be necessary to replace Bound<'py, T> by its detached form Py<T> as it looses information (about still being tied to the GIL).

are these lifetime inference problems new with Bound?

The version using a GIL ref

fn to_vec(&self, py: Python) -> PyResult<&PyBytes> {
    match self.to_buf() {
        Ok(v) => Ok(PyBytes::new_with(py, v.len(), |b| {
            b.copy_from_slice(&v);
            Ok(())
        })?),
        Err(e) => Err(PyException::new_err(e.to_string())),
    }
}

yields the same error

error: lifetime may not live long enough
  --> src/main.rs:17:22
   |
15 |       fn to_vec(&self, py: Python) -> PyResult<&PyBytes> {
   |                 -      -- has type `pyo3::Python<'1>`
   |                 |
   |                 let's call the lifetime of this reference `'2`
16 |           match self.to_buf() {
17 |               Ok(v) => Ok(PyBytes::new_with(py, v.len(), |b| {
   |  ______________________^
18 | |                 b.copy_from_slice(&v);
19 | |                 Ok(())
20 | |             })?),
   | |________________^ method was supposed to return data with lifetime `'2` but it is returning data with lifetime `'1`

in addition to a deprecation warning.

Personally, I think the self lifetime being preferred is relatively common paper cut when returning something with a non-static lifetime (but generally useful enough to carry its weight in pure Rust code), e.g. the variant

#[pyfunction]
fn to_vec(instance: &Instance, py: Python) -> PyResult<&PyBytes> {
    match instance.to_buf() {
        Ok(v) => Ok(PyBytes::new_with(py, v.len(), |b| {
            b.copy_from_slice(&v);
            Ok(())
        })?),
        Err(e) => Err(PyException::new_err(e.to_string())),
    }
}

does make the issue much easier to understand

error[E0106]: missing lifetime specifier
  --> src/main.rs:14:56
   |
14 | fn to_vec(instance: &Instance, py: Python) -> PyResult<&PyBytes> {
   |                     ---------      ------              ^ expected named lifetime parameter
   |
   = help: this function's return type contains a borrowed value, but the signature does not say whether it is borrowed from `instance` or `py`
help: consider introducing a named lifetime parameter
   |
14 | fn to_vec<'a>(instance: &'a Instance, py: Python<'a>) -> PyResult<&'a PyBytes> {
   |          ++++            ++                     ++++               ++

Hence, I think a section in the guide might be helpful even if the issue is not new as people often appear to discover Rust via PyO3 e.g. when optimizing a larger Python project.