initial module import in REPL is raising ImportError for multiple initialization

[davidhewitt]

After maturin develop and attempting import supermodule.submodule in a REPL I'm now getting:

thread '<unnamed>' panicked at 'failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }', supermodule/src/lib.rs:432:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python3.9/site-packages/supermodule/__init__.py", line 1, in <module>
    from .supermodule import *
pyo3_runtime.PanicException: failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }

I don't see any issues in this repo on this particular error (multiple initialization) - any idea where I've gone wrong?

(I see from https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/guide/src/migration.md#each-pymodule-can-now-only-be-initialized-once-per-process that this is new behavior as of the 28 August release of 0.17 but it's not clear from that writeup what 'the ability to initialize a #[pymodule] more than once in the same process' means - i.e. I am just starting a REPL and importing the package one time to trigger the error).

Originally posted by @jdiggans-twist in #759 (comment)

[davidhewitt]

@jdiggans-twist our maturin_starter example does not exhibit this behaviour and explicitly tests the submodule case. Can you please share a minimal repro of how you're hitting the error?

[davidhewitt]

For clarity, initialization means "create the module object". I deliberately chose not to use the word "import" in the error message because caching in sys.modules means that if you run import foo twice in a row you only create the foo module object once, and then the second time the import statement just returns the previously cached foo.

Suggestions for alternative wording always welcome.

[jdiggans-twist]

Thanks @davidhewitt - your explanation makes sense. Re: the code I used in terms of MWE, this was taken directly from #759 (comment):

/// Export constants into Python
#[pymodule]
fn constants(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add("VERSION", VERSION)?;
}

/// Python module initialization
#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let submodule = wrap_pymodule!(constants);
    m.add_wrapped(submodule)?;
    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.constants", submodule(py))?;

    Ok(())
}

Moving to the version in maturin_starter fixed the issue:

    m.add_wrapped(wrap_pymodule!(constants))?;
    let sys = PyModule::import(py, "sys")?;
    let sys_modules: &PyDict = sys.getattr("modules")?.downcast()?;
    sys_modules.set_item("supermodule.constants", m.getattr("constants")?)?;

and let me re-upgrade to PyO3 v0.17.1 - thank you for the help and pointer!

[davidhewitt]

Aha great - yes m.add_wrapped(submodule)? and submodule(py) would both be creating new copies of the module, so in updating this has also helped you avoid weird effects of having two copies of the module object 😄