Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better reproducibility wrapping Python ArviZ #187

Closed
sethaxen opened this issue May 27, 2022 · 4 comments
Closed

Better reproducibility wrapping Python ArviZ #187

sethaxen opened this issue May 27, 2022 · 4 comments
Labels
discussion enhancement New feature or request

Comments

@sethaxen
Copy link
Member

One of the ways the Julia ecosystem achieves reproducibility even with binary dependencies is by defining glue packages that wrap the binaries and are tagged for specific versions. e.g. https://github.com/JuliaBinaryWrappers/wget_jll.jl. This way Julia packages can specify exact supported versions for these dependencies.

I'm considering doing something like that here until #130 is completely resolved. Specifically, we would create an ArviZ_py.jl (or PyArviZ) package that only depends on PyCall. v0.12.0 of the package would install v0.12.0 of the ArviZ Python package upon loading, and upon start-up it would check that the exact Python ArviZ version is installed and otherwise prompt the user to update (what we currently do in this package). When a new Python ArviZ release is made, a new release is tagged there, and then we only update our compat here when we fully support that version (and drop compat for versions we don't support). As an added bonus, we maybe could ensure that on the main branch, the main branch of Python ArviZ is always installed, so testing Julia ArviZ against future releases of Python ArviZ would be as easy as

] add ArviZ, ArviZ_py#main

Since Python ArviZ releases are relatively rare, this would not be much of an additional maintenance burden, and it would let us be at least more reproducible than we currently are.

@sethaxen sethaxen added enhancement New feature or request discussion labels May 27, 2022
@sethaxen
Copy link
Member Author

Apparently PythonCall allows for providing a TOML file for dependencies to ensure reproducibility, so that's an alternative. I'm not certain how well PyPlot will work without PyPlot.jl, which uses PyCall not PythonCall.

@devmotion
Copy link

devmotion commented Jun 30, 2022

PythonPlot is the PythonCall version of PyPlot: https://github.com/stevengj/PythonPlot.jl I've never used it, so I can't comment on the differences.

@sethaxen
Copy link
Member Author

sethaxen commented Oct 3, 2022

I spent some time replacing PyCall with PythonCall, but it turns out to require quite a lot of surgery. PythonCall by default does no conversions between Python and Julia, and we extensively rely on PyCall's automated conversions, so it would require quite some code for ArviZ to "just work" with PythonCall the way it does with PyCall. Plus, every time one installs ArviZ, a new Conda environment is created. Already-downloaded conda packages are not reused, so package installation becomes much slower and internet-bound.

I've opted for #234 to hopefully resolve our major dependency issues, and the time that would go toward using PythonCall will instead go toward removing Python as a dependency.

@sethaxen
Copy link
Member Author

With #290, we no longer use Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants