Cooperation on Pure-Python Implementation?

[RobertRosca]

Hey Konstin, you recently messaged me on Mastodon about my proof of concept which is conceptually extremely similar to your work here, just far less mature.

I've had a look through this repository and the goals and approaches of Monotrail are pretty much exactly what I was thinking of and started doing, which is awesome! But we do have different goals, my rough plan is to:

1. Develop a standalone proof of concept
2. Pick some popular python packages with good test suites, and use them for automated testing
3. Integrate it as a plugin for the main venv-based project management tools (namely Poetry, PDM, Pipenv, and Hatch)
4. If it is supported and liked by the community, create a PEP for this and attempt to integrate it into core python

Integration within the python ecosystem would be much easier with a pure python package, so I think I'll carry on with my proof of concept , borrowing some ideas from Monotrail (with credit of course 😉) along the way, if that's alright? I'll also occasionally post updates on both the thread on the python forums and on Mastodon.

Right now this approach is unknown enough in python that people don't seem too interested, but I will keep all of my notes and progress updates within the repository and project page (https://github.com/users/RobertRosca/projects/2) in case this does end up becoming more popular, so you're also welcome to chip in on the discussions or contribute if you're interested.

(this isn't an issue but I don't think the repo has discussions enabled)

[konstin]

Hi!

I think the easiest to collaborate on would be the directory structure. I'm currently using the following (on linux/mac, windows is slightly different):

{user cache dir}/{monotrail}/installed/{distribution}/{version}/{wheel tags}/lib/python/site-packages/{package}/{path/to/file}

(note that it's /python/ not /pythonx.y/, so we don't have the python version in there). Example:

~/.cache/monotrail/installed/numpy/1.22.3/cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python/site-packages/numpy/random/__init__.py

I expect that doing this properly will require some patches to pypa/installer. I'm also doing some hackery in the PathFinder to make it work with packages that ship multiple modules or modules under the wrong name (such as pillow), namespaces (such as poetry-core with poetry), path files (such as matplotlib), importlib.metadata (find_distributions) and pkg_resources (patching working_set).

I also recommend PyOxy, because since you're not bound to a venv anymore, you can combine it with an arbitrary version python launcher (and essentially get tox' main feature for free).

I'd love to co-author a PEP on this and get this implemented in the wider ecosystem! I think poetry and hatch plugins to support this would be also really cool to have.

Integration within the python ecosystem would be much easier with a pure python package, so I think I'll carry on with my proof of concept , borrowing some ideas from Monotrail (with credit of course wink) along the way, if that's alright? I'll also occasionally post updates on both the thread on the python forums and on Mastodon.

yep, a pure python implementation will certainly be much easier to get officially adopted! I'm planning to build this in rust though. I'm slowly reimplementing everything to do dependency resolution in rust. You might want to try install-wheel-rs as an optional, faster and more parallel rust component though (ideally, this should be maybe 10 lines on top of pypa/installer usage, depending on how much parallelism you want to have).

(this isn't an issue but I don't think the repo has discussions enabled)

This is perfect for issues :)

[RobertRosca]

Thanks for the reply! Sorry for the late reply from me, I was quite sick for the past week or so.

I think the easiest to collaborate on would be the directory structure.

The structure you're using already looks very good. Keeping the full scheme for each package internally is nice as it provides a way to handle entry points as well.

Few questions/ideas:

1. The path has {wheel tags}, if installation is done from source distribution would that just be set to source?
2. Did you consider dropping the human-readable paths and just using hashes instead? If you require lock files to be used then the hash would always be specified, and the path can be simplified to {package_name}/{short_hash}.

I expect that doing this properly will require some patches to pypa/installer. I'm also doing some hackery in the PathFinder to make it work with packages that ship multiple modules or modules under the wrong name (such as pillow), namespaces (such as poetry-core with poetry), path files (such as matplotlib), importlib.metadata (find_distributions) and pkg_resources (patching working_set).

Ah yes, that's a problem I completely ignored in my basic implementation. Another approach, which may be simpler, could be to use a modified version of the existing RECORD mechanisms to track exactly what modules provided by a specific python package (name).

I also recommend PyOxy, because since you're not bound to a venv anymore, you can combine it with an arbitrary version python launcher (and essentially get tox' main feature for free).

Sounds interesting, I'll have a look at this 😄

I'd love to co-author a PEP on this and get this implemented in the wider ecosystem! I think poetry and hatch plugins to support this would be also really cool to have.

Yeah, I feel that with this idea it would make sense to create a barebones package which can be re-used by plugins in poetry/hatch/pdm, and then if it is popular work on a PEP to standardise the approach more and open it up to other package managers.

The proposal for Julia's Pkg3 outlines how the 'depots' (equivalent to site-packages) work, as well as environments and package metadata, it might be worth looking through for some inspiration: https://github.com/JuliaLang/Juleps/blob/master/Pkg3.md