Make rez wheels production ready #1536
Conversation
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
2 times, most recently
from
ba20e6e
to
984b310
Compare
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
from
97dd248
to
94924b0
Compare
|
@JeanChristopheMorinPerso , might this sort of workflow open the door for us to be able to have rez and rez plugins exist slightly closer together in a standard way? (Thought provoked by TSC meeting) |
|
@maxnbk I'm not sure how to answer this. Can you clarify what you mean by "open the door for us to be able to have rez and rez plugins exist slightly closer together in a standard way" please? |
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
3 times, most recently
from
b604807
to
1f73a94
Compare
|
I think I finally got it working with I also adapted the PR so that it works on Linux and macOS too. For that, I had to modify the scripts we generate on unix to This is needed because pip doesn't keep the arguments in the shebang. On Windows, I simply inject Next step, clean up, write a proper workflow to generate the wheels on all platforms, figure out where to store the launcher and write at least one integration test that would do this:
I think that this is enough to prove that our install is production ready and works as intended. TO be even more sure, we could try to install with Eventually, we could review the flags used. -E gives us some guarantees, but we can provide even more guarantees by using other flags, like -I (which is is a combination of |
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
5 times, most recently
from
50be3f0
to
b4b685e
Compare
|
Umm, it seems like it doesn't work when using |
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
13 times, most recently
from
2030a61
to
8959315
Compare
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
JeanChristopheMorinPerso
force-pushed
the
wheel_prod_install
branch
from
553eb74
to
1e26691
Compare
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
JeanChristopheMorinPerso
changed the title
|
|
||
|
|
||
| @register("_rez-install-test") | ||
| def run_rez_install_test(): |
There was a problem hiding this comment.
This command is part of the integration tests. It allows us to introspect how rez was launched. It'll print a json with the path to the python executable, the arguments from argv and the interpreter flags (sys.flags).
install.py
Outdated
| @@ -162,17 +162,14 @@ def install(dest_dir, print_welcome=False, editable=False): | |||
| install_rez_from_source(dest_dir, editable=editable) | |||
|
|
|||
| # patch the rez binaries | |||
| patch_rez_binaries(dest_dir) | |||
| # patch_rez_binaries(dest_dir) | |||
There was a problem hiding this comment.
Should we feature flag this?
…e-executing ourselves on Unix. Signed-off-by: Jean-Christophe Morin <[email protected]>
…arning Signed-off-by: Jean-Christophe Morin <[email protected]>
Signed-off-by: Jean-Christophe Morin <[email protected]>
| # Call python with -E directly to avoid the double cost of executing | ||
| # rezolve when the CLI is installed via pip. | ||
| rezolve = os.path.join(system.rez_bin_path, 'rezolve') | ||
| ex.command(f'{sys.executable} -E {rezolve} context') |
There was a problem hiding this comment.
I'm kind of making the assumption that this is only run when using rez-env... Can this code ever run when solely using the API?
Make rez fully installable with pip, even for production uses. This takes some of the ideas in #1039 by @davidlatwe and improves them to make the whole concept work.
Included in the PR:
.rez_production_install.wheel.yamlworkflow).TODO:
What is a production install today?
Our
install.pyscript does mainly three things, out of which two are to guarantee that rez will work in a production environment:pip install .).$PATHbut we don't want thepythonexecutable from the virtualenv to be on PATH. We also can't remove thepythonexecutable from the bin folder because that would make it impossible to install rez plugins.#!/path/to/rez/venv/bin/pythonto#!/path/to/rez/venv/bin/python -E(we add-E) and we generate custom.exes on Windows to do the same thing.-Eis important for us because we don't want PYTHONPATH to affect the rez commands.The goal of all this is to insulate the rez install from everything. Nothing should affect it. It must work under all conditions. Additionally, rez must not be affected by a rez environment. For example, if your rez install uses Python 3.7 and you
rez-env python-3.12, running rez commands inside that new environment must use the Python 3.7 interpreter from the rez install! Rez must also ignore any environment variables that could affect it (PYTHONPATH, etc).The install script is needed today because the Python ecosystem was not designed with these goals in mind. When you install a package with pip (or any other installer) and that package has commands (console entry points), installers will create small script wrappers that look like this:
As you can see, it doesn't have the
-Eflag in the shebang. And most importantly, there is no way to tellinstallers to use a custom shebang or where the scripts will be installed. They always go in
bin(macOS, Linux) orScripts(Windows).The
install.pyscript exists to fix all this. It was created because it was the easiest thing to do.Why is it a problem to have to run the install.py script?
Python users are used to simply install things with their preferred installer (pip, etc). So it is natural that they would also want to decide how they want to install rez.
But with what we've seen above, they can't. Or more, they can but they'll see
warnings every time they use the CLI.
We get asked quite often why can't they simply pip install rez.
It's very annoying to have to clone a repo and have to run a script. The alternative is to download an archive from a release and unpack it and run the install script. Both options are annoying IMO, and I'm not the only one annoyed by this based on all the discussions we've had on that subject in the last couple of years.
How do we fix this?
This PR utilizes the .data directory of the wheel format. The
.datacan contain a couple of folders, one of which is thescriptsfolder. Thescriptsfolder is understood and supported by installers. Anything inscriptswill go in<prefix>/bin(macOS, Linux) or<prefix>/Scripts(Windows) as is (or almost as is).So what we do is generate the console scripts at build time and store them inside
rez-<version>.data/scripts/rezin the wheel. When pip installs the wheel, it will unpack everything from<dist>-<version>.data/scriptsinto thebinfolder and it will respect the hierarchy defined in the wheel! Pip will also replace the interpreter path found in the script's shebang with the venv's one. Again this is important because we want rez to use its own Python. So that's one problem solved (point 2 from above).Example of what is inside the wheel
Now, how do we solve the fact that we can't easily add
-Eto the shebang? This is solved with this little trick in our custom console scripts:This is the rez-build script that we store in our wheel. Note that pip will automatically replace
/usr/bin/pythonby<absolute path to venv>/bin/pythonat install time.Because we want to let pip bake the absolute path to the interpreter at install time, we can't pass any custom flags in there. If we were to specify
#!/usr/bin/python -E, pip would not replace the path with the path we want.So what we do is to check how the script was executed. It's possible to know if Python was started with
-Eor not by querying sys.flags.ignore_environment. If it was not started with-E, we re-exec ourselves with-Eprepended to the arguments.At least that's on Linux. On Windows, things are more complicated (as usual).
On Windows, we need executables to be able to just run "rez" or "rez-env", etc. Why? Because on Windows, you can't execute a random file without an extension. So you have to wrap the little Python script above with an executable. Pip usually takes care of that with entry points, but it doesn't do so for scripts, by design.
These executables are called "launchers" in the Python ecosystems. CPython has a built-in one, then there is the original simple launcher by Vinay Sajip and there is also a derivative of the simple launcher in distlib also by Vinay Sajip. There is a fourth one in setuptools, but let's ignore this one. Pip uses the one from distlib.
What I did was to take the one from
distliband put it into thelaunchersfolder. I then patched it to fit our needs. The patch is small:Diff
We basically
APPENDED_ARCHIVEpreprocessor directive so that the console script is not embedded in the EXE file.USE_ENVIRONMENTpreprocessor directive to not use any environment variables to resolve the path to python.SUPPORT_RELATIVE_PATHpreprocessor directive to make sure that the embedded python path doesn't use a relative path.REZ_LAUNCHER_DEBUGthat will tell the launcher to print the full command line that will be used to start python (and run the console script).-Eto the arguments used when launching python.We put the launchers in the wheel too. And this solves the last remaining problem we had.
Example of what is inside the wheel
The cool part is that the launchers are only compiled when building a wheel on Windows, and Visual Studio is not even needed! We use the awesome C/C++ compiler provided by the Zig language. Since it's available as an official package on PyPI (
ziglang), everything just works.That's it?
Yes, that's pretty much it. Or almost. We have to talk about wheels a little bit more. Because who doesn't like to talk about wheels?
There is a last detail to talk about and it's the final wheels. We will generate three of them:
rez-<version>-py3-none-any.whl.rez-<version>-py3-none-win_amd64.whl.rez-<version>-py3-none-win_arm64.whl.By generating a generic wheel for non-Windows platforms, pip will be able to install rez on everything that is not Windows (macOS, Linux, etc).
And because installers have to prefer more specific wheels before generic ones, they will install the Windows variants if installed on Windows.