Introduce the environment builder API and implement a micromamba-based build handler #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There are three kinds of events possible: * Progress update on some action - e.g. a file download * Output message - e.g. the stdout stream of a micromamba invocation * Error message - e.g. the stderr stream of a micromamba invocation
So that every single build doesn't need to download and unpack micromamba, then download and unpack all conda environment packages.
Yeah, I know it's failing because it's using the system python on the system path. I guess ProcessBuilder does that, similar to a shell. But I wanted to add the debug output just to be friendly and clear. The question is: how do we configure ProcessBuilder not to do that?
This sidesteps the thorny issue of environment activation within the same process, in favor of micromamba activating+running the desired worker launch command within the correct environment prefix, such that activation scripts get sourced properly beforehand.
Just a succinct shorthand for subscribing to progress, stdout, and stderr, printing all the events coming in.
Only inject micromamba config (`mamba run -f ...`) launch args if the Appose environment is a conda one. And clean up the no-pythons-to-be-found-here directory on exit. This fixes the ApposeTest#testServiceStartupFailure.
Yellow for stdout, red for stderr.
|
I'm looking for feedback on the builder's API design. If no one has time to take a look and comment within the next couple of days, I'll probably just go ahead and merge. But if you have time to take a quick peek into |
ctrueden
force-pushed
the
env-builder
branch
from
7fa7c7b to
5fff306
Compare
On macOS, the size of the allocated shared memory block might not precisely match the request. But it needs to be at least as large as the requested size. E.g.: sizes less than 16384 round up to 16384. You might think it would be a good idea to make the size() method of SharedMemory return the actually requested size, rather than the weird rounded up size, but that's how it works on macOS on Python, and we're trying to make the Java implementation match the Python behavior as closely as possible, so... ¯\_(ツ)_/¯
ctrueden
force-pushed
the
env-builder
branch
from
a34b2ce to
4ea317b
Compare
|
Tests all passing on all platforms. I would still value API design feedback from @carlosuc3m @tpietzsch @hinerm and anyone with time and interest, though! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch adds the logic needed to build conda environments from an
environment.ymlfile, by downloading and calling micromamba in a subprocess. It does not yet support composing environments from multiple YAML files, nor declaring environments package by package. But it does provide a general and extensible API for more environment build handlers in the future, such as a builder that fetches and installs Maven artifacts, or one that installs packages from node.js.