Skip to content

Commit e3a9906

Browse files
RobertRoscaRobertRosca
authored
Initial Development MR (#1)
1 parent 317fe93 commit e3a9906

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+3298
-15
lines changed

‎.github/Dockerfile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM ecpe4s/ubuntu20.04:22.08
2+
3+
RUN . /spack/share/spack/setup-env.sh && spack install python && spack install py-pip
4+
5+
RUN apt install python3.8-venv
6+
7+
RUN python3 -m pip install --upgrade pip
8+
RUN python3 -m pip install pipx
9+
10+
RUN pipx install pipenv
11+
RUN pipx install poetry
12+
13+
ENV PATH="${PATH}:/root/.local/bin"
14+
15+
LABEL org.opencontainers.image.source https://github.com/European-XFEL/pyvarium

‎.github/build-test-image.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: Build Test Image
2+
3+
on:
4+
workflow_dispatch:
5+
push:
6+
paths:
7+
- ".github/Dockerfile"
8+
9+
env:
10+
REGISTRY: ghcr.io
11+
IMAGE_NAME: ${{ github.repository }}-test-image
12+
13+
jobs:
14+
build-and-push-image:
15+
runs-on: ubuntu-latest
16+
permissions:
17+
contents: read
18+
packages: write
19+
20+
steps:
21+
- name: Checkout repository
22+
uses: actions/checkout@v3
23+
24+
- name: Log in to the Container registry
25+
uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
26+
with:
27+
registry: ${{ env.REGISTRY }}
28+
username: ${{ github.actor }}
29+
password: ${{ secrets.GITHUB_TOKEN }}
30+
31+
- name: Extract metadata (tags, labels) for Docker
32+
id: meta
33+
uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
34+
with:
35+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
36+
37+
- name: Build and push Docker image
38+
uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
39+
with:
40+
context: .github
41+
push: true
42+
tags: ${{ steps.meta.outputs.tags }}
43+
labels: ${{ steps.meta.outputs.labels }}

‎.github/workflows/docs.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: ci
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
jobs:
9+
deploy:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v2
13+
- uses: actions/setup-python@v2
14+
with:
15+
python-version: 3.x
16+
- name: Cache
17+
uses: actions/cache@v1
18+
with:
19+
path: ~/.cache/pip
20+
key: pyproject-${{ hashFiles('**/pyproject.toml') }}
21+
- name: Setup
22+
run: |
23+
python3 -m pip install --upgrade pip poetry
24+
poetry install --only docs
25+
- run: poetry run mkdocs gh-deploy --force

‎.github/workflows/publish.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Publish
2+
3+
on:
4+
release:
5+
types:
6+
- created
7+
8+
jobs:
9+
publish:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v3
13+
- name: Set up Python
14+
uses: actions/setup-python@v4
15+
with:
16+
python-version: "3.8"
17+
- uses: actions/cache@v3
18+
id: cache
19+
with:
20+
path: ${{ env.pythonLocation }}
21+
key: ${{ runner.os }}-python-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}-publish
22+
- name: Install build dependencies
23+
if: steps.cache.outputs.cache-hit != 'true'
24+
run: pip install build
25+
- name: Build distribution
26+
run: python -m build
27+
- name: Publish
28+
uses: pypa/[email protected]
29+
with:
30+
password: ${{ secrets.PYPI_API_TOKEN }}

‎.github/workflows/tests.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
tests:
11+
runs-on: ubuntu-20.04
12+
container:
13+
image: ghcr.io/european-xfel/pyvarium:test
14+
credentials:
15+
username: ${{ github.actor }}
16+
password: ${{ secrets.GITHUB_TOKEN }}
17+
defaults:
18+
run:
19+
shell: bash
20+
steps:
21+
- uses: actions/checkout@v3
22+
- name: Cache
23+
uses: actions/cache@v1
24+
with:
25+
path: ~/.cache/pip
26+
key: pyproject-${{ hashFiles('**/pyproject.toml') }}
27+
- name: Setup
28+
run: |
29+
poetry install
30+
- name: Test
31+
run: |
32+
source /spack/share/spack/setup-env.sh
33+
poetry run pytest
34+
- name: Upload to Codecov
35+
uses: codecov/codecov-action@v2

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.coverage
2+
__pycache__
3+
coverage.xml

‎README.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# Pyvarium
2+
3+
Pyvarium is a tool designed to help create environments which are managed by [Spack](github.com/spack/spack/) and a pure python environment manager like [Pipenv](https://pipenv.pypa.io/en/latest/) or [Poetry](https://github.com/python-poetry/poetry).
4+
5+
It aims to help tackle the problem of long term reproducibility and portability of software environments, especially in the area of scientific HPC, by combining the ability of Spack to compile arbitrary software (as long as a package Spack package has been written for it) and flexibility of python environments.
6+
7+
## Installation
8+
9+
Simplest installation is with [pipx](https://github.com/pypa/pipx):
10+
11+
```shell
12+
pipx install pyvarium
13+
```
14+
15+
Note that Pyvarium is designed to **manage** Spack and python environments, so an existing spack installation as well as a supported backend (e.g. pipenv) is required for Pyvarium to run.
16+
17+
## Quick Start
18+
19+
```shell
20+
$ pyvarium --help
21+
Usage: pyvarium [OPTIONS] COMMAND [ARGS]...
22+
23+
Deploy mixed computational environments with dependencies and packages provided by Spack and Pipenv
24+
25+
Options:
26+
--log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL]
27+
Pick which level of output to show
28+
[default: INFO]
29+
30+
-v, --verbose [default: 0]
31+
--install-completion Install completion for the current shell.
32+
--show-completion Show completion for the current shell, to
33+
copy it or customize the installation.
34+
35+
--help Show this message and exit.
36+
37+
Commands:
38+
add Add packages via spack or pipenv.
39+
config Modify user settings for pyvarium.
40+
install Concretize and install an existing environment.
41+
modulegen Generate modulefile to load the environment.
42+
new Create a new combined Spack and Pipenv environment.
43+
sync Sync Spack-managed packages with Pipenv.
44+
```
45+
46+
To create a new environment:
47+
48+
```
49+
$ pyvarium new ./environment-demo
50+
```
51+
52+
This creates a new directory `./environment-demo` which has Pipenv (`Pipfile`, `Pipfile.lock`) and Spack (`spack.yaml`, `spack.lock`) environment specification files in it. By default these will only specify `python` and `pip` as requirements.
53+
54+
In general setting up the environment should be done in a specific order, you must install any python packages which have **external dependencies** via Spack **first**, then install any required pure python packages via pipenv afterwards. Otherwise python packages with external dependencies will use their bundled binaries instead of whatever is provided via spack.
55+
56+
57+
Installation of dependencies can be done with `pyvarium add {spack,pipenv}`, this command is a wrapper around the respective add/install commands which ensures that the dependencies specified by both programs remain in sync:
58+
59+
```shell
60+
$ pyvarium add spack py-numpy py-h5py
61+
$ pyvarium add pipenv extra-data
62+
```
63+
64+
The above commands would install numpy and h5py via spack, compiling dependencies along the way, sync these packages with `Pipfile` so that pipenv is aware of what is already within the environment, and then install any specified packages via pipenv.
65+
66+
The environment can be activated as a normal venv with `source .venv/bin/activate`, or a module file can be created for it with with `pyvarium modulegen`.
67+
68+
## Usage
69+
70+
```shell
71+
Usage: pyvarium [OPTIONS] COMMAND [ARGS]...
72+
73+
Deploy mixed computational environments with dependencies and packages provided by Spack and Pipenv
74+
75+
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────╮
76+
│ --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|C Pick which level of output to show │
77+
│ RITICAL] [default: LogLevel.info] │
78+
│ --install-completion Install completion for the current │
79+
│ shell. │
80+
│ --show-completion Show completion for the current │
81+
│ shell, to copy it or customize the │
82+
│ installation. │
83+
│ --help Show this message and exit. │
84+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
85+
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────╮
86+
│ add Add packages via spack or pipenv. │
87+
│ config Modify user settings for pyvarium. │
88+
│ install Concretize and install an existing environment. │
89+
│ modulegen Generate modulefile to load the environment. │
90+
│ new Create a new combined Spack and Pipenv environment. │
91+
│ sync Sync Spack-managed packages with Pipenv. │
92+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
93+
```
94+
95+
### `config`
96+
97+
```shell
98+
Usage: pyvarium config [OPTIONS] COMMAND [ARGS]...
99+
100+
Modify user settings for pyvarium.
101+
102+
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────╮
103+
│ --help Show this message and exit. │
104+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
105+
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────╮
106+
│ info Show information about the current configuration. │
107+
│ list List configuration. │
108+
│ set Set a key value pair for configuration, e.g. `pyvarium set poetry_exec │
109+
│ /opt/poetry/bin/poetry`. │
110+
│ unset Remove a custom settings from configuration (this reverts to the default, to disable a │
111+
│ default set it to an empty string, e.g. `pyvarium set poetry_exec ""`). │
112+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
113+
```
114+
115+
`info` will show the absolute location of the configuration files being used, in the order they are loaded.
116+
117+
`list` shows the current configuration.
118+
119+
`set` can be used to set a key-value pair, and `unset` can be used to remove one. Setting and unsetting configurations can be performed at different scopes: `local` for the local directory (e.g. creating a `pyvarium.toml` file), `user` for the user configuration (`~/.config/pyvarium/settings.toml`), and `builtin` to modify the default configuration within the package.
120+
121+
The configuration is a TOML file containing the paths to the executables to be used by pyvarium, for example:
122+
123+
```toml
124+
pipenv = "/home/roscar/.local/bin/pipenv"
125+
pipx = "/home/roscar/.local/bin/pipx"
126+
poetry = "/sbin/poetry"
127+
spack = "/opt/spack"
128+
```
129+
130+
By default, if an entry is not present in the file for one of these programs, it is automatically set to the path of `which $program`.
131+
132+
### `new`
133+
134+
```shell
135+
Usage: pyvarium new [OPTIONS] PATH COMMAND [ARGS]...
136+
137+
Create a new combined Spack and Pipenv environment.
138+
139+
╭─ Arguments ────────────────────────────────────────────────────────────────────────────────────────╮
140+
│ * path DIRECTORY [default: None] [required] │
141+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
142+
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────╮
143+
│ --help Show this message and exit. │
144+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
145+
```
146+
147+
### `add`
148+
149+
```shell
150+
Usage: pyvarium add [OPTIONS] COMMAND [ARGS]...
151+
152+
Add packages via spack or pipenv.
153+
154+
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────╮
155+
│ --help Show this message and exit. │
156+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
157+
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────╮
158+
│ pipenv │
159+
│ spack │
160+
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
161+
```

‎docs/css/material.css‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/* More space at the bottom of the page. */
2+
.md-main__inner {
3+
margin-bottom: 1.5rem;
4+
}

‎docs/css/mkdocstring.css‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/* Indentation. */
2+
div.doc-contents:not(.first) {
3+
padding-left: 25px;
4+
border-left: .05rem solid var(--md-typeset-table-color);
5+
}
6+
7+
/* Mark external links as such */
8+
a.autorefs-external::after {
9+
/* https://primer.style/octicons/arrow-up-right-24 */
10+
background-image: url('data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="rgb(0, 0, 0)" d="M18.25 15.5a.75.75 0 00.75-.75v-9a.75.75 0 00-.75-.75h-9a.75.75 0 000 1.5h7.19L6.22 16.72a.75.75 0 101.06 1.06L17.5 7.56v7.19c0 .414.336.75.75.75z"></path></svg>');
11+
content: ' ';
12+
13+
display: inline-block;
14+
position: relative;
15+
top: 0.1em;
16+
margin-left: 0.2em;
17+
margin-right: 0.1em;
18+
19+
height: 1em;
20+
width: 1em;
21+
border-radius: 100%;
22+
background-color: var(--md-typeset-a-color);
23+
}
24+
a.autorefs-external:hover::after {
25+
background-color: var(--md-accent-fg-color);
26+
}
27+
28+
.highlight .gp, .highlight .go { /* Generic.Prompt, Generic.Output */
29+
user-select: none;
30+
}

‎docs/gen_ref_pages.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
"""Generate the code reference pages and navigation."""
2+
from pathlib import Path
3+
4+
import mkdocs_gen_files
5+
6+
for path in sorted(Path("src").rglob("*.py")):
7+
if ".local" in str(path):
8+
continue
9+
10+
module_path = path.relative_to("src").with_suffix("")
11+
doc_path = path.relative_to("src").with_suffix(".md")
12+
full_doc_path = Path("code-reference", doc_path)
13+
14+
parts = tuple(module_path.parts)
15+
16+
if parts[-1] == "__init__":
17+
parts = parts[:-1]
18+
doc_path = doc_path.with_name("index.md")
19+
full_doc_path = full_doc_path.with_name("index.md")
20+
elif parts[-1] == "__main__":
21+
continue
22+
23+
full_doc_path = Path("code-reference") / full_doc_path.relative_to(
24+
"code-reference"
25+
)
26+
27+
with mkdocs_gen_files.open(full_doc_path, "w") as fd:
28+
ident = ".".join(parts)
29+
fd.write(f"::: {ident}")
30+
31+
mkdocs_gen_files.set_edit_path(full_doc_path, path)

0 commit comments

Comments
 (0)