FEAT: nix & apptainer stable on multiple machines (#50)

* FIX: Nix * FEAT: Update to pyproject.toml * FIX: in progress * FIX: nix installation script * FIX: Attempt to remove the venv if it can use local LD_LIBRARY_PATH * FEAT: Try again * FIX: patch setup.py * FIX: Nearly fixed envrionemnt * FEAT: Add virtualisation packages * FEAT: Develop apptainer environment * FEAT: Working apptainer isntall * FEAT: Improving mamba install * FEAT: Improving mamba install * FEAT: Improving mamba install errors * FIX: piel status * FEAT: Working micromamba installation * FEAT: Working apptainer built environment * FEAT: so far * FIX: Still getting the development setup to build * FEAT: Further status * FEAT: Start apptainer script * FEAT: Working kind of * FEAT: Working kind of * FEAT: Kind of stable * FEAT: Stable and working apptainer --------- Co-authored-by: Jadcocker <[email protected]> Co-authored-by: Dario Quintero Dominguez <[email protected]>
daquintero · Oct 3, 2023 · 945c0ce · 945c0ce
1 parent 85bf038
commit 945c0ce
Show file tree

Hide file tree

Showing 13 changed files with 177 additions and 14 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -3,7 +3,7 @@ current_version = 0.0.54
 commit = True
 tag = True
 
-[bumpversion:file:setup.py]
+[bumpversion:file:pyproject.toml]
 search = "{current_version}"
 replace = "{new_version}"
 

diff --git a/.gitignore b/.gitignore
@@ -196,3 +196,5 @@ waves.shm/
 ivl_vhdl_work/
 RUN_*
 *.raw
+
+*.sif
diff --git a/docs/sections/environment/apptainer/relevant_apptainer_commands.rst b/docs/sections/environment/apptainer/relevant_apptainer_commands.rst
@@ -0,0 +1,16 @@
+``apptainer`` Useful Commands
+------------------------------------------
+
+.. list-table:: Useful ``apptainer`` commands
+   :header-rows: 1
+
+   * - Description
+     - Command
+   * - Build production shell with all available CPU cores
+     - ``apptainer instance start piel.sif pielapptainer``
+   * - Build production shell with all available CPU cores
+     - ``apptainer shell instance://pielapptainer``
+   * - Build production shell with all available CPU cores
+     - ``apptainer build piel.sif piel.def``
+   * - Build production shell with all available CPU cores
+     - ``apptainer instance stop pielapptainer``
diff --git a/docs/sections/environment/index.rst b/docs/sections/environment/index.rst
@@ -22,6 +22,7 @@ Environment
 ``apptainer`` is a good open-source container management system that aims to be optimised for high performance computation. We want to have a distributed container environment where all the open-source ``piel`` toolchain is pre-installed and ready for custom design. What this distribution aims to provide is both an easy installation script for Ubuntu environments which are common in open-source development and a specific environment configuration that resolves the particular supported versions of the toolchains.
 
 .. include:: apptainer/apptainer_install.rst
+.. include:: apptainer/relevant_apptainer_commands.rst
 
 
 ``docker`` Configuration (In Passive Development)
@@ -33,7 +34,16 @@ Environment
 .. include:: docker/developer_docker_configuration.rst
 
 
+``mamba`` Recipie Configuration (In Active Development)
+=================================================
+
+.. include:: mamba/relevant_mamba_commands.rst
+
+
 .. include:: installation.rst
 .. include:: project_structure.rst
 .. include:: design_files_interaction.rst
 .. include:: relevant_python_commands.rst
+
+
+
diff --git a/environment/apptainer/build_apptainer.sh b/environment/apptainer/build_apptainer.sh
@@ -0,0 +1,2 @@
+cd ~/piel/environment/apptainer
+apptainer build --sandbox piel.sif piel.def
diff --git a/environment/apptainer/piel.def b/environment/apptainer/piel.def
@@ -0,0 +1,38 @@
+Bootstrap: docker
+From: efabless/openlane
+
+
+%post
+    yum -y update
+    yum install -y bzip2 wget
+
+    # Install micromamba
+    apt-get update && apt-get install -y wget
+    wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
+    mkdir -p /opt/micromamba
+    mv bin/micromamba /opt/micromamba/micromamba
+    chmod +x /opt/micromamba/micromamba
+    export PATH=/opt/micromamba:$PATH
+
+
+    # Create pielenv environment in micromamba
+    micromamba shell init -s bash 
+    micromamba create -p /opt/envs/pielenv -c conda-forge python=3.10.9
+    micromamba run -p /opt/envs/pielenv micromamba install gdstk -c conda-forge   
+
+%environment
+    export PATH=/opt/micromamba:$PATH
+    export PIEL_DIRECTORY="/opt/piel/"
+
+%runscript
+    micromamba shell init -s bash
+    micromamba run -p /opt/envs/pielenv pip install -r $PIEL_DIRECTORY/requirements_dev.txt --break-system-packages
+    micromamba run -p /opt/envs/pielenv pip install -e $PIEL_DIRECTORY --break-system-packages
+
+
+
+%startscript
+    micromamba shell init -s bash
+    micromamba run -p /opt/envs/pielenv pip install -r $PIEL_DIRECTORY/requirements_dev.txt --break-system-packages
+    micromamba run -p /opt/envs/pielenv pip install -e $PIEL_DIRECTORY --break-system-packages
+
diff --git a/environment/apptainer/start_apptainer.sh b/environment/apptainer/start_apptainer.sh
@@ -0,0 +1,3 @@
+apptainer instance start -w piel.sif pielapptainerdev
+apptainer run instance://pielapptainerdev
+apptainer shell instance://pielapptainerdev
diff --git a/environment/nix/install/start_piel_nix_shell.sh b/environment/nix/install/start_piel_nix_shell.sh
diff --git a/environment/nix/install/ubuntu_install_nix.sh b/environment/nix/install/ubuntu_install_nix.sh
@@ -5,4 +5,4 @@ sh <(curl -L https://nixos.org/nix/install) --no-daemon --yes
 . $HOME/.nix-profile/etc/profile.d/nix.sh
 nix-env -f "<nixpkgs>" -iA cachix
 cachix use openlane
-git clone https://github.com/efabless/openlane.git -o ../../../../
+git clone https://github.com/efabless/openlane2.git ../../../../openlane2
diff --git a/environment/nix/shell.nix b/environment/nix/shell.nix
@@ -3,7 +3,7 @@
 
 pkgs.mkShell {
   buildInputs = [
-    pkgs.jupyter
+    pkgs.stdenv.cc.cc.lib
     pkgs.which
     pkgs.htop
     pkgs.zlib
@@ -12,20 +12,37 @@ pkgs.mkShell {
     pkgs.gtkwave # 3.3.117, from Aug 2023 (latest)
     pkgs.xyce # 7.6, from Nov 2022 (7.7 is latest)
     pkgs.verilog # 12.0, from Jun 2023 (latest)
+    pkgs.nodejs
+    pkgs.micromamba
+    pkgs.vtk
   ];
 
+  nativeBuildInputs = [
+    pkgs.autoPatchelfHook
+  ];
 
   shellHook = ''
+    export LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath [
+      pkgs.stdenv.cc.cc
+    ]}
     if [ -e ../../.venv/bin/activate ];
-     then source ../../.venv/bin/activate;
+     then
+      # source ../../.venv/bin/activate;
+      echo "Environment configured";
     else
-      pip install --upgrade pip;
-      python -m venv ../../.venv;
-      source ../../.venv/bin/activate;
-      pip install -r ../../requirements_dev.txt;
-      pip install -r ../../../openlane2/requirements_dev.txt;
-      pip install -e ../../;
-      source ../../.venv/bin/activate;
+      # pip install --upgrade pip;
+      set -e
+      eval "$(micromamba shell hook --shell=bash)"
+      micromamba shell init --shell=bash --prefix=~/micromamba
+      micromamba create --yes -q -n pielenv -c conda-forge python=3.10
+      # micromamba install --yes --file ../../requirements_dev.txt -c conda-forge;
+      micromamba activate pielenv --yes
+      set +e
+      export PATH="$PATH:$HOME/.local/bin/"
+      micromamba run -n pielenv pip install -r ../../requirements_dev.txt --user --break-system-packages;
+      micromamba run -n pielenv pip install -r ../../../openlane2/requirements_dev.txt --user --break-system-packages;
+      micromamba run -n pielenv pip install -e ../../ --user --break-system-packages;
+      # source ../../.venv/bin/activate;
     fi
     nix-shell ../../../openlane2/shell.nix
   '';

diff --git a/environment/nix/shell_future.nix b/environment/nix/shell_future.nix
@@ -0,0 +1,34 @@
+# An example package with dependencies defined via pyproject.toml
+{
+  config,
+  lib,
+  dream2nix,
+}: let
+  pyproject = lib.importTOML (config.mkDerivation.src + ../../pyproject.toml);
+in {
+  imports = [
+    dream2nix.modules.dream2nix.pip
+  ];
+
+  inherit (pyproject.project) name version;
+
+  mkDerivation = {
+    src = ./.;
+  };
+
+  buildPythonPackage = {
+    format = lib.mkForce "pyproject";
+    pythonImportsCheck = [
+      "piel"
+    ];
+  };
+
+  pip = {
+    pypiSnapshotDate = "2023-08-27";
+    requirementsList =
+      pyproject.build-system.requires
+      or []
+      ++ pyproject.project.dependencies;
+    flattenDependencies = true;
+  };
+}
diff --git a/environment/nix/shell_old.nix b/environment/nix/shell_old.nix
@@ -0,0 +1,42 @@
+{ pkgs ? import <nixpkgs> {}
+}:
+
+pkgs.mkShell {
+  buildInputs = [
+    pkgs.stdenv.cc.cc.lib
+    pkgs.jupyter
+    pkgs.which
+    pkgs.htop
+    pkgs.zlib
+    pkgs.pandoc
+    pkgs.ngspice # 41 (latest)
+    pkgs.gtkwave # 3.3.117, from Aug 2023 (latest)
+    pkgs.xyce # 7.6, from Nov 2022 (7.7 is latest)
+    pkgs.verilog # 12.0, from Jun 2023 (latest)
+    pkgs.python3Packages.virtualenv # run virtualenv .
+    pkgs.python3Packages.numpy
+  ];
+
+  nativeBuildInputs = [
+    pkgs.autoPatchelfHook
+  ];
+
+  shellHook = ''
+    export LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath [
+      pkgs.stdenv.cc.cc
+    ]}
+    if [ -e ../../.venv/bin/activate ];
+     then source ../../.venv/bin/activate;
+    else
+      pip install --upgrade pip;
+      python -m venv ../../.venv;
+      source ../../.venv/bin/activate;
+      pip install -r ../../requirements_dev.txt;
+      pip install -r ../../../openlane2/requirements_dev.txt;
+      pip install -e ../../;
+      source ../../.venv/bin/activate;
+    fi
+    nix-shell ../../../openlane2/shell.nix
+  '';
+  LOCALE_ARCHIVE="/usr/lib/locale/locale-archive";  # let's nix read the LOCALE, to silence warning messages
+}
diff --git a/requirements_dev.txt b/requirements_dev.txt
@@ -16,6 +16,7 @@ femwell==0.1.8
 flake8
 kfactory[git,ipy]==0.7.5
 kweb==0.1.1
+gplugins
 gdsfactory==7.2.1 # Pinned for pydantic <v2 compatibility and examples compatibility till porting.
 hdl21==4.0.0
 ipyevents
@@ -24,6 +25,7 @@ ipywidgets>=7.6.0,<9
 jax==0.4.14
 jaxlib==0.4.14
 jupyter
+jupyter_core
 jupyter_bokeh
 jupyter_packaging~=0.7.9
 jupyterlab~=3.0