DRAFT: Add bash based build on Ubuntu for faiss library and prerequisites #64

rstarmer · 2023-07-12T18:51:47Z

add_dev.sh: ensure disk space for build and python prerequisites
build-prereqs.sh: install prerequisites (intel, nvidia)
build-faiss.sh: build faiss libs and python (and python swig) libs

Produces a python .whl
Produces a .tar.gz of the generated libraries

add_dev.sh: ensure disk space for build and python prerequisites build-prereqs.sh: install prerequisites (intel, nvidia) build-faiss.sh: build faiss libs and python (and python swig) libs Produces a python .whl Produces a .tar.gz of the generated libraries

sdake · 2023-07-15T02:15:58Z

This is not specifically related to this draft; its the next step in the build pipeline.

https://github.com/artificialwisdomai/apt.oifv.ai/blob/main/build.sh

build.sh - build as "root" configured for docker/podman README.md - instructions for running podman build add_dev/build-prereqs.sh - shuffle python install

sdake

There is a lot of repetition and redundancy in this set of scripts. If this is for your personal use, I don't mind merging it with a README.md stating as such. If its for general developer use, it should be cleaned up significantly.

The following activities should be isolated:

base os installation
upstream base os dependency installation
upstream package build (per package)
custom package build installation
custom package build test

There may be other things. The jist is that these actions should not be strongly coupled, as they are in this PR set.

All that said, happy to merge as is, once the top level dir makes sense. Such as

platform/packaging/dev or platform/dev/. I am not sure what you are after as this PR has many directions, so I am not sure what to suggest as a top level dir. platform/build-faiss doesn't make sense, as follow-on work to build a new package (e.g. autofaiss) may set an expectation that all of these shell scripts are needed.

Which I want to avoid.

sdake · 2023-07-16T14:19:48Z

.gitignore

@@ -1 +1,4 @@
-build*/
+build/
+faiss*


I don't follow the need for these .gitignore changes. With this, a git status will not show the dirty state of the repository in these directories.

I don't think that is right.

sdake · 2023-07-16T14:21:18Z

platform/build-faiss/README.md

+sudo apt install podman -y
+```
+
+Run build in podman:


I am familiar with podman. Dan Walsh is a great fella, although, with Red Hat's recent changes, it is not clear to me that we should accept a dependency on any single-vendor project, especially those sponsored directly by Red Hat.

Good point. I personally would recommend ociTools if we want to efficiently declare containers, which would let us rely on a much better-supported project with a more diverse user base. For now, I view Dockerfiles as a happy intermediate way to use either Docker or Podman, and we can always just remember to type "docker" on Docker-based machines.

IMO, this is where its ↩ https://github.com/containerd/nerdctl

We could have continuous battles about what tools we use. These battle, ie. religious wars, are why (successful) tech firms build their own toolchain, top to bottom.

Are we, as a team of founding engineers, willing to adopt a pragmatic perspective?

Sure. Pragmatically, containers run on k8s, Oracle, and Podman. That's it. Docker is a fading bad dream. For me, what runs the containers is uninteresting, thanks to OCI. And what builds the containers is also uninteresting, as long as it's reproducible; I can learn whatever builder you want to use.

You state it well. We should, as a firm, attempt to eliminate ambiguity aversion. We do that by working on high-value areas. Packaging is not high value unless it directly relates to distribution.

I think we understand our differing points of view. I also believe we all agree that containerd is the runtime we will use, if we need to run container workloads. All daemon (docker and a million clones including cog. In addition there are rootless variants (podman, nerdctl, and many more).

containerd is one of many OCI runtimes. This runtime is widely used and has wide investment across the technology ecosystem. This component provides:

Lifecycle Management (LCM):

start

stop

delete

Runtime:

exec

logs

FWIIW,I agree, docker is and was a massive swing and a miss.

sdake · 2023-07-16T14:23:02Z

platform/build-faiss/add_dev.sh

+
+# mount nvme disk on /models
+sudo mkdir /models
+sudo mkfs.xfs /dev/nvme0n1


Not a fan of destructive operations in shell scripts. Especially those that could eat the entire filesystem if in error.

your assumption in line 30 sort of covers this. Sort of, as in no destructive operation should be done anywhere without extreme clarity.. I would prefer you pull this destructive operatoin out, as I could never run add_dev.sh on beast, as it would wipe my main disk...........

sdake · 2023-07-16T14:24:52Z

platform/build-faiss/add_dev.sh

+mkdir /models/dev
+ln -s /models/dev
+mkdir /models/local
+ln -s /models/local ~/.local


The goal of this script is a little mysterious. It appears like it is meant to create local storage for development? Is that faiss specific? If not, could you pull this file out, and put it in a different PR? Also, a comment at the top of the file describing the purpose would help.

sdake · 2023-07-16T14:26:10Z

platform/build-faiss/build-faiss.sh

+git clone https://github.com/facebookresearch/faiss
+cd faiss
+
+# Configure paths and set environment variables


nit: pointless comment. The problem with these types of comments is, over time, they introduce code smells.

sdake · 2023-07-16T14:41:45Z

platform/build-faiss/build-prereqs.sh

+
+# Add a couple Python prerequisites
+pip install -U pip setuptools wheel
+pip install numpy swig torch


sdake · 2023-07-16T14:43:34Z

platform/build-faiss/build-prereqs.sh

+
+
+sudo -E apt update
+sudo -E apt install dkms intel-basekit -y


Why is dkms explicitly needed? I know intel-basekit installs drivers, although I don't think we require all of OneAPI. The part we do require does not possess drivers. Is the problem that dkms is an unspecified dependency by intel?

sdake · 2023-07-16T14:44:06Z

platform/build-faiss/build-prereqs.sh

+
+## Get CUDA and install it
+
+curl -sLO https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run


Why not use distro packaging built by NVIDIA?

platform/build-faiss/build-prereqs.sh

sdake · 2023-07-16T14:46:05Z

platform/build-faiss/build-prereqs.sh

+
+#Verify python and pytorch work
+
+python3 -c 'import torch; print(f"Is CUDA Available: {torch.cuda.is_available()}")'


How about the comment (f"Pytorch and CUDA are operating correctly). The problem is pytorch is not installed...

sdake · 2023-07-22T10:42:14Z

#77

sdake · 2023-07-22T10:43:56Z

here is my go.

#78

These two PRs represent two forward passes.

Thanks
-steve

sdake

Please re-home as follows:

/origin/platform
/origin/platform/build/faiss
/origin/workloads/retrofromer

Separately we should move packaging to build as packaging is a type of build process.

Besides the destructive operation on the disk, this looks great @rstarmer !

sdake · 2023-07-29T02:24:24Z

platform/build-faiss/README.md

+sudo apt install podman -y
+```
+
+Run build in podman:


I think we understand our differing points of view. I also believe we all agree that containerd is the runtime we will use, if we need to run container workloads. All daemon (docker and a million clones including cog. In addition there are rootless variants (podman, nerdctl, and many more).

containerd is one of many OCI runtimes. This runtime is widely used and has wide investment across the technology ecosystem. This component provides:

Lifecycle Management (LCM):

start

stop

delete

Runtime:

exec

logs

FWIIW,I agree, docker is and was a massive swing and a miss.

sdake · 2023-07-29T02:25:52Z

platform/build-faiss/README.md

+
+Run build in podman:
+
+```sh


console is the only markup in .md that github understands.

sdake · 2023-07-29T02:27:24Z

platform/build-faiss/README.md

+
+  a python wheel for faiss deployment
+
+* faiss-libs.tgz


nice! I am pulling this PR now to see it in action!

sdake · 2023-07-29T02:28:41Z

platform/build-faiss/add_dev.sh

+
+# mount nvme disk on /models
+sudo mkdir /models
+sudo mkfs.xfs /dev/nvme0n1


your assumption in line 30 sort of covers this. Sort of, as in no destructive operation should be done anywhere without extreme clarity.. I would prefer you pull this destructive operatoin out, as I could never run add_dev.sh on beast, as it would wipe my main disk...........

platform/build-faiss/build-faiss.sh

sdake · 2023-07-29T02:33:57Z

platform/build-faiss/build-prereqs.sh

+export PATH=$HOME/.local/bin:$PATH
+export DEBIAN_FRONTEND=noninteractive
+
+cat <<EOF | sudo tee /etc/apt/sources.list.d/debian-contrib.list


like this approach, I am borrowing it!

platform/build-faiss/build-prereqs.sh

sdake · 2023-07-29T02:38:21Z

platform/build-faiss/build.sh

+
+## Get CUDA and install it
+
+curl -sLO https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run


why not use the apt packaging? Sure, the run packaging works, was this just testing?

sdake · 2023-07-29T02:39:04Z

platform/build-faiss/build.sh

+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+dpkg -i cuda-keyring_1.1-1_all.deb
+apt-get update
+apt-get install cmake cuda-toolkit -y


strongly conflicts with 45.

rstarmer requested a review from sdake as a code owner July 12, 2023 18:51

cla-bot bot added the CLA CLA signed? label Jul 12, 2023

rstarmer marked this pull request as draft July 12, 2023 18:52

Add docker build.sh

1e3c079

build.sh - build as "root" configured for docker/podman README.md - instructions for running podman build add_dev/build-prereqs.sh - shuffle python install

sdake requested changes Jul 16, 2023

View reviewed changes

sdake mentioned this pull request Jul 17, 2023

Bump FAISS container base to Debian 12 and fix up various issues #73

Merged

rstarmer added 2 commits July 22, 2023 19:16

Fix erroneous linebreak in prereqs

998f97a

Update build and preqs add run_training

6aa2974

sdake reviewed Jul 29, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRAFT: Add bash based build on Ubuntu for faiss library and prerequisites #64

DRAFT: Add bash based build on Ubuntu for faiss library and prerequisites #64

rstarmer commented Jul 12, 2023

sdake commented Jul 15, 2023

sdake left a comment

sdake Jul 16, 2023

sdake Jul 16, 2023

MostAwesomeDude Jul 16, 2023

sdake Jul 17, 2023

MostAwesomeDude Jul 17, 2023

sdake Jul 22, 2023

sdake Jul 29, 2023

sdake Jul 16, 2023

sdake Jul 29, 2023

sdake Jul 16, 2023

sdake Jul 16, 2023

sdake Jul 16, 2023

sdake Jul 16, 2023

sdake Jul 16, 2023

sdake Jul 16, 2023

sdake commented Jul 22, 2023

sdake commented Jul 22, 2023

sdake left a comment

sdake Jul 29, 2023

sdake Jul 29, 2023

sdake Jul 29, 2023

sdake Jul 29, 2023

sdake Jul 29, 2023

sdake Jul 29, 2023

sdake Jul 29, 2023

@@ @@ -1 +1,4 @@ @@
-              build*/
+              build/
+              faiss*


		## Get CUDA and install it

		curl -sLO https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run


		#Verify python and pytorch work

		python3 -c 'import torch; print(f"Is CUDA Available: {torch.cuda.is_available()}")'


		Run build in podman:

		```sh

DRAFT: Add bash based build on Ubuntu for faiss library and prerequisites #64

Are you sure you want to change the base?

DRAFT: Add bash based build on Ubuntu for faiss library and prerequisites #64

Conversation

rstarmer commented Jul 12, 2023

sdake commented Jul 15, 2023

sdake left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdake commented Jul 22, 2023

sdake commented Jul 22, 2023

sdake left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment