This setup guide provides step-by-step instructions for setting up the environment and installing the dependencies for performing fingerprinting on models. This addresses both the bare metal and AWS EC2 environments.
The recommended AMI for EC2 is Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.3.0 (Amazon Linux 2) 20240625
. This AMI already installs necessary python version and CUDA toolkit. After choosing this AMI, you can follow the following steps to setup the environment.
python3 -m venv .venv
source .venv/bin/activate
pip3 install --upgrade pip
pip3 install -r requirements.txt
It is observed that Deepspeed conflicts with the installation from the requirements.txt. So, we recommend to install it from source. DS_BUILD_OPS=1
is required to build the ops AoT instead of the default JIT compilation of options.
git clone https://github.com/microsoft/DeepSpeed.git /tmp/DeepSpeed && \
cd /tmp/DeepSpeed && \
DS_BUILD_OPS=1 \
pip install . --no-build-isolation && \
rm -rf /tmp/DeepSpeed
This should allow you to run the finetune_multigpu.py
and other scripts and fingerprint models.
For bare metal, you can use docker/cpu/base/Dockerfile or docker/cuda/base/Dockerfile to build the image and run the scripts. This ensures reproducibility and consistency across different machines. For instructions on how to use these Dockerfiles, refer to these docs. If you want to run the scripts without Docker, you can follow the following steps to setup the environment.
Ths scripts work with Python >= 3.10.14. If you don't have compatible version, you can install it using the following steps on Ubuntu 22.04 otherwise skip this section. For OSes other than Ubuntu, this guide might be helpful.
sudo apt update &&
sudo apt install -y \
wget build-essential \
zlib1g-dev libffi-dev libssl-dev \
libbz2-dev libreadline-dev \
libsqlite3-dev libncurses5-dev \
libgdbm-dev libnss3-dev liblzma-dev
wget https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz
tar -xvf Python-3.10.14.tgz
cd Python-3.10.14
./configure --enable-optimizations
make -j$(nproc)
sudo make altinstall
We recommend installing CUDA toolkit version 12.1 and nvcc
. Installation instructions for the same on Ubuntu 22.04 are provided here:
sudo apt install -y \
build-essential \
wget \
curl \
gnupg2 \
ca-certificates
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" \
> /etc/apt/sources.list.d/cuda.list
sudo apt-get update && \
sudo apt-get install -y \
cuda-toolkit-12-1
Once you have the CUDA toolkit and necessary python version, you can setup the environment following the steps specified in the Installation steps on AWS EC2 section.