Build Updates for Ubuntu24.04 #7758
Conversation
This reverts commit 8407cef.
TODO: check on user ID 1000
| RUN apt update -q=2 \ | ||
| && apt install -y gpg wget \ | ||
| && wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null \ | ||
| && . /etc/os-release \ | ||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | ||
| && apt-get update -q=2 \ | ||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* | ||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* |
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
There was a problem hiding this comment.
The version installed is 3.28.3, which includes the bugfix for above issue - Kitware/CMake@f69b9b7
I could pin it further to include patch version, if you think that is better. I don't mind using a recent release as well, just not sure if it will introduce any new issues. What would you recommend?
| @@ -96,7 +97,7 @@ RUN apt update -q=2 \ | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* \ | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* \ | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -1107,7 +1106,7 @@ def create_dockerfile_buildbase(ddir, dockerfile_name, argmap): | |||
| && . /etc/os-release \\ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \\ | |||
| && apt-get update -q=2 \\ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -45,7 +45,7 @@ apt update -q=2 \ | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* \ | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* \ | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -51,7 +51,7 @@ install_build_deps() { | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -48,7 +48,7 @@ apt update -q=2 \ | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* \ | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* \ | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -66,7 +66,7 @@ apt update -q=2 \ | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* rapidjson-dev | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* rapidjson-dev | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -38,7 +38,7 @@ apt update -q=2 \ | |||
| && . /etc/os-release \ | |||
| && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ | |||
| && apt-get update -q=2 \ | |||
| && apt-get install -y --no-install-recommends cmake=3.27.7* cmake-data=3.27.7* | |||
| && apt-get install -y --no-install-recommends cmake=3.28.* cmake-data=3.28.* | |||
There was a problem hiding this comment.
Version 3.28 has a bug with CUDA Toolkit, it was fixed in latest versions.
https://gitlab.kitware.com/cmake/cmake/-/issues/25484
| @@ -221,7 +221,7 @@ ExternalProject_Add(triton-server | |||
| -Dre2_DIR:PATH=${TRITON_THIRD_PARTY_INSTALL_PREFIX}/re2/${LIB_DIR}/cmake/re2 | |||
| -Dabsl_DIR:PATH=${TRITON_THIRD_PARTY_INSTALL_PREFIX}/absl/${LIB_DIR}/cmake/absl | |||
| -DCURL_DIR:STRING=${TRITON_THIRD_PARTY_INSTALL_PREFIX}/curl/${LIB_DIR}/cmake/CURL | |||
| -Dnlohmann_json_DIR:PATH=${TRITON_THIRD_PARTY_INSTALL_PREFIX}/nlohmann_json/${LIB_DIR}/cmake/nlohmann_json | |||
| -Dnlohmann_json_DIR:PATH=${TRITON_THIRD_PARTY_INSTALL_PREFIX}/nlohmann_json/share/cmake/nlohmann_json | |||
There was a problem hiding this comment.
I this location valid for RHEL as well?
There was a problem hiding this comment.
yes, the location was updated in newer version of nlohmann_json so should work for all
| @@ -59,20 +59,19 @@ RUN apt-get update && \ | |||
| libboost-dev \ | |||
| python3-dev \ | |||
| python3-pip \ | |||
| python3-wheel \ | |||
| python3-setuptools \ | |||
There was a problem hiding this comment.
Wy awe adding setuptools package?
| @@ -78,15 +79,15 @@ RUN apt-get update && \ | |||
| python3 \ | |||
| python3-pip \ | |||
| python3-dev \ | |||
| python3-wheel \ | |||
| python3-setuptools \ | |||
There was a problem hiding this comment.
Why we are adding setuptools package?
There was a problem hiding this comment.
It was removed from the pip install below and added here, cause of the new pip behaviour with system installations. No new install overall
| @@ -57,6 +57,7 @@ FROM ${BASE_IMAGE} AS sdk_build | |||
|
|
|||
| # Ensure apt-get won't prompt for selecting options | |||
| ENV DEBIAN_FRONTEND=noninteractive | |||
| ENV PIP_BREAK_SYSTEM_PACKAGES=1 | |||
There was a problem hiding this comment.
--break-system-packages
Allow pip to modify an EXTERNALLY-MANAGED Python installation
Could you please clarify which package require it?
There was a problem hiding this comment.
All non virtual env installation via pip require this flag now - https://stackoverflow.com/questions/75602063/pip-install-r-requirements-txt-is-failing-this-environment-is-externally-mana
I have added this flag for now to avoid failure down the stack and avoid updating all pip commands. We would need to evaluate switching to python venv's in future for pip install commands
There was a problem hiding this comment.
I see that link you are referring to is related to https://peps.python.org/pep-0668/ which is outdated.
Could you please elaborate your decision.
There was a problem hiding this comment.
Link further points to updated links i think. Anyway, pip has a behaviour change wrt to all system wide installation (non virtual env install). Python base system environment is considered as externally managed and fails to install any packages without this flag. Recommendation from pip side is to either use OS package manager to install such packages on system level, use new pipx tool which behind the scene creates a temporary venv and installs things there (mainly targeted for any CLI python tools) or switch to a virtual env install. You can read the full error message on above stackoverflow link.
This PIP_BREAK_SYSTEM_PACKAGES flag allows us to override this new behavior and continue running pip install commands without any venv activated. However, it still wont allow any upgrades to debian installed packags like python3-pip, wheel, setuptools etc. So i switched whatever packages are available as debian install to install via apt and rest pip install commands will utilize this flag to continue for now with base installation.
There was a problem hiding this comment.
Then logical way here should be usage of virtual environment otherwise we are mixing up system and user space.
There was a problem hiding this comment.
Yes, i agree. But right now we are using pip install at a lot of places, so i wasn't sure how to best update/integrate everything with a virtual env
Thanks for submitting a PR to Triton!
Please go the the
Previewtab above this description box and select the appropriate sub-template:If you already created the PR, please replace this message with one of
and fill it out.