feat(build): Integrate and build turbomind backend directly in setup.py (#3726)

windreamer · web-flow · commit 200d87692306 · 2025-07-16T20:04:43.000+08:00
* fix: add cmake_build_extension for turbomind extension building.

* fix: use regex to extract version instead of exec().

* fix: make cmake_build_extention a build requirement for setuptools.

* fix: enable turbomind only with cuda, and fix cuda package dependencies.

* fix: fix yaml-cpp cmake policy issue by hotfix patch.

* fix: tweak for ci build.

* fix: auto detect cudaver for turbomaind.

* doc: update installation docs
diff --git a/.github/workflows/cuda11.8-whl-release.yml b/.github/workflows/cuda11.8-whl-release.yml
@@ -59,6 +59,9 @@ jobs:
         pyver: ['3.9', '3.10', '3.11', '3.12', '3.13']
     runs-on: windows-latest
     steps:
+      - name: Set git for windows
+        run: |
+          git config --global core.longpaths true
       - name: Checkout repository
         uses: actions/checkout@v3
       - name: Set up python
@@ -67,7 +70,7 @@ jobs:
           python-version: ${{ matrix.pyver }}
       - name: Install python packages
         run: |
-          pip install pybind11 wheel
+          pip install build change-wheel-version
       - name: Setup CUDA Toolkit
         id: cuda-toolkit
         shell: pwsh
@@ -76,19 +79,8 @@ jobs:
             INPUT_CUDA_VERSION: '11.8.0'
       - name: Build wheel
         run: |
-          mkdir build
-          cd build
-          pip install -U setuptools
-          ..\builder\windows\generate.ps1
-          cmake --build . --config Release -- /m /v:q
-          if (-Not $?) {
-            echo "build failed"
-            exit 1
-          }
-          cmake --install . --config Release
-          cd ..
-          rm build -Force -Recurse
-          python setup.py bdist_wheel -d build/wheel
+          python -m build --wheel -o build/wheel
+          change_wheel_version build/*.whl --local-version cu118 --delete-old-wheel
       - name: Upload Artifacts
         uses: actions/upload-artifact@v4
         with:
diff --git a/.github/workflows/linux-x64-gpu.yml b/.github/workflows/linux-x64-gpu.yml
@@ -55,6 +55,5 @@ jobs:
             cd /work
             source /opt/conda/bin/activate
             conda activate py310
-            mkdir build && cd build
-            bash ../generate.sh make
-            make -j$(nproc) && make install
+            pip install build
+            python -m build --wheel
diff --git a/.github/workflows/pr_ete_test.yml b/.github/workflows/pr_ete_test.yml
@@ -65,14 +65,11 @@ jobs:
           make -j$(nproc) && make install && cd .. && rm -rf openmpi-4.1.5*
           export PATH=$PATH:/usr/local/openmpi/bin
           export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/openmpi/lib
-          python3 -m pip install cmake packaging wheel transformers_stream_generator transformers==4.51.3 datasets openai einops timm decord
-          python3 -m pip install -r requirements_cuda.txt -r requirements/test.txt -r requirements/build.txt
-          mkdir -p build && cd build &&\
-          sh ../generate.sh &&\
-          ninja -j$(nproc) && ninja install &&\
-          cd .. &&\
-          python3 -m pip install -e . &&\
-          rm -rf build
+          # We need to pin transformers version (<4.52.0) to avoid test failures due to breaking changes.
+          python3 -m pip install transformers==4.51.3 timm
+          python3 -m pip install -r requirements/lite.txt
+          python3 -m pip install -r requirements/test.txt
+          python3 -m pip install -e .
           # the install packeage from. https://github.com/Dao-AILab/flash-attention/releases
           python3 -m pip install /nvme/qa_test_models/offline_pkg/flash_attn-2.8.0.post2-cp310-cp310-linux_x86_64.whl
       - name: Check env
diff --git a/.github/workflows/pypi.yml b/.github/workflows/pypi.yml
@@ -58,6 +58,9 @@ jobs:
         pyver: ['3.9', '3.10', '3.11', '3.12', '3.13']
     runs-on: windows-latest
     steps:
+      - name: Set git for windows
+        run: |
+          git config --global core.longpaths true
       - name: Checkout repository
         uses: actions/checkout@v3
       - name: Set up python
@@ -66,8 +69,7 @@ jobs:
           python-version: ${{ matrix.pyver }}
       - name: Install python packages
         run: |
-          pip install -r requirements/build.txt
-          pip install wheel
+          pip install build change-wheel-version
       - name: Setup CUDA Toolkit
         id: cuda-toolkit
         shell: pwsh
@@ -76,20 +78,8 @@ jobs:
             INPUT_CUDA_VERSION: '12.1.0'
       - name: Build wheel
         run: |
-          mkdir build
-          cd build
-          # https://github.com/pypa/setuptools/issues/1631
-          pip install -U setuptools
-          ..\builder\windows\generate.ps1
-          cmake --build . --config Release -- /m /v:q
-          if (-Not $?) {
-            echo "build failed"
-            exit 1
-          }
-          cmake --install . --config Release
-          cd ..
-          rm build -Force -Recurse
-          python setup.py bdist_wheel -d build/wheel
+          python -m build --wheel -o build/wheel
+          change_wheel_version build/*.whl --local-version cu121 --delete-old-wheel
       - name: Upload Artifacts
         uses: actions/upload-artifact@v4
         with:
diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml
@@ -57,33 +57,13 @@ jobs:
           python3 -V
           python3 -m pip cache dir
           python3 -m pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu118
-      - name: Build lmdeploy
-        run: |
-          python3 -m pip install cmake matplotlib
-          python3 -m pip install -r requirements/build.txt
-          mkdir build
-          cd build
-          cp -r /root/packages/_deps _deps
-          cmake .. \
-              -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-              -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
-              -DCMAKE_INSTALL_PREFIX=./install \
-              -DBUILD_PY_FFI=ON \
-              -DBUILD_MULTI_GPU=OFF \
-              -DCMAKE_CUDA_FLAGS="-lineinfo" \
-              -DUSE_NVTX=ON \
-              -DSM=80 \
-              -DCMAKE_CUDA_ARCHITECTURES=80 \
-              -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
-              -DBUILD_TEST=OFF
-          make -j$(nproc) && make install
       - name: Install lmdeploy
         run: |
-          python3 -m pip install pynvml packaging protobuf transformers_stream_generator
+          python3 -m pip install pynvml packaging protobuf transformers_stream_generator matplotlib
           # manually install flash attn
           python3 -m pip install /root/packages/cu118/flash_attn-*.whl
           python3 -m pip install -r requirements_cuda.txt -r requirements/test.txt
-          python3 -m pip install .
+          python3 -m pip install -e .
       - name: Check env
         run: |
           python3 -m pip list
diff --git a/.github/workflows/windows-x64-gpu.yml b/.github/workflows/windows-x64-gpu.yml
@@ -32,6 +32,9 @@ jobs:
     name: cuda-${{ matrix.cudaver }}
     runs-on: windows-latest
     steps:
+      - name: Set git for windows
+        run: |
+          git config --global core.longpaths true
       - name: Checkout repository
         uses: actions/checkout@v3
       - name: Set up python
@@ -40,8 +43,7 @@ jobs:
           python-version: '3.10'
       - name: Install python packages
         run: |
-          pip install -r requirements/build.txt
-          pip install wheel
+          pip install build
       - name: Setup CUDA Toolkit
         id: cuda-toolkit
         shell: pwsh
@@ -50,16 +52,4 @@ jobs:
             INPUT_CUDA_VERSION: ${{ matrix.cudaver }}
       - name: Build wheel
         run: |
-          $env:BUILD_TEST="OFF"
-          mkdir build
-          cd build
-          ..\builder\windows\generate.ps1
-          cmake --build . --config Release -- /m /v:n
-          if (-Not $?) {
-            echo "build failed"
-            exit 1
-          }
-          cmake --install . --config Release
-          cd ..
-          rm build -Force -Recurse
-          python setup.py bdist_wheel -d build/wheel
+          python -m build --wheel
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -27,6 +27,7 @@ repos:
         args: ["--remove"]
       - id: mixed-line-ending
         args: ["--fix=lf"]
+
   - repo: https://github.com/executablebooks/mdformat
     rev: 0.7.9
     hooks:
@@ -55,3 +56,8 @@ repos:
     hooks:
     -   id: check-copyright
         args: ["lmdeploy"]
+
+exclude: |
+  (?x)(
+    ^cmake/.*\.patch$
+  )
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -18,7 +18,7 @@ project(TurboMind LANGUAGES CXX CUDA)
 if (MSVC)
     # use standard conformant preprocessor
     add_compile_options($<$<COMPILE_LANGUAGE:CXX>:/Zc:preprocessor>)
-    set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Xcompiler=/Zc:preprocessor")
+    set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Xcompiler=/Zc:preprocessor -Xcompiler=/Zc:__cplusplus")
 endif ()
 
 find_package(CUDAToolkit REQUIRED)
@@ -52,6 +52,7 @@ FetchContent_Declare(
   GIT_REPOSITORY https://github.com/NVIDIA/cutlass.git
   GIT_TAG        v3.9.2
   GIT_SHALLOW ON
+  EXCLUDE_FROM_ALL
 )
 
 set(CUTLASS_ENABLE_SM90_EXTENDED_MMA_SHAPES ON CACHE BOOL "Enable extended GMMA shapes")
@@ -63,6 +64,8 @@ FetchContent_Declare(
   yaml-cpp
   GIT_REPOSITORY https://github.com/jbeder/yaml-cpp.git
   GIT_TAG 0.8.0
+  PATCH_COMMAND git apply ${CMAKE_CURRENT_SOURCE_DIR}/cmake/yaml-cpp_cmake_policy.patch
+  UPDATE_DISCONNECTED 1
 )
 set(YAML_BUILD_SHARED_LIBS OFF CACHE BOOL "Build static library of yaml-cpp")
 FetchContent_MakeAvailable(yaml-cpp)
@@ -233,5 +236,9 @@ add_subdirectory(src)
 
 # install python api
 if (BUILD_PY_FFI)
-  install(TARGETS _turbomind DESTINATION ${CMAKE_SOURCE_DIR}/lmdeploy/lib)
+  if (CALL_FROM_SETUP_PY)
+    install(TARGETS _turbomind DESTINATION ${CMAKE_INSTALL_PREFIX})
+  else()
+    install(TARGETS _turbomind DESTINATION ${CMAKE_SOURCE_DIR}/lmdeploy/lib)
+  endif()
 endif ()
diff --git a/builder/manywheel/entrypoint_build.sh b/builder/manywheel/entrypoint_build.sh
@@ -5,24 +5,19 @@ export PYTHON_VERSION=$PYTHON_VERSION
 export PLAT_NAME=$PLAT_NAME
 export USERID=${USERID}
 export GROUPID=${GROUPID}
-export CUDAVER=$(nvcc --version | sed -n 's/^.*release \([0-9]\+\).*$/\1/p')
 export NCCL_INCLUDE_DIR=/usr/local/cuda/include
 export NCCL_LIB_DIR=/usr/local/cuda/lib64
 
 source /opt/conda/bin/activate
 conda activate $PYTHON_VERSION
 
 cd lmdeploy
-rm -rf lmdeploy/lib
-mkdir -p build && cd build && rm -rf *
-bash ../generate.sh make
-make -j$(nproc) && make install
-if [ $? != 0 ]; then
-    echo "build failed"
-    exit 1
-fi
-cd ..
-rm -rf build
-python setup.py bdist_wheel --cuda=${CUDAVER} --plat-name $PLAT_NAME -d /tmpbuild/
+pip install build change-wheel-version
+python -m build --wheel -o /tmpbuild/
+for file in $(find /tmpbuild/ -name "*.whl")
+do
+    platform_tag="$(basename $file | cut -d- -f3-4)-${PLAT_NAME}"
+    change_wheel_version /tmpbuild/*.whl --delete-old-wheel --platform-tag ${platform_tag}
+done
 chown ${USERID}:${GROUPID} /tmpbuild/*
 mv /tmpbuild/* /lmdeploy_build/
diff --git a/builder/windows/README.md b/builder/windows/README.md
@@ -9,12 +9,6 @@
 ## Build lmdeploy wheel
 
 ```powershell
-mkdir build
-cd build
-..\builder\windows\generate.ps1
-cmake --build . --config Release -- /m
-cmake --install . --config Release
-cd ..
-rm build -Force -Recurse
-python setup.py bdist_wheel -d build\wheel
+pip install build
+python -m build --wheel
 ```
diff --git a/cmake/yaml-cpp_cmake_policy.patch b/cmake/yaml-cpp_cmake_policy.patch
@@ -0,0 +1,11 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 46dc180..b746ac1 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -1,5 +1,5 @@
+ # 3.5 is actually available almost everywhere, but this a good minimum
+-cmake_minimum_required(VERSION 3.4)
++cmake_minimum_required(VERSION 3.5)
+ 
+ # enable MSVC_RUNTIME_LIBRARY target property
+ # see https://cmake.org/cmake/help/latest/policy/CMP0091.html
diff --git a/docs/en/get_started/installation.md b/docs/en/get_started/installation.md
@@ -34,47 +34,24 @@ The release frequency of LMDeploy is approximately once or twice monthly. If you
 
 ## Install from source
 
-If you are using the PyTorch Engine for inference, the installation from the source is quite simple:
+By default, LMDeploy will build with NVIDIA CUDA support, utilizing both the Turbomind and PyTorch backends. Before installing LMDeploy, ensure you have successfully installed the CUDA Toolkit.
 
-```shell
-git clone https://github.com/InternLM/lmdeploy.git
-cd lmdeploy
-pip install -e .
-```
-
-But if you are using the TurboMind Engine, you have to build the source as shown below. The `openmmlab/lmdeploy:{tag}` docker image is strongly recommended.
-
-**Step 1** - Get the docker image of LMDeploy
+Once the CUDA toolkit is successfully set up, you can build and install LMDeploy with a single command:
 
 ```shell
-docker pull openmmlab/lmdeploy:latest
+pip install git+https://github.com/InternLM/lmdeploy.git
 ```
 
-```{note}
-The "openmmlab/lmdeploy:latest" is based on "nvidia/cuda:12.4.1-devel-ubuntu22.04". If you are working on a platform with cuda 11+ driver, please use "openmmlab/lmdeploy:latest-cu11".
-The pattern of the LMDeploy docker image tag is "openmmlab/lmdeploy:{version}-cu(11|12)" since v0.5.3.
-```
-
-**Step 2** - Clone LMDeploy source code and change to its root directory
+You can also explicitly disable the Turbomind backend to avoid CUDA compilation by setting the `DISABLE_TURBOMIND` environment variable:
 
 ```shell
-git clone https://github.com/InternLM/lmdeploy.git
-cd lmdeploy
+DISABLE_TURBOMIND=1 pip install git+https://github.com/InternLM/lmdeploy.git
 ```
 
-**Step 3** - launch docker container in interactive mode
+If you prefer a specific version instead of the `main` branch of LMDeploy, you can specify it in your command:
 
 ```shell
-docker run --gpus all --net host --shm-size 16g -v $(pwd):/opt/lmdeploy --name lmdeploy -it openmmlab/lmdeploy:latest /bin/bash
+pip install https://github.com/InternLM/lmdeploy/archive/refs/tags/v0.9.2.zip
 ```
 
-**Step 4** - build and installation
-
-```shell
-cd /opt/lmdeploy
-mkdir -p build && cd build
-bash ../generate.sh make
-make -j$(nproc) && make install
-cd ..
-pip install -e .
-```
+If you want to build LMDeploy with support for Ascend, Cambricon, or MACA, install LMDeploy with the corresponding `LMDEPLOY_TARGET_DEVICE` environment variable.
diff --git a/docs/zh_cn/get_started/installation.md b/docs/zh_cn/get_started/installation.md
diff --git a/generate.sh b/generate.sh
diff --git a/lmdeploy/turbomind/generate_gemm_config.py b/lmdeploy/turbomind/generate_gemm_config.py
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/requirements/build.txt b/requirements/build.txt
diff --git a/setup.py b/setup.py
