Wheels: ADIOS2 -> v2.9.0, blosc2 (#1387)

* Update blosc -> blosc2, ADIOS2 -> v2.9.0-rc1 ADIOS2 should be replace by v2.9.0 finally * Fix Blosc2 CMake params * Guard against CMAKE_OSX_ARCHITECTURES * For now, deactivate the pthread patch in ADIOS2 * Test small patch for blosc2 CMake * Add -fPIC flag * MacOS compatibility * -DADIOS2_USE_Blosc2=ON * Build atl and ffs manually to avoid CMake import error * Revert "Build atl and ffs manually to avoid CMake import error" This reverts commit c5469a3. * RC1 -> full release * Don't build ADIOS1 * -DADIOS2_USE_MHS=OFF * Use -DADIOS2_INSTALL_GENERATE_CONFIG=OFF workaround See ornladios/ADIOS2#3348 (comment) * Do not disable find calls * Revert "Do not disable find calls" This reverts commit 7065b89. * Try setting ADIOS2_PATH * Keep ADIOS1 + c-blosc1 * Undo `build.yml` Changes * ADIOS2: no c-blosc1 pthread issue * HDF5: 1.14.1-2 * macOS CI: Now has 3 cores * C-Blosc2 Updates - CMake options - Windows - build stamp * Windows Updates * HDF5 Patches * C-Blosc 2.9.3 + CMake Patch * ARM: H5Detect * Windows: Avoid CXX20 in C-Blosc2 * macOS arm64 HDF5: H5detect cross-compile * HDF5 arm64: cross-compiling H5detect * [Patch] ADIOS2 c-blosc2 windows.h std::min * [Patch] C-Blosc2: External Zlib * [Hack] Windows: Ignore Ext. ZLIB for c-blosc2 Just build again and close eyes. * ppc64le: Skip C-Blosc & ADIOS1 Due to time limits on Travis-CI. * Unix: Keep HDF5 1.12.2 for Now - issues with cross-compile cannot quickly be solved for macOS arm64/aarch64 - disable HDF5 support in ADIOS2 for now * [Patch] C-Blosc2 PUBLIC Linkage Zlib * Use Blosc2Config.cmake Package * Cleanup and `cat` ADIOS2 Config File * [Patch] [Hack] CMake --trace-expand * Cleanup: cat adios2-config.cmake * [Travis] PPC64le: `travis_wait 30` Takes longer now w/o output to the terminal than the default wait time of 10min. https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received * [ADIOS2] CMake --trace-expand * Undo Trace Expand Hacks * [Travis] PPC64le: `travis_wait 45` * ADIOS2: w/ SST * Blosc2: CMake Config PR Merged * macOS (x86): 10.15 -> 11.0 The macOS 10.15 build images on GH Actions are gone, so we remove this EOL platform for good now. * Windows: Skip Building Blosc1 Co-authored-by: Franz Pöschel <[email protected]> * Python: 3.8+ Skip older Python releases, 3.7 is EOL since last month. * [Patch] ADIOS2: Remove FindBlosc2.cmake * C-Blosc 2.10.1 * Win: ADIOS 2.9.1 * ADIOS2_Blosc2_PREFER_SHARED=OFF ornladios/ADIOS2#3715 (comment) * cibuildwheel 2.12.1 -> 2.14.1 For `pp310-manylinux_aarch64` --------- Co-authored-by: Axel Huebl <[email protected]>
openPMD · Aug 17, 2023 · 9d46036 · 9d46036
1 parent 25dec08
commit 9d46036
Show file tree

Hide file tree

Showing 4 changed files with 150 additions and 104 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -40,10 +40,10 @@ jobs:
               CMAKE_GENERATOR: "Visual Studio 16 2019"
               CMAKE_GENERATOR_PLATFORM: "Win32"
 
-          - os: macos-10.15
+          - os: macos-11
             arch: "x86_64"
             env:
-              MACOSX_DEPLOYMENT_TARGET: 10.15
+              MACOSX_DEPLOYMENT_TARGET: 11.0
 
           # Apple Silicon M1/arm64/aarch64 builds:
           #   https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon
@@ -79,55 +79,45 @@ jobs:
     - uses: actions/setup-python@v1
       name: Install Python
       with:
-        python-version: '3.7'
+        python-version: '3.8'
 
     - name: Install cibuildwheel
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install cibuildwheel==2.12.1
-
-#    # 0.15.0: CMake: Fix Python Install Directory
-#    # https://github.com/openPMD/openPMD-api/pull/1393
-#    - name: Download Patch 1/2
-#      uses: suisei-cn/actions-download-file@v1
-#      id: patch1
-#      with:
-#        url: "https://github.com/ax3l/openPMD-api/commit/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch"
-#        target: src/.patch/
-#
-#    # 0.15.0.post2 bump
-#    - name: Download Patch 2/2
-#      uses: suisei-cn/actions-download-file@v1
-#      id: setupversion
-#      with:
-#        url: "https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/023cbd69e06715e5191ece741bb1de56560a9f96/0001-Bump-setup.py.patch"
-#        target: src/.patch/
-
-#    - name: Apply Patches
-#      run: |
-#        python -m pip install "patch==1.*"
-#        cd src
-#        python -m patch .patch/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch
-#        python -m patch .patch/0001-Bump-setup.py.patch
+        python -m pip install cibuildwheel==2.14.1
+
+    # Hack: cmake --trace-expand
+    #- name: Download Patch 2/2
+    #  uses: suisei-cn/actions-download-file@v1
+    #  id: setupversion
+    #  with:
+    #    url: "https://gist.githubusercontent.com/ax3l/9b15c5bc73d7f3c1c66784187a016474/raw/e31dd54498c9c8fc465a7cbb6f900f768b9d0c0b/0001-Hack-setup.py-CMake-trace-expand.patch"
+    #    target: src/.patch/
+
+    #- name: Apply Patches
+    #  run: |
+    #    python -m pip install "patch==1.*"
+    #    cd src
+    #    python -m patch .patch/0001-Hack-setup.py-CMake-trace-expand.patch
 
     - name: Build wheel
       env:
         # (1) Disable PyPy (manylinux image: yum repo issues)
         #     https://github.com/pypa/manylinux/issues/899
         CIBW_SKIP: "pp*-manylinux*"
         CIBW_ARCHS: "${{ matrix.arch }}"
-        CIBW_PROJECT_REQUIRES_PYTHON: ">=3.7"
+        CIBW_PROJECT_REQUIRES_PYTHON: ">=3.8"
         # Install dependencies
         CIBW_BEFORE_BUILD_LINUX: bash -x .github/library_builders.sh
         CIBW_BEFORE_BUILD_MACOS: bash -x .github/library_builders.sh
         CIBW_BEFORE_BUILD_WINDOWS: 'cmd /E:ON /V:ON /C .github\library_builders.bat'
         # for the openPMD-api build, CMake shall search for
         # static dependencies of HDF5 and ADIOS1 (see setup.py)
         CIBW_ENVIRONMENT: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' ADIOS_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON'
-        CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
+        CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/blosc2;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
         # C++17 support in macOS 10.13+ (partial) and 10.14+ (std::visit) and 10.15+ (std::filesystem::path)
         #   https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions
-        # arm64 Python interpreters are built with 11.0
+        # arm64/aarch64 (M1/M2) requires 11.0+
         MACOSX_DEPLOYMENT_TARGET: "${{ matrix.env.MACOSX_DEPLOYMENT_TARGET }}"
         CMAKE_OSX_ARCHITECTURES: "${{ matrix.env.CMAKE_OSX_ARCHITECTURES }}"
         # Windows cross-compilation

diff --git a/.travis.yml b/.travis.yml
@@ -10,7 +10,7 @@ env:
   global:
     - OPENPMD_GIT_REF="0.15.1"
 
-    - CIBW_PROJECT_REQUIRES_PYTHON=">=3.7"
+    - CIBW_PROJECT_REQUIRES_PYTHON=">=3.8"
     # Install dependencies on Linux and OSX
     - CIBW_BEFORE_BUILD="bash -x .github/library_builders.sh"
     # for the openPMD-api build, CMake shall search for
@@ -28,17 +28,17 @@ jobs:
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp37-manylinux_aarch64 cp38-manylinux_aarch64"
+        - CIBW_BUILD="cp38-manylinux_aarch64 cp39-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp39-manylinux_aarch64 cp310-manylinux_aarch64"
+        - CIBW_BUILD="cp310-manylinux_aarch64 cp311-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp311-manylinux_aarch64 pp37-manylinux_aarch64"
+        - CIBW_BUILD="pp310-manylinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
@@ -49,24 +49,14 @@ jobs:
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp37-musllinux_aarch64 cp38-musllinux_aarch64"
+        - CIBW_BUILD="cp38-musllinux_aarch64 cp39-musllinux_aarch64"
     - services: docker
       arch: arm64
       dist: focal
       env:
-        - CIBW_BUILD="cp39-musllinux_aarch64 cp310-musllinux_aarch64"
-    - services: docker
-      arch: arm64
-      dist: focal
-      env:
-        - CIBW_BUILD="cp311-musllinux_aarch64"
+        - CIBW_BUILD="cp310-musllinux_aarch64 cp311-musllinux_aarch64"
 
     # perform a linux PPC64LE build
-    - services: docker
-      arch: ppc64le
-      dist: focal
-      env:
-        - CIBW_BUILD="cp37-manylinux_ppc64le"
     - services: docker
       arch: ppc64le
       dist: focal
@@ -87,11 +77,6 @@ jobs:
       dist: focal
       env:
         - CIBW_BUILD="cp311-manylinux_ppc64le"
-    - services: docker
-      arch: ppc64le
-      dist: focal
-      env:
-        - CIBW_BUILD="cp37-musllinux_ppc64le"
     - services: docker
       arch: ppc64le
       dist: focal
@@ -131,7 +116,7 @@ install:
   - git clone --branch ${OPENPMD_GIT_REF} --depth 1 https://github.com/openPMD/openPMD-api.git src
   - cp library_builders.sh src/.github/
   - python -m pip install --upgrade pip setuptools wheel
-  - python -m pip install cibuildwheel==2.12.1
+  - python -m pip install cibuildwheel==2.14.1
   # twine & cryptography: see
   #   https://github.com/scikit-build/cmake-python-distributions/blob/4730aeee240917303f293dffc89a8d8d5a4787c4/requirements-deploy.txt
   #   https://github.com/pyca/cryptography/issues/6086
@@ -155,7 +140,7 @@ install:
 
 script:
   - cd src
-  - python3 -m cibuildwheel --output-dir ../wheelhouse
+  - travis_wait 45 python3 -m cibuildwheel --output-dir ../wheelhouse
   - cd ..
 
 deploy:

diff --git a/library_builders.bat b/library_builders.bat
@@ -16,25 +16,22 @@ exit /b 0
 
 :build_adios2
   if exist adios2-stamp exit /b 0
-  curl -sLo adios2-2.7.1.zip ^
-    https://github.com/ornladios/ADIOS2/archive/v2.7.1.zip
-  powershell Expand-Archive adios2-2.7.1.zip -DestinationPath dep-adios2
+  curl -sLo adios2-2.9.1.zip ^
+    https://github.com/ornladios/ADIOS2/archive/v2.9.1.zip
+  powershell Expand-Archive adios2-2.9.1.zip -DestinationPath dep-adios2
 
-  :: Patch Blosc Compression w/ ADIOS 2.7.1
-  curl -sLo adios2-blosc.patch ^
-    https://patch-diff.githubusercontent.com/raw/ornladios/ADIOS2/pull/2746.patch
-  python -m patch -p 1 -d dep-adios2/ADIOS2-2.7.1 adios2-blosc.patch
-
-  cmake -S dep-adios2/ADIOS2-2.7.1 -B build-adios2 ^
+  cmake -S dep-adios2/ADIOS2-2.9.1 -B build-adios2 ^
     -DCMAKE_BUILD_TYPE=Release  ^
     -DBUILD_SHARED_LIBS=OFF     ^
     -DBUILD_TESTING=OFF         ^
     -DADIOS2_USE_MPI=OFF        ^
     -DADIOS2_BUILD_EXAMPLES=OFF ^
-    -DADIOS2_USE_Blosc=ON       ^
+    -DADIOS2_Blosc2_PREFER_SHARED=OFF ^
+    -DADIOS2_USE_Blosc2=ON      ^
     -DADIOS2_USE_BZip2=OFF      ^
     -DADIOS2_USE_Fortran=OFF    ^
     -DADIOS2_USE_HDF5=OFF       ^
+    -DADIOS2_USE_MHS=OFF        ^
     -DADIOS2_USE_PNG=OFF        ^
     -DADIOS2_USE_Profiling=OFF  ^
     -DADIOS2_USE_Python=OFF     ^
@@ -88,14 +85,47 @@ exit /b 0
   if errorlevel 1 exit 1
 exit /b 0
 
+:build_blosc2
+  if exist blosc2-stamp exit /b 0
+
+  curl -sLo blosc2-2.10.1.zip ^
+    https://github.com/Blosc/c-blosc2/archive/refs/tags/v2.10.1.zip
+  powershell Expand-Archive blosc2-2.10.1.zip -DestinationPath dep-blosc2
+
+  cmake -S dep-blosc2/c-blosc2-2.10.1 -B build-blosc2 ^
+    -DCMAKE_BUILD_TYPE=Release  ^
+    -DBUILD_SHARED=OFF          ^
+    -DBUILD_STATIC=ON           ^
+    -DBUILD_BENCHMARKS=OFF      ^
+    -DBUILD_EXAMPLES=OFF        ^
+    -DBUILD_FUZZERS=OFF         ^
+    -DBUILD_PLUGINS=OFF         ^
+    -DBUILD_TESTS=OFF
+::    -DPREFER_EXTERNAL_ZLIB=ON
+::    -DZLIB_USE_STATIC_LIBS=ON
+  if errorlevel 1 exit 1
+
+  cmake --build build-blosc2 --parallel %CPU_COUNT%
+  if errorlevel 1 exit 1
+
+  cmake --build build-blosc2 --target install --config Release
+  if errorlevel 1 exit 1
+
+  rmdir /s /q build-blosc2
+  if errorlevel 1 exit 1
+
+  break > blosc2-stamp
+  if errorlevel 1 exit 1
+exit /b 0
+
 :build_hdf5
   if exist hdf5-stamp exit /b 0
 
-  curl -sLo hdf5-1.12.2.zip ^
-    https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.zip
-  powershell Expand-Archive hdf5-1.12.2.zip -DestinationPath dep-hdf5
+  curl -sLo hdf5-1.14.1-2.zip ^
+    https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.1/src/hdf5-1.14.1-2.zip
+  powershell Expand-Archive hdf5-1.14.1-2.zip -DestinationPath dep-hdf5
 
-  cmake -S dep-hdf5/hdf5-1.12.2 -B build-hdf5 ^
+  cmake -S dep-hdf5/hdf5-1.14.1-2 -B build-hdf5 ^
     -DCMAKE_BUILD_TYPE=Release  ^
     -DCMAKE_VERBOSE_MAKEFILE=ON ^
     -DBUILD_SHARED_LIBS=OFF     ^
@@ -193,7 +223,7 @@ call :install_buildessentials
 call :build_zlib
 :: build_bzip2
 :: build_szip
-call :build_blosc
 call :build_zfp
+call :build_blosc2
 call :build_hdf5
 call :build_adios2