Skip to content

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages #47864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
cd694c0
GH-47798: [CI][Packaging] Enable reproducible builds for Linux packages
raulcd Oct 20, 2025
5dbc84e
Set reproducible environment variable only on debian-trixie-amd64 tem…
raulcd Oct 20, 2025
eab5aa7
Remove file pushed unintentionally
raulcd Oct 20, 2025
5290012
Temporarily remove some linux packages
raulcd Oct 20, 2025
ab460b0
Add apt/reprotest.patch to .gitignore
raulcd Oct 20, 2025
ee198d1
Maybe using the correct environment variable name would do something :)
raulcd Oct 20, 2025
dcee108
Commit missing SOURCE_DIR
raulcd Oct 20, 2025
609a52e
Double timeout as we have to build twice now and check architecture f…
raulcd Oct 20, 2025
81ba9ca
Add Python3-venv to apache-arrow-apt-source/apt/debian-trixie/Dockerf…
raulcd Oct 20, 2025
76661b5
Add missing faketime to Dockerfile
raulcd Oct 21, 2025
78b67ea
Fix gpg homedir so we don't get differences with reproducible builds …
raulcd Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 5 additions & 20 deletions .github/workflows/package_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,32 +73,13 @@ jobs:
needs.check-labels.outputs.force == 'true' ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
timeout-minutes: 75
timeout-minutes: 240
strategy:
fail-fast: false
matrix:
id:
- almalinux-8-amd64
- almalinux-8-arm64
- almalinux-9-amd64
- almalinux-9-arm64
- almalinux-10-amd64
- almalinux-10-arm64
- amazon-linux-2023-amd64
- amazon-linux-2023-arm64
- centos-9-stream-amd64
- centos-9-stream-arm64
- centos-7-amd64
- debian-bookworm-amd64
- debian-bookworm-arm64
- debian-trixie-amd64
- debian-trixie-arm64
- debian-forky-amd64
- debian-forky-arm64
- ubuntu-jammy-amd64
- ubuntu-jammy-arm64
- ubuntu-noble-amd64
- ubuntu-noble-arm64
env:
DOCKER_VOLUME_PREFIX: ".docker/"
steps:
Expand Down Expand Up @@ -221,6 +202,10 @@ jobs:
dev/release/utils-watch-gh-workflow.sh \
${GITHUB_REF_NAME} \
release_candidate.yml
- name: Set Reproducible Build Environment
if: env.ARCHITECTURE == 'amd64'
run: |
echo "REPROTEST=yes" >> "${GITHUB_ENV}"
- name: Build
run: |
pushd dev/tasks/linux-packages
Expand Down
1 change: 1 addition & 0 deletions dev/release/rat_exclude_files.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ dev/tasks/linux-packages/apache-arrow/debian/rules
dev/tasks/linux-packages/apache-arrow/debian/shlibs.local
dev/tasks/linux-packages/apache-arrow/debian/source/format
dev/tasks/linux-packages/apache-arrow/debian/watch
dev/tasks/linux-packages/apt/reprotest.patch
dev/tasks/requirements*.txt
dev/tasks/conda-recipes/*
docs/requirements.txt
Expand Down
2 changes: 2 additions & 0 deletions dev/tasks/linux-packages/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
/*/apt/build.sh
/*/apt/build/
/*/apt/env.sh
/*/apt/reprotest.patch
/*/apt/reprotest.sh
/*/apt/repositories/
/*/apt/tmp/
/*/yum/build.sh
Expand Down
4 changes: 3 additions & 1 deletion dev/tasks/linux-packages/apache-arrow-apt-source/apt/debian-trixie/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ RUN \
debhelper \
devscripts \
fakeroot \
faketime \
gnupg \
lsb-release && \
lsb-release \
python3-venv && \
apt clean
2 changes: 2 additions & 0 deletions dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ export DH_OPTIONS
override_dh_auto_build:
gpg \
--no-default-keyring \
--homedir /tmp \
--keyring ./apache-arrow-apt-source.kbx \
--import KEYS
gpg \
--no-default-keyring \
--homedir /tmp \
--keyring ./apache-arrow-apt-source.kbx \
--armor \
--export > apache-arrow-apt-source.asc
Expand Down
2 changes: 2 additions & 0 deletions dev/tasks/linux-packages/apache-arrow/apt/debian-trixie/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ RUN \
cmake \
debhelper \
devscripts \
faketime \
gi-docgen \
git \
libboost-filesystem-dev \
Expand Down Expand Up @@ -77,6 +78,7 @@ RUN \
protobuf-compiler-grpc \
python3-dev \
python3-pip \
python3-venv \
rapidjson-dev \
tzdata \
valac \
Expand Down
20 changes: 20 additions & 0 deletions dev/tasks/linux-packages/apt/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ LANG=C

set -u

SOURCE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

run()
{
"$@"
Expand Down Expand Up @@ -80,6 +82,21 @@ case "${VERSION}" in
${PACKAGE}-${VERSION}
;;
esac

if [ "${REPROTEST:-no}" = "yes" ]; then
# Prepare the reprotest environment by copying reportest.sh and
# adding execution permissions.
run cp /host/reprotest.sh .
run chmod +x reprotest.sh
# Pin reprotest to 0.7.30 in order to apply patch that removes unconditionall call
# to setarch. See: https://salsa.debian.org/reproducible-builds/reprotest/-/issues/15
run python3 -m venv /tmp/venv
run . /tmp/venv/bin/activate
run pip install reprotest==0.7.30
# TODO: Programatically find path to apply patch instead of hardcoding.
patch -p1 -f /tmp/venv/lib/python3.13/site-packages/reprotest/build.py -i ${SOURCE_DIR}/reprotest.patch
fi

run cd ${PACKAGE}-${VERSION}/
platform="${distribution}-${code_name}"
if [ -d "/host/tmp/debian.${platform}-${architecture}" ]; then
Expand All @@ -93,6 +110,9 @@ fi
# DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS} noopt"
export DEB_BUILD_OPTIONS
df -h
if [ "${REPROTEST:-no}" = "yes" ]; then
run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host,-build_path -s .. ./reprotest.sh **.deb
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am currently investigating why if build_path is exercised, basically applying the following diff:

Suggested change
run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host,-build_path -s .. ./reprotest.sh **.deb
run reprotest --verbosity 2 --vary=-kernel,-fileordering,-domain_host -s .. ./reprotest.sh **.deb

The binaries generated (**.deb) are not reproducible, I've tried different approaches like:

diff --git a/dev/tasks/linux-packages/apache-arrow/debian/rules b/dev/tasks/linux-packages/apache-arrow/debian/rules
index 19dba393b1..17ef34fc4b 100755
--- a/dev/tasks/linux-packages/apache-arrow/debian/rules
+++ b/dev/tasks/linux-packages/apache-arrow/debian/rules
@@ -6,7 +6,7 @@
 # This has to be exported to make some magic below work.
 export DH_OPTIONS
 
-export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+export DEB_BUILD_MAINT_OPTIONS= hardening=+all reproducible=-timeless,+fixfilepath
 
 BUILD_TYPE=relwithdebinfo
 
@@ -31,6 +31,7 @@ override_dh_auto_configure:
          --builddirectory=cpp_build                            \
          --buildsystem=cmake+ninja                             \
          --                                                    \
+         $(shell dpkg-buildflags --export=configure)                   \
          -DARROW_AZURE=$${ARROW_AZURE}                         \
          -DARROW_BUILD_UTILITIES=ON                            \
          -DARROW_COMPUTE=ON                                    \

but no luck so far. More info about build_path in reproducible builds:
https://reproducible-builds.org/docs/build-path/

fi
if [ "${DEBUG:-no}" = "yes" ]; then
run debuild "${debuild_options[@]}" "${dpkg_buildpackage_options[@]}"
else
Expand Down
13 changes: 13 additions & 0 deletions dev/tasks/linux-packages/apt/reprotest.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
diff --git a/reprotest/build.py b/reprotest/build.py
index 52eff03..fa5140c 100644
--- a/reprotest/build.py
+++ b/reprotest/build.py
@@ -302,7 +302,7 @@ def home(ctx, build, vary):
def kernel(ctx, build, vary):
_ = build
if not vary:
- _ = _.append_setup_exec_raw('SETARCH_ARCH=$(uname -m)')
+ return _
else:
_ = _.append_setup_exec_raw('SETARCH_ARCH=$(for a in $(setarch --list); do setarch $a true && echo $a || true; done)')
# Perform realistic shuffling of architectures depending
18 changes: 18 additions & 0 deletions dev/tasks/linux-packages/package-task.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ def docker_run(os, architecture, console: false)
end
pass_through_env_names = [
"DEB_BUILD_OPTIONS",
"REPROTEST",
"RPM_BUILD_NCPUS",
]
pass_through_env_names.each do |name|
Expand Down Expand Up @@ -343,6 +344,15 @@ def apt_build(console: false)
ENV
end

reprotest_sh = "#{apt_dir}/reprotest.sh"
rm_rf(reprotest_sh)
File.open(reprotest_sh, "wx") do |file|
file.puts(<<-BASH)
cd #{@package}-#{@deb_upstream_version}
debuild -us -uc
BASH
end

apt_targets.each do |target|
cd(apt_dir) do
distribution, version, architecture = split_target(target)
Expand All @@ -362,13 +372,21 @@ def define_apt_task
cp(source_build_sh, build_sh)
end

source_reprotest_patch = "#{__dir__}/apt/reprotest.patch"
reprotest_patch = "#{apt_dir}/reprotest.patch"
file reprotest_patch => source_reprotest_patch do
cp(source_reprotest_patch, reprotest_patch)
end


directory repositories_dir

desc "Build deb packages"
if enable_apt?
build_dependencies = [
deb_archive_name,
build_sh,
reprotest_patch,
repositories_dir,
]
else
Expand Down