archive - add reproducible_tar option #8691

glennpratt · 2024-07-30T02:33:14Z

SUMMARY

archive - add reproducible_tar option to make tar archives vary less given the same input file content

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

community.general.archive new parameter reproducible_archive

ADDITIONAL INFORMATION

Used diffoscope to compare with GNU tar on an example directory

tar --create --file ../roles/k8s_node/files/lelastic2.tgz --gzip \
	--mtime='@0' \
	--sort=name \
	--numeric-owner --owner=0 --group=0 \
	--mode=go+u,go-w \
	--directory ../charts lelastic)

The most obvious remaining improvement would be to add the root directory and sort all files.

I removed such a change here because it caused test failures I haven't debugged.

Before diffoscope

diffoscope ./roles/k8s_node/files/lelastic2.tgz ./roles/k8s_node/files/lelastic.tgz
--- ./roles/k8s_node/files/lelastic2.tgz
+++ ./roles/k8s_node/files/lelastic.tgz
├── filetype from file(1)
│ @@ -1 +1 @@
│ -gzip compressed data, from Unix
│ +gzip compressed data, was "lelastic.tgz", last modified: Tue Jul 30 16:09:12 2024, max compression
│   --- lelastic2.tgz-content
├── +++ lelastic.tgz-content
│ ├── file list
│ │ @@ -1,6 +1,5 @@
│ │ -drwxr-xr-x   0        0        0        0 1970-01-01 00:00:00.000000 lelastic/
│ │ --rw-r--r--   0        0        0      349 1970-01-01 00:00:00.000000 lelastic/.helmignore
│ │ --rw-r--r--   0        0        0     1158 1970-01-01 00:00:00.000000 lelastic/Chart.yaml
│ │ -drwxr-xr-x   0        0        0        0 1970-01-01 00:00:00.000000 lelastic/templates/
│ │ --rw-r--r--   0        0        0     1855 1970-01-01 00:00:00.000000 lelastic/templates/lelastic-deployment.yaml
│ │ --rw-r--r--   0        0        0     1428 1970-01-01 00:00:00.000000 lelastic/values.yaml
│ │ +drwxr-xr-x   0 gpratt     (503) staff       (20)        0 2024-07-26 19:51:37.315485 lelastic/templates/
│ │ +-rw-r--r--   0 gpratt     (503) staff       (20)     1158 2023-09-14 23:52:21.421117 lelastic/Chart.yaml
│ │ +-rw-r--r--   0 gpratt     (503) staff       (20)      349 2023-09-14 23:52:21.421037 lelastic/.helmignore
│ │ +-rw-r--r--   0 gpratt     (503) staff       (20)     1428 2023-09-14 23:52:21.421620 lelastic/values.yaml
│ │ +-rw-r--r--   0 gpratt     (503) staff       (20)     1855 2024-07-26 19:51:37.315540 lelastic/templates/lelastic-deployment.yaml
│ ├── filetype from file(1)
│ │ @@ -1 +1 @@
│ │ -POSIX tar archive (GNU)
│ │ +POSIX tar archive

After diffoscope

❯ diffoscope ./roles/k8s_node/files/lelastic2.tgz ./roles/k8s_node/files/lelastic.tgz 
--- ./roles/k8s_node/files/lelastic2.tgz
+++ ./roles/k8s_node/files/lelastic.tgz
├── filetype from file(1)
│ @@ -1 +1 @@
│ -gzip compressed data, from Unix
│ +gzip compressed data
│   --- lelastic2.tgz-content
├── +++ lelastic.tgz-content
│ ├── file list
│ │ @@ -1,6 +1,5 @@
│ │ -drwxr-xr-x   0        0        0        0 1970-01-01 00:00:00.000000 lelastic/
│ │ --rw-r--r--   0        0        0      349 1970-01-01 00:00:00.000000 lelastic/.helmignore
│ │ --rw-r--r--   0        0        0     1158 1970-01-01 00:00:00.000000 lelastic/Chart.yaml
│ │  drwxr-xr-x   0        0        0        0 1970-01-01 00:00:00.000000 lelastic/templates/
│ │ --rw-r--r--   0        0        0     1855 1970-01-01 00:00:00.000000 lelastic/templates/lelastic-deployment.yaml
│ │ +-rw-r--r--   0        0        0     1158 1970-01-01 00:00:00.000000 lelastic/Chart.yaml
│ │ +-rw-r--r--   0        0        0      349 1970-01-01 00:00:00.000000 lelastic/.helmignore
│ │  -rw-r--r--   0        0        0     1428 1970-01-01 00:00:00.000000 lelastic/values.yaml
│ │ +-rw-r--r--   0        0        0     1855 1970-01-01 00:00:00.000000 lelastic/templates/lelastic-deployment.yaml
│ ├── filetype from file(1)
│ │ @@ -1 +1 @@
│ │ -POSIX tar archive (GNU)
│ │ +POSIX tar archive

ansibullbot · 2024-07-30T02:35:35Z

cc @bendoh
click here for bot help

felixfontein

Thanks for your contribution!

plugins/modules/archive.py

felixfontein · 2024-08-05T19:45:20Z

plugins/modules/archive.py

+                    gid=0,
+                    uname="",
+                    gname="",
+                )


Hmm, this is a very special interpretation of 'reproducible tarfile' IMO. Also this is potentially dangerous since files that are protected (only readable by specific users/groups) before archiving are suddenly publicly readable after extraction.

Maybe it would be better to make the level of reproducibility configurable? On the other hand, that would make the interface also pretty complicated.

I guess this needs to be discussed first. Maybe create a thread in https://forum.ansible.com/c/project/collection-development/27 for that?

It's not intended to be particularly special. I compare it to tar in the summary.

https://reproducible-builds.org/docs/archives/
https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html
https://github.com/drivendataorg/repro-tarfile

I'll make a thread when I have more time, I'll just leave some initial notes here.

The danger is on extract of course.

Unless you are extracting into a path that's already readable, it won't become readable:
root@dev-master-1:~# tar -xzf coreutils-0.0.27-x86_64-unknown-linux-musl.tar.gz root@dev-master-1:~# sudo -u nobody ls /root/coreutils-0.0.27-x86_64-unknown-linux-musl ls: cannot access '/root/coreutils-0.0.27-x86_64-unknown-linux-musl': Permission denied root@dev-master-1:~# sudo -u nobody ls /root/coreutils-0.0.27-x86_64-unknown-linux-musl/LICENSE ls: cannot access '/root/coreutils-0.0.27-x86_64-unknown-linux-musl/LICENSE': Permission denied

Extracting files as root (common with ansible) with odd source attributes (localhost ansible is not running as root), which was what was happening without this for me, is also potentially dangerous - UID collision leads to write privilege escalation from non-root.

ansibullbot · 2024-08-06T03:36:22Z

@glennpratt This PR contains @ mentions in at least one commit message. Those mentions can cause cascading notifications through GitHub and need to be removed. Please squash or amend your commits to remove the mentions.

click here for bot help

felixfontein · 2024-10-27T08:47:54Z

plugins/modules/archive.py

+    description:
+      - Set tar metadata and gzip headers to vary less given the same input file content.
+      - Useful for minimizing unneeded archive changes and avoiding handlers that may trigger on such changes.
+    type: bool


I think as the bare minimum this shouldn't be of type bool, but of type str with choices, so that it's possible to add other ways to make it reproducible later.

felixfontein · 2024-12-30T11:56:18Z

@glennpratt ping!

needs_info

glennpratt · 2024-12-31T00:51:13Z

Hi @felixfontein, I'm about to be away for a week at least, so I won't be getting back to this soon.

Create reproducible tar archives

51a4112

ansibullbot added WIP Work in progress feature This issue/PR relates to a feature request module module new_contributor Help guide this first time contributor plugins plugin (any type) labels Jul 30, 2024

glennpratt added 2 commits July 29, 2024 19:44

changelog

370f19b

fix bare except

5313ff2

This comment was marked as outdated.

Sign in to view

glennpratt added 3 commits July 29, 2024 20:45

fixes

0f3f99c

fix test

e827a8f

revert sort changes

b68c377

ansibullbot added tests tests unit tests/unit labels Jul 30, 2024

glennpratt marked this pull request as ready for review July 30, 2024 05:17

ansibullbot removed the WIP Work in progress label Jul 30, 2024

glennpratt changed the title ~~Create reproducible tar archives~~ archive - add reproducible_tar option Jul 30, 2024

felixfontein added check-before-release PR will be looked at again shortly before release and merged if possible. backport-9 Automatically create a backport for the stable-9 branch labels Jul 30, 2024

keep executable bit and carry mode if passed in

d1ba9f8

felixfontein reviewed Aug 5, 2024

View reviewed changes

ansibullbot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR label Aug 6, 2024

address feedback from felixfontein

3445dee

glennpratt force-pushed the reproducible-tar branch from a09d6da to 3445dee Compare August 6, 2024 03:37

python 2.7...

dea03d8

ansibullbot removed the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR label Aug 6, 2024

missed feedback

38a8271

ansibullbot added the stale_ci CI is older than 7 days, rerun before merging label Aug 16, 2024

felixfontein removed the backport-9 Automatically create a backport for the stable-9 branch label Oct 7, 2024

felixfontein reviewed Oct 27, 2024

View reviewed changes

felixfontein added the backport-10 Automatically create a backport for the stable-10 branch label Nov 4, 2024

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label Dec 30, 2024

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label Dec 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

archive - add reproducible_tar option #8691

archive - add reproducible_tar option #8691

glennpratt commented Jul 30, 2024 •

edited

Loading

ansibullbot commented Jul 30, 2024

This comment was marked as outdated.

felixfontein left a comment

felixfontein Aug 5, 2024

glennpratt Aug 6, 2024

glennpratt Aug 6, 2024 •

edited

Loading

ansibullbot commented Aug 6, 2024

felixfontein Oct 27, 2024

felixfontein commented Dec 30, 2024

glennpratt commented Dec 31, 2024 •

edited

Loading

archive - add reproducible_tar option #8691

Are you sure you want to change the base?

archive - add reproducible_tar option #8691

Conversation

glennpratt commented Jul 30, 2024 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Before diffoscope

After diffoscope

ansibullbot commented Jul 30, 2024

This comment was marked as outdated.

felixfontein left a comment

Choose a reason for hiding this comment

felixfontein Aug 5, 2024

Choose a reason for hiding this comment

glennpratt Aug 6, 2024

Choose a reason for hiding this comment

glennpratt Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

ansibullbot commented Aug 6, 2024

felixfontein Oct 27, 2024

Choose a reason for hiding this comment

felixfontein commented Dec 30, 2024

glennpratt commented Dec 31, 2024 • edited Loading

glennpratt commented Jul 30, 2024 •

edited

Loading

glennpratt Aug 6, 2024 •

edited

Loading

glennpratt commented Dec 31, 2024 •

edited

Loading