Fix gravity.sh -- Alpine / Docker specific issue

[karolkania]

Versions

• Core version is v6.0.4 (Latest: v6.0.4)
• Web version is v6.0.1 (Latest: v6.0.1)
• FTL version is v6.0.2 (Latest: v6.0.3)

Platform

• OS and version: Alpine

• Platform: Docker

Expected behavior

Update Gravity (list of blocked domains) should update the database

Actual behavior / bug

Update Gravity (list of blocked domains) does not happen

Steps to reproduce

Steps to reproduce the behavior:

Add a proper list and go to /admin/gravity; click update.

Output:

  Gravity temporary directory does not exist or is not a writeable directory, falling back to /tmp. 
  [✓] DNS resolution is available

  [i] Neutrino emissions detected...

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [✓] Pulling blocklist source list into range
  [i] Using libz compression


  *** HERE's the bug ***
  [✗] Unable to write to /etc/pihole/listsCache
      Please run pihole -g as root

  [✓] Building tree
  [i] Number of gravity domains: 0 (0 unique domains)
  [i] Number of exact denied domains: 0
  [i] Number of regex denied filters: 0
  [i] Number of exact allowed domains: 0
  [i] Number of regex allowed filters: 0
  [✓] Optimizing database
  [✓] Swapping databases
  [✓] The old database remains available
  [✓] Cleaning up stray matter

  [✓] Done.

---

Not digging too deep, I was able to figure out that this is most likely related to differences between how the shell script is interpreted by Bash - on Alpine (Ash, not Bash) this seems to trigger unexpected results.

I've raised a quick PR -- pi-hole/pi-hole@a9650ae

and this is the script output after the fix is applied:

Gravity temporary directory does not exist or is not a writeable directory, falling back to /tmp. 
  [✓] DNS resolution is available

  [i] Neutrino emissions detected...

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [✓] Pulling blocklist source list into range
  [i] Using libz compression

directory_permissions: 775
saveLocation_permissions: 660
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: No changes detected
  [✓] Parsed 127472 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

directory_permissions: 775
saveLocation_permissions: 664
  [i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 91588 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

  [✓] Building tree
  [i] Number of gravity domains: 219060 (131136 unique domains)
  [i] Number of exact denied domains: 0
  [i] Number of regex denied filters: 0
  [i] Number of exact allowed domains: 0
  [i] Number of regex allowed filters: 0
  [✓] Optimizing database
  [✓] Swapping databases
  [✓] The old database remains available
  [✓] Cleaning up stray matter

  [✓] Done.

---

This is a very Alpine specific fix, most likely won't work as expected on other distros / platforms. That said I was just trying to solve it quickly for my use-case, haven't been working on it to make the fix everything-agnostic.

---

Debug Token

• URL:

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[karolkania]

PR reference: pi-hole/pi-hole#6030

[yubiuser]

Interestingly, I tried running this interactivity in the container using only sh and it worked nicely. Also the set interpreter seems to be respected.

chris@T14Gen5:~/Downloads/piv6$ docker compose exec pihole sh
/ # pihole -g
  [✓] DNS resolution is available

  [i] Neutrino emissions detected...

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [✓] Pulling blocklist source list into range
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: No changes detected
  [✓] Parsed 127472 exact domains and 0 ABP-style domains (blocking, ignored 0 non-domain entries)

/ # apk del bash
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory
(1/2) Purging bash (5.2.37-r0)
Executing bash-5.2.37-r0.pre-deinstall
(2/2) Purging readline (8.2.13-r0)
Executing busybox-1.37.0-r12.trigger
OK: 48 MiB in 83 packages
/ # pihole -g
env: ‘bash’: No such file or directory

[mattforscale]

I can confirm the PR fixes #1747 . Thanks! I was totally stumped

[yubiuser]

I'm still struggling with this issue. What type of file system are you using to host your docker system and/or your volumes?
NFS?

[mattforscale]

I'm still struggling with this issue. What type of file system are you using to host your docker system and/or your volumes? NFS?

Nope. Just ext4 off the SD card

[karolkania]

I would say it's very armv7 image specific; @yubiuser have you tried it on such a platform?
the bottom line is -

either

[ -w path/to/file ]

condition does not work in bash there the way it works everywhere else,
or
armv7 image doesn't really execute bash -- it stays in busybox / ash so we end up in a limited shell capabilities (my guess here)

---

In theory this should not matter since we're talking about Docker, but I'll leave it here just in case -- I'm running it on an old Rpi2 (w/ raspbian on board)

CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7)
Machine model: Raspberry Pi 2 Model B Rev 1.1

seems to match #1747

[yubiuser]

I don't know if this is ARMv7 specific, I have no device to test.
But an other good hint might be pi-hole/pi-hole#6047 (comment)

[mwoolweaver]

Putting my comment from the other issue here for visibility

it seems finding an alternative to -w flag has been an issue for a few years
alpinelinux/docker-alpine#156

[mwoolweaver]

i have done some more digging on this issue and it seems to be related to

in this GitHub issue clearlinux/distribution#2361 (comment) they point to this on Stack Exchange

This is related to faccessat2 syscall has been enabled in musl in alpine starting 3.14. Due to a bug in runc it fails if your kernel do not support faccessat2

which ultimately it seems most people are pointing to these release notes
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2

The fix appears to be one of the following

1. runc v1.0.0-rc93

   1. if using Docker's repositories, this is part of containerd.io 1.4.3-2
   2. if using Docker Desktop for Windows or Mac, this is part of Docker Desktop 3.3.0

2. Docker 20.10.0 (which contains moby commit a181391) or greater, AND libseccomp 2.4.4 (which contains backported libseccomp commit 5696c89) or greater. In this case, to check if your host libseccomp is faccessat2-compatible, invoke scmp_sys_resolver faccessat2. If 439 is returned, faccessat2 is supported. If -1 is returned, faccessat2 is not supported. Note that if runc is older than v1.0.0-rc93, Docker must still be at least version 20.10.0, regardless of the result of this command.

Note also that when using nested Docker, every layer must meet one of the above requirements, since if any layer improperly denies the use of faccessat2, Alpine Linux 3.14 will not function correctly.

---

in other issues it is mentioned to pin the base image to 3.13 which is likely the least acceptable answer i found
shivpatel/bitwarden_rs_dropbox_backup#10 (comment)

[mwoolweaver]

so the answer for this seems to be provided by checking the host OS by running(or potentially in the container as well)

scmp_sys_resolver faccessat2

If 439 is returned, faccessat2 is supported. If -1 is returned, faccessat2 is not supported.

[yubiuser]

The stackexchange suggests this could be mitigates by not using bash but busybox's [. Maybe one of the affected users could try this command once in bash and once in sh

if [ ! -w /etc/pihole/listsCache` ]; then echo "not writeable"; fi

[LizenzFass78851]

I'm not sure if this is the correct way to test it, but I was able to reproduce the Alpine Bash problem as follows:
I used a Lubuntu 20.04 VM (20.04.5 ISO and updated all packages after installation).
Then I installed Docker-CE using the get.docker.com script and found that this problem doesn't occur with the latest Docker-CE version.
If you install a specific version of containerd.io (or older), the problem actually occurs.

Step-by-step to reproduce:

1. Create a VM locally (anywhere) and install Lubuntu 20.04 (link to the ISO)
2. update all packages (apt update && apt dist-upgrade -yy && apt autoremove -yy) and reboot.
3. install Docker-CE using the https://get.docker.com script.
4. run the Pihole Docker container and enter the test command mentioned above.
   • if [ ! -w /etc/pihole/listsCache ]; then echo "not writeable"; fi
     (result = no output)
5. stop and remove the Pihole container again.
6. manually download the appropriate *.deb files and install them via apt (not directly via dpkg due to dependencies).

wget https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb && \
wget https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/docker-ce_19.03.15~3-0~ubuntu-focal_amd64.deb && \
apt install ./containerd.io_1.4.3-1_amd64.deb ./docker-ce_19.03.15~3-0~ubuntu-focal_amd64.deb -y --allow-downgrades && \
rm ./containerd.io_1.4.3-1_amd64.deb ./docker-ce_19.03.15~3-0~ubuntu-focal_amd64.deb

7. run the Pihole container again and try the test command again.
   • if [ ! -w /etc/pihole/listsCache ]; then echo "not writeable"; fi
     (result = "not writeable" is visible although the specified path has write permissions)

At least for me it only occurs with a certain older containerd.io version (and older).
That's at least the way I can reproduce this Alpine Bash problem, and it's architecture-independent.

refs: containerd/containerd#6148 (comment)

[LizenzFass78851]

@yubiuser
I tried your test command.
Once in bash and once in sh.
For transparency, both within the new and the affected containerd.io version.

my test procedure:

# on host
containerd --version
docker exec -it pihole bash

# on container
bash -c 'echo Shell: $(readlink /proc/$$/exe) && \
echo Output: $(if [ ! -w /etc/pihole/listsCache ]; then echo "not writeable"; fi) && \
echo Permissions: $(ls -ldha /etc/pihole/listsCache)'
sh -c 'echo Shell: $(readlink /proc/$$/exe) && \
echo Output: $(if [ ! -w /etc/pihole/listsCache ]; then echo "not writeable"; fi) && \
echo Permissions: $(ls -ldha /etc/pihole/listsCache)'

results:

• with new containerd version (which works)

containerd containerd.io 1.7.25 bcc810d6b9066471b0b6fa75f557a15a1cbf31bb

Shell: /bin/bash
Output:
Permissions: drwxr-xr-x 2 pihole pihole 4.0K Mar 13 17:33 /etc/pihole/listsCache
Shell: /bin/busybox
Output:
Permissions: drwxr-xr-x 2 pihole pihole 4.0K Mar 13 17:33 /etc/pihole/listsCache

• with the containerd version affected by the problem (which does not work)

containerd containerd.io 1.4.3 269548fa27e0089a8b8278fc4fc781d7f65a939b

Shell: /bin/bash
Output: not writeable
Permissions: drwxr-xr-x 2 pihole pihole 4.0K Mar 13 17:36 /etc/pihole/listsCache
Shell: /bin/busybox
Output:
Permissions: drwxr-xr-x 2 pihole pihole 4.0K Mar 13 17:36 /etc/pihole/listsCache

[yubiuser]

Thanks for testing.
See the last code block, it fails on bash but works on BusyBox.
One way of fixing this would be to overwrite bash's build in test with busybox test on the docker image.

[PromoFaux]

Just for my understanding here...

Are we saying that this issue is only occuring on machines running Docker 20.10.0?

[mwoolweaver]

Worth noting, OP of this issue thread never gave what version of containerd.io or Docker they are using.

Seems to be an important question is this scenario.

[LizenzFass78851]

@PromoFaux I'm assuming that using a recent operating system, and therefore automatically the latest docker-ce and containerd.io versions. Accordingly, I haven't noticed the problem so far. If that still using an older operating system (hence the example with Ubuntu 20.04) and that may have docker.io (not docker.ce via apt) or even an old containerd.io version (from docker.ce) installed, then the problem occurs. In the instructions for reproducing the problem, I also specified an old docker-CE version (pre-20.x) when downloading the deb package (wget command), because docker-CE expects containerd.io as a dependency, and therefore it's probably a combination of the installed docker-CE components. (Predominantly an old container.d version independent of Docker-CE, but at least an old version of Docker-CE that accepts this old containerd.io version as a dependency.)

---

edit: The Docker-CE components and versions used in my example:

• containerd.io_1.4.3-1
• docker-ce_19.03.15 (also 20.10.0 and older)

these specified versions are affected, as are older versions. (I have not tested containerd.io 1.4.3-1 with Docker-CE 20.10.x)

[LizenzFass78851]

@yubiuser I've now figured out how to reproduce the issue without having to set up a VM. Anything with the latest version of Docker is sufficient.

This is a supplement to my comment #1767 (comment) where I explained how to reproduce this

I found and adapted a Dockerfile for this.

FROM docker:20.10.0 AS old
FROM docker:28.0.2

COPY --from=old /usr/local/bin/containerd* /usr/local/bin/
COPY --from=old /usr/local/bin/ctr /usr/local/bin/
COPY --from=old /usr/local/bin/runc /usr/local/bin/
COPY --from=old /usr/local/bin/docker /usr/local/bin/

RUN apk add --no-cache iptables bash

WORKDIR /workspace

ENTRYPOINT [ "dockerd" ]

The following commands are used to build, start, and attach the container.

docker build . -t docker-dind:latest

docker run -d -it --privileged --name alpine-bash-test docker-dind:latest 

docker exec -it alpine-bash-test bash

In the "docker-dind" container, the pihole can then be started and attached using the familiar docker commands, allowing tests to be performed in the pihole.
It's essentially a dockercontainer(pihole) in docker(dind = docker in docker) in docker(host) solution.

This is a way to run an old Docker affected by the Alpine Bash issue in a current Docker and then test Pihole, including the [ and test commands and its output. (Perhaps this will speed up the process of thoroughly testing the changes from my pull request #1786.)

[mwoolweaver]

So the fix is for EOL versions of docker?

[LizenzFass78851]

So the fix is for EOL versions of docker?

The fix from PR #1786 is intended to resolve this issue with Alpine Bash. This fix is ​​effective for all users. Only the affected users (who probably use an old version of docker) will notice that this issue has been resolved, as the if [ ... query in gravity.sh will then run with the correct result.

Edit: This fix has no impact on unaffected users.

[PromoFaux]

Edit: This fix has no impact on unaffected users.

So what we're saying here is that #1786 sticks a bandaid on an issue within the image for users that are on outdated versions of Docker for whatever reason?

The real fix here is to update systems to latest docker/containerd, right?

I'm uncomfortable making a change that only affects a handful of users who could otherwise "just" update their outdated systems...

ETA: That's not to say I don't appreciate the work you've put into looking at this!

[PromoFaux]

@mwoolweaver - I noted that you mentioned libseccomp in one of your comments above... has anyone confirmed that updating that also fixes this issue? Maybe we just need to add this back to the Readme file (Can't remember why it was removed, might not have been on purpose!)

Due to moby/moby#40734, you may run into issues running 2022.04 and later on host systems with an older version of libseccomp2 (Such as Debian/Raspbian buster or Ubuntu 20.04, and maybe CentOS 7).

The first recommendation is to upgrade your host OS, which will include a more up to date (and fixed) version of libseccomp.

If you absolutely cannot do this, some users #1042 (comment) success in updating libseccomp2 via backports on debian, or similar via updates on Ubuntu. You can try this workaround at your own risk (Note, you may also find that you need the latest docker.io (more details here)

[LizenzFass78851]

So what we're saying here is that #1786 sticks a bandaid on an issue within the image for users that are on outdated versions of Docker for whatever reason?

yes.

The real fix here is to update systems to latest docker/containerd, right?

yes.

With the gitee image, they solved the problem in a different way by adding a check to the image to see if an outdated containerd.io version is being used and notifying the user to update Docker.

[mwoolweaver]

I noted that you mentioned libseccomp in one of your comments above...

I was just quoting the alpine release notes

It seems like pi-hole/pi-hole#6030 was merged and shouldn't have been?

[yubiuser]

With the gitee image, they solved the problem in a different way by adding a check to the image to see if an outdated containerd.io version is being used and notifying the user to update Docker.

This is an elegant solution as well.

[PromoFaux]

@karolkania an @mattforscale note that if you are on older operating systems/versions of docker-ce, you should upgrade these.

mattforscale, I notice from #1747 that you are on Raspbian 10 - which has been EOL for quite some time now - the latest version is 12 at time of writing.

karolkania - you haven't mention which version of OS/docker you are running (unless I've missed it) but it could be that you're facing the same issue.