Fix download and extraction of NINJA_SYNTAX #295

eregon · 2025-12-05T21:44:35Z

With this Dockerfile:

FROM oraclelinux:10
RUN dnf install git
COPY . /mx
ENV PATH=/mx:$PATH
WORKDIR /mx
RUN mx version
RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*
RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
RUN mx path --download NINJA_SYNTAX
RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*

Before this commit it's clear that ~/.mx/cache/NINJA_SYNTAX*/*/* is empty and remains empty even after mx path --download NINJA_SYNTAX.
After this commit, NINJA_SYNTAX is downloaded as soon as NinjaProject.resolveDeps is called.

I found this bug while using mx in Docker and this bug in initialization causes permanent errors like:

Building com.oracle.truffle.attach_linux-amd64-glibc with Ninja...
Archiving TRUFFLE_JSON...
Exception in thread Thread-115 (executeTask):
Traceback (most recent call last):
  File "/usr/lib64/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/truffleruby-ws/mx/src/mx/_impl/mx.py", line 398, in run
    super(_DummyProcess, self).run()
  File "/usr/lib64/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/truffleruby-ws/mx/src/mx/_impl/mx.py", line 15131, in executeTask
    task.execute()
  File "/truffleruby-ws/mx/src/mx/_impl/build/tasks/build.py", line 165, in execute
    _built = self.build()
             ^^^^^^^^^^^^
  File "/truffleruby-ws/mx/src/mx/_impl/mx_native.py", line 809, in build
    self.subject.generate_manifest_for_task(self, output_dir, tmpfilename)
  File "/truffleruby-ws/mx/src/mx/_impl/mx_native.py", line 1083, in generate_manifest_for_task
    with NinjaManifestGenerator(self, output_dir, filename, toolchain=task.toolchain) as gen:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/truffleruby-ws/mx/src/mx/_impl/mx_native.py", line 838, in __init__
    import ninja_syntax
ModuleNotFoundError: No module named 'ninja_syntax'

eregon · 2025-12-05T21:45:32Z

Full output before:

STEP 1/12: FROM oraclelinux:10
STEP 2/12: RUN dnf install git
--> Using cache 4b22e68cb3d089877970914b485b5eb459767312c7c9f1adc59ad4a9511db24a
--> 4b22e68cb3d0
STEP 3/12: COPY . /mx
--> 14c1569ae1fb
STEP 4/12: ENV PATH=/mx:$PATH
--> c79e93720c40
STEP 5/12: WORKDIR /mx
--> 606599bdd904
STEP 6/12: RUN mx version
7.68.6
--> f5ef50a6d213
STEP 7/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 12
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ..
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ninja-syntax.extracted
--> 540da24da842
STEP 8/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*
total 12
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ..
drwxr-xr-x. 2 root root 4096 Dec  5 21:37 ninja_syntax-1.7.2
--> 3e06242d56eb
STEP 9/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 8
drwxr-xr-x. 2 root root 4096 Dec  5 21:37 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ..
--> c9602d64e7dd
STEP 10/12: RUN mx path --download NINJA_SYNTAX
Downloading NINJA_SYNTAX from ['https://lafo.ssw.uni-linz.ac.at/pub/graal-external-deps/ninja_syntax-1.7.2.tar.gz']
/mx/src/mx/_impl/mx.py:8387: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. 
  return ar.extractall(dst)
/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.extracted
--> cfbdb5f8e7dd
STEP 11/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 20
drwxr-xr-x. 3 root root 4096 Dec  5 21:38 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:38 ..
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ninja-syntax.extracted
-rw-r--r--. 1 root root 3129 Dec  5 21:38 ninja-syntax.tar.gz
-rw-r--r--. 1 root root  128 Dec  5 21:38 ninja-syntax.tar.gz.sha512
--> 6c30736db3dc
STEP 12/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 8
drwxr-xr-x. 2 root root 4096 Dec  5 21:37 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:37 ..
COMMIT

Full output after:

STEP 1/12: FROM oraclelinux:10
STEP 2/12: RUN dnf install git
--> Using cache 4b22e68cb3d089877970914b485b5eb459767312c7c9f1adc59ad4a9511db24a
--> 4b22e68cb3d0
STEP 3/12: COPY . /mx
--> ef55dddc2a9e
STEP 4/12: ENV PATH=/mx:$PATH
--> 4d415be4aea7
STEP 5/12: WORKDIR /mx
--> 5474df3f3435
STEP 6/12: RUN mx version
Downloading NINJA_SYNTAX from ['https://lafo.ssw.uni-linz.ac.at/pub/graal-external-deps/ninja_syntax-1.7.2.tar.gz']
/mx/src/mx/_impl/mx.py:8387: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. 
  return ar.extractall(dst)
7.68.6
--> 417ddda40c29
STEP 7/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 20
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 ..
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 ninja-syntax.extracted
-rw-r--r--. 1 root root 3129 Dec  5 21:39 ninja-syntax.tar.gz
-rw-r--r--. 1 root root  128 Dec  5 21:39 ninja-syntax.tar.gz.sha512
--> 97838a046370
STEP 8/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*
-rw-r--r--. 1 root root 3129 Dec  5 21:39 /root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.tar.gz
-rw-r--r--. 1 root root  128 Dec  5 21:39 /root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.tar.gz.sha512

/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.extracted:
total 12
drwxr-xr-x. 3 root root  4096 Dec  5 21:39 .
drwxr-xr-x. 3 root root  4096 Dec  5 21:39 ..
drwxr-xr-x. 2  501 games 4096 Jan 24  2017 ninja_syntax-1.7.2
--> 39ad737a7ece
STEP 9/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 28
drwxr-xr-x. 2  501 games 4096 Jan 24  2017 .
drwxr-xr-x. 3 root root  4096 Dec  5 21:39 ..
-rw-r--r--. 1  501 games  626 Jan 24  2017 PKG-INFO
-rw-r--r--. 1  501 games 6189 Jan 24  2017 ninja_syntax.py
-rwxr-xr-x. 1  501 games 1893 Jan 24  2017 setup.py
-rw-r--r--. 1  501 games   21 Jan 24  2017 version.py
--> 2bf74244ac3d
STEP 10/12: RUN mx path --download NINJA_SYNTAX
/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.extracted
--> f83f24e5d3dd
STEP 11/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 20
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 .
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 ..
drwxr-xr-x. 3 root root 4096 Dec  5 21:39 ninja-syntax.extracted
-rw-r--r--. 1 root root 3129 Dec  5 21:39 ninja-syntax.tar.gz
-rw-r--r--. 1 root root  128 Dec  5 21:39 ninja-syntax.tar.gz.sha512
--> ae5141aa04d3
STEP 12/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 28
drwxr-xr-x. 2  501 games 4096 Jan 24  2017 .
drwxr-xr-x. 3 root root  4096 Dec  5 21:39 ..
-rw-r--r--. 1  501 games  626 Jan 24  2017 PKG-INFO
-rw-r--r--. 1  501 games 6189 Jan 24  2017 ninja_syntax.py
-rwxr-xr-x. 1  501 games 1893 Jan 24  2017 setup.py
-rw-r--r--. 1  501 games   21 Jan 24  2017 version.py
COMMIT

* With this Dockerfile: FROM oraclelinux:10 RUN dnf install git COPY . /mx ENV PATH=/mx:$PATH WORKDIR /mx RUN mx version RUN ls -la ~/.mx/cache/NINJA_SYNTAX* RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/* RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/* RUN mx path --download NINJA_SYNTAX RUN ls -la ~/.mx/cache/NINJA_SYNTAX* RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/* * Before this commit it's clear that ~/.mx/cache/NINJA_SYNTAX*/*/* is empty and remains empty even after `mx path --download NINJA_SYNTAX`. * After this commit, NINJA_SYNTAX is downloaded as soon as NinjaProject.resolveDeps is called.

eregon · 2025-12-05T22:00:24Z

For some reason NinjaProject.resolveDeps() is called quite early, e.g. even by mx version.
An alternative if we don't want an early download would be to just copy those 4 files from ninja_syntax-1.7.2 in the repository (as an aside that archive has weird uid/gid).

rschatz · 2025-12-09T11:00:30Z

Huh... I think this is actually a corner case in the SafeDirectoryUpdater. Because if I add the verbose flag (-v) to the mx commands in your Dockerfile, it says it's extracting it.

But after extracting, it's failing to delete the already existing directory. The error is:

[Errno 18] Invalid cross-device link: '/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.extracted' -> '/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/tmpfrznwwvm/to_delete_ninja-syntax.extracted'

This is the error you get when you try to do an atomic move/rename across device boundaries.

Why move? We're moving the directory out of the way before rmtree to prevent race conditions with other processes accessing the mx cache.

Why device boundaries? Because the "old" (empty) ninja-syntax.extracted is not in the same layer we are. It was created by RUN mx version, we are in the next layer already.

mx ignores any error at that stage, just assuming there was a race condition with someone else extracting the same thing at the same time. But we can probably just rmtree at that point. If we're currently building a container layer, there shouldn't be any race conditions as long as we stay inside the layer we're currently building.

Note that this won't happen if you put the mx cache in a volume. Which is probably why we're not catching this in our internal testing. But we should still fix that bug.

gilles-duboscq · 2025-12-09T13:01:46Z

Note that this won't happen if you put the mx cache in a volume. Which is probably why we're not catching this in our internal testing. But we should still fix that bug.

FYI there is a long standing issue (GR-48582) "Transient failure to find ninja_syntax"

eregon · 2025-12-09T13:39:38Z

Thank you for the analysis, your fix sounds good.
Just to confirm, did you test it with that Dockerfile?

I wonder if we should still just vendor ninja_syntax.py and call it a day, would be much simpler, but if things work I don't mind either way.

rschatz · 2025-12-09T15:56:35Z

Yes if works now with your Dockerfile. Full output:

STEP 1/12: FROM oraclelinux:10
STEP 2/12: RUN dnf install git
--> Using cache 641d90fc7b48b8d86e9b9a3d29c9404a4c37a34d485106842ab23da1a5e43100
--> 641d90fc7b48
STEP 3/12: COPY . /mx
--> 5edbd3e03ea0
STEP 4/12: ENV PATH=/mx:$PATH
--> 2dcdc9134e7d
STEP 5/12: WORKDIR /mx
--> a012965c2e57
STEP 6/12: RUN mx version
7.68.7
--> 27581c06a1c7
STEP 7/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 12
drwxr-xr-x 3 root root 4096 Dec  9 15:53 .
drwxr-xr-x 3 root root 4096 Dec  9 15:53 ..
drwxr-xr-x 3 root root 4096 Dec  9 15:53 ninja-syntax.extracted
--> 47155bd1280b
STEP 8/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*
total 12
drwxr-xr-x 3 root root 4096 Dec  9 15:53 .
drwxr-xr-x 3 root root 4096 Dec  9 15:53 ..
drwxr-xr-x 2 root root 4096 Dec  9 15:53 ninja_syntax-1.7.2
--> 4f5cea1ce4ca
STEP 9/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 8
drwxr-xr-x 2 root root 4096 Dec  9 15:53 .
drwxr-xr-x 3 root root 4096 Dec  9 15:53 ..
--> 2536df4e2bf5
STEP 10/12: RUN mx path --download NINJA_SYNTAX
Downloading NINJA_SYNTAX from ['https://lafo.ssw.uni-linz.ac.at/pub/graal-external-deps/ninja_syntax-1.7.2.tar.gz']
/mx/src/mx/_impl/mx.py:8387: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. 
  return ar.extractall(dst)
/root/.mx/cache/NINJA_SYNTAX_8c9756de31a88be913f9bb9ff440c58a5c109721348cb59542fb1eee6f95d99f686121b2ab31622b37683632b1a9391285906e31d13f79b82b9e0980681dee4d/ninja-syntax.extracted
--> b945a5af1c83
STEP 11/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*
total 20
drwxr-xr-x 1 root root 4096 Dec  9 15:53 .
drwxr-xr-x 1 root root 4096 Dec  9 15:53 ..
drwxr-xr-x 3 root root 4096 Dec  9 15:53 ninja-syntax.extracted
-rw-r--r-- 1 root root 3129 Dec  9 15:53 ninja-syntax.tar.gz
-rw-r--r-- 1 root root  128 Dec  9 15:53 ninja-syntax.tar.gz.sha512
--> df9a4b70fc89
STEP 12/12: RUN ls -la ~/.mx/cache/NINJA_SYNTAX*/*/*
total 28
drwxr-xr-x 2  501 games 4096 Jan 24  2017 .
drwxr-xr-x 3 root root  4096 Dec  9 15:53 ..
-rw-r--r-- 1  501 games  626 Jan 24  2017 PKG-INFO
-rw-r--r-- 1  501 games 6189 Jan 24  2017 ninja_syntax.py
-rwxr-xr-x 1  501 games 1893 Jan 24  2017 setup.py
-rw-r--r-- 1  501 games   21 Jan 24  2017 version.py
COMMIT
--> eeb2c128a6de
eeb2c128a6ded509627dadf9f410ac0a5069077a0bf96c8b90c580d00044b853

eregon · 2025-12-09T18:23:26Z

Fixed by 61380fd

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Dec 5, 2025

eregon force-pushed the fix-ninja-syntax-download branch from 742fbe9 to 0b93908 Compare December 5, 2025 21:57

rschatz mentioned this pull request Dec 9, 2025

Fix SafeDirectoryUpdater when running in a layered container build. #296

Closed

rschatz self-assigned this Dec 9, 2025

eregon closed this Dec 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix download and extraction of NINJA_SYNTAX #295

Fix download and extraction of NINJA_SYNTAX #295

Uh oh!

eregon commented Dec 5, 2025 •

edited

Loading

Uh oh!

eregon commented Dec 5, 2025

Uh oh!

eregon commented Dec 5, 2025 •

edited

Loading

Uh oh!

rschatz commented Dec 9, 2025

Uh oh!

gilles-duboscq commented Dec 9, 2025

Uh oh!

eregon commented Dec 9, 2025

Uh oh!

rschatz commented Dec 9, 2025

Uh oh!

eregon commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix download and extraction of NINJA_SYNTAX #295

Fix download and extraction of NINJA_SYNTAX #295

Uh oh!

Conversation

eregon commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eregon commented Dec 5, 2025

Uh oh!

eregon commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rschatz commented Dec 9, 2025

Uh oh!

gilles-duboscq commented Dec 9, 2025

Uh oh!

eregon commented Dec 9, 2025

Uh oh!

rschatz commented Dec 9, 2025

Uh oh!

eregon commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eregon commented Dec 5, 2025 •

edited

Loading

eregon commented Dec 5, 2025 •

edited

Loading