Skip to content

Update to version 3.64.0#73

Closed
tjanez wants to merge 3 commits intoflathub:masterfrom
tjanez:tjanez/fix-build
Closed

Update to version 3.64.0#73
tjanez wants to merge 3 commits intoflathub:masterfrom
tjanez:tjanez/fix-build

Conversation

@tjanez
Copy link
Copy Markdown

@tjanez tjanez commented Nov 9, 2024

No description provided.

flathubbot and others added 3 commits October 28, 2024 12:53
ffmpeg: Update ffmpeg-6.1.1.tar.xz to 7.1
libvips: Update vips-8.15.1.tar.xz to 8.16.0
Note that updating the generated sources is more complicated at the
moment until flatpak-node-generator starts supporting Yarn version 2+.

I had to use the code in pull request flatpak/flatpak-builder-tools#252
which adds support for Yarn version 2+.

The exact commit that I used is:
flatpak/flatpak-builder-tools@eb43ec2.

A newer update of this pull request also removes the need to execute
"$FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh" before running
"yarn install" so I removed that line from the build-commands.

Commands that I used to update the generated sources are:
wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock
pipx run "flatpak_node_generator @ git+https://github.com/flatpak/flatpak-builder-tools.git@refs/pull/252/head#subdirectory=node" yarn yarn.lock --output generated-sources.json
rm yarn.lock
@flathubbot
Copy link
Copy Markdown
Contributor

Started test build 160652

@flathubbot
Copy link
Copy Markdown
Contributor

Build 160652 failed

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Nov 9, 2024

Build 160652 failed

Hmm... the failures are in the yarn install step with countless RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com errors.

Excerpt from the build log:

... trimmed ...
Running: yarn install
FB: Running 'flatpak build --die-with-parent --env=FLATPAK_BUILDER_BUILDDIR=/run/build/lossless-cut --nofilesystem=host:reset --filesystem=/srv/buildbot/worker/build-x86_64-4/build/.flatpak-builder/build/lossless-cut-1 --bind-mount=/run/build/lossless-cut=/srv/buildbot/worker/build-x86_64-4/build/.flatpak-builder/build/lossless-cut-1 --build-dir=/run/build/lossless-cut --bind-mount=/run/ccache=/srv/buildbot/worker/build-x86_64-4/build/.flatpak-builder/ccache --unshare=network --env=SOURCE_DATE_EPOCH=1731173419 '--env=CFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer ' '--env=CXXFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer ' '--env=LDFLAGS=-L/app/lib -Wl,-z,relro,-z,now -Wl,--as-needed ' --env=XDG_CACHE_HOME=/run/build/lossless-cut/flatpak-node/cache --env=npm_config_nodedir=/usr/lib/sdk/node18 --env=NPM_CONFIG_LOGLEVEL=info --env=CCACHE_DIR=/run/ccache/disabled --env=PATH=/app/bin:/usr/bin:/usr/lib/sdk/node18/bin --env=LD_LIBRARY_PATH=/app/lib --env=PKG_CONFIG_PATH=/app/lib/pkgconfig:/app/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig --env=FLATPAK_BUILDER_N_JOBS=64 /srv/buildbot/worker/build-x86_64-4/build/.flatpak-builder/rofiles/rofiles-m9VQTu /bin/sh -c 'yarn install'' on host
➤ YN0065: Yarn will periodically gather anonymous telemetry: https://yarnpkg.com/advanced/telemetry
➤ YN0065: Run yarn config set --home enableTelemetry 0 to disable

➤ YN0000: · Yarn 4.4.0
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0s 379ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0060: │ typescript is listed by your project with version 5.5.4 (p6348b), which doesn't satisfy what eslint-config-mifi and other dependencies request (~5.2.0).
➤ YN0002: │ lossless-cut@workspace:. doesn't provide react-is (p6ee21), requested by evergreen-ui.
➤ YN0086: │ Some peer dependencies are incorrectly met by your project; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0086: │ Some peer dependencies are incorrectly met by dependencies; run yarn explain peer-requirements for details.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0001: │ RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com
    at ClientRequest.<anonymous> (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:147:14258)
    at Object.onceWrapper (node:events:632:26)
    at ClientRequest.emit (node:events:529:35)
    at u.emit (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:142:14855)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)

... trimmed ...

➤ YN0000: └ Completed in 5m 1s
➤ YN0000: · Failed with errors in 5m 1s
FB: host_command_exited_cb 15936 256

�]2;flatpak-builder: Cleanup lossless-cut�Error: module lossless-cut: Child process exited with code 1
FB: Unmounting read-only fs: fusermount -uz /srv/buildbot/worker/build-x86_64-4/build/.flatpak-builder/rofiles/rofiles-m9VQTu
program finished with exit code 1
elapsedTime=663.968307

@muelli
Copy link
Copy Markdown
Collaborator

muelli commented Nov 10, 2024

countless RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com errors.

yeah, we need full offline builds.

how have you built the generated-sources?

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Nov 11, 2024

countless RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com errors.

yeah, we need full offline builds.

How could we change the builds steps to do that?

how have you built the generated-sources?

I had to use the code in pull request flatpak/flatpak-builder-tools#252 which adds support for Yarn version 2+.

The exact commit that I used is: flatpak/flatpak-builder-tools@eb43ec2.

A newer update of flatpak/flatpak-builder-tools#252 also removes the need to execute $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh before running yarn install so I removed that line from the build-commands.

Commands that I used to update the generated sources are:

wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock
pipx run "flatpak_node_generator @ git+https://github.com/flatpak/flatpak-builder-tools.git@refs/pull/252/head#subdirectory=node" yarn yarn.lock --output generated-sources.json
rm yarn.lock

@muelli
Copy link
Copy Markdown
Collaborator

muelli commented Nov 11, 2024

How could we change the builds steps to do that?

The whole Javascript ecosystem is, IMHO, not very well set up for this requirement and embraces an insecure YOLO culture of just downloading and running everything from the Internet, but I disgress.

Your approach seems to be sane. But reading up now on the history I remember that I also had trouble making it work. So maybe you need to double and triple check the lock files you're using with the node generator.

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Nov 11, 2024

How could we change the builds steps to do that?

The whole Javascript ecosystem is, IMHO, not very well set up for this requirement and embraces an insecure YOLO culture of just downloading and running everything from the Internet, but I disgress.

Yeah, I'm not a fan of the JS packaging ecosystem either.

I've asked in the pull request flatpak/flatpak-builder-tools#252 (comment) if the author has any suggestions on which command to use to make the Yarn package fetching and installation work with the latest Yarn version (LosslessCut currently uses version 4.4.0).

Your approach seems to be sane. But reading up now on the history I remember that I also had trouble making it work. So maybe you need to double and triple check the lock files you're using with the node generator.

I took a second look and compared the yarn.lock in the 3.64.0 release tarball with the one at https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock:

[tadej@toronto ~]$ pushd $(mktemp -d bla-XXXXX)
~/bla-HPzuw ~
[tadej@toronto bla-HPzuw]$ wget https://github.com/mifi/lossless-cut/archive/v3.64.0.tar.gz
v3.64.0.tar.gz       100% [===========================================================================================================>]    2.11M    4.47MB/s
                          [Files: 1  Bytes: 2.11M [1.64MB/s] Redirects: 1  Todo: 0  Errors: 0                                          ]
[tadej@toronto bla-HPzuw]$ tar -xf v3.64.0.tar.gz 
[tadej@toronto bla-HPzuw]$ sha256sum lossless-cut-3.64.0/yarn.lock 
def16157a78fec434c1ff09e157b25d8b42ed00b45192df8b51e0cb32ede39ab  lossless-cut-3.64.0/yarn.lock
[tadej@toronto bla-HPzuw]$ wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock
yarn.lock            100% [===========================================================================================================>]  131.79K    1.69MB/s
                          [Files: 1  Bytes: 131.79K [237.47KB/s] Redirects: 0  Todo: 0  Errors: 0                                      ]
[tadej@toronto bla-HPzuw]$ sha256sum yarn.lock 
def16157a78fec434c1ff09e157b25d8b42ed00b45192df8b51e0cb32ede39ab  yarn.lock
[tadej@toronto bla-HPzuw]$

They are identical.

@muelli
Copy link
Copy Markdown
Collaborator

muelli commented Nov 11, 2024

hrm.
I remember something around you having to actually download and install everything first before running the node-generator.
Have you done a full install and then used the directory in the state after the installation for the node-generator?

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Nov 11, 2024

hrm. I remember something around you having to actually download and install everything first before running the node-generator. Have you done a full install and then used the directory in the state after the installation for the node-generator?

No, I didn't do a full install beforehand.

Now tried with a full install in the lossless-cut v3.64.0 checkout and it produces the same generated-sources.json file.

@catsout
Copy link
Copy Markdown

catsout commented Dec 26, 2024

Sorry for the late reply.
For details on how to use, you can refer to https://github.com/flathub/net.cozic.joplin_desktop/blob/master/net.cozic.joplin_desktop.yml.

@holubv
Copy link
Copy Markdown

holubv commented Mar 16, 2025

I tried to build it using @catsout 's build tools flatpak/flatpak-builder-tools#252 which successfully generated generated-sources.json. However, the build step that replaces urls in yarn.lock with local files failed due to a different lock file format. LosslessCut uses yarn4 which is incompatible with the flatpak-node/flatpak-yarn.js plugin.

Similar case: flathub/social.whalebird.WhalebirdDesktop#23

Maybe it is possible to generate generated-sources.json in a way to populate yarn cache directory and then use yarn offline mode without modifying yarn.lock file at all. See https://yarnpkg.com/configuration/yarnrc#enableOfflineMode

By the way, are there any drawbacks to using the pre-built linux release provided by the upstream repository? Could we skip the build steps and simply download and package LosslessCut-linux-x64.tar.bz2 ?
I successfully built it using this manifest: https://gist.github.com/holubv/ac46a48692b6df5a4e02e0426db87427

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Apr 7, 2025

@holubv, thanks for taking a deep dive into this and writing up what you've discovered!

I tried to build it using @catsout 's build tools flatpak/flatpak-builder-tools#252 which successfully generated generated-sources.json. However, the build step that replaces urls in yarn.lock with local files failed due to a different lock file format. LosslessCut uses yarn4 which is incompatible with the flatpak-node/flatpak-yarn.js plugin.

@catsout, do you plan on updating the flatpak-node/flatpak-yarn.js plugin to support Yarn4?

Similar case: flathub/social.whalebird.WhalebirdDesktop#23

Maybe it is possible to generate generated-sources.json in a way to populate yarn cache directory and then use yarn offline mode without modifying yarn.lock file at all. See https://yarnpkg.com/configuration/yarnrc#enableOfflineMode

Would this perhaps be a viable option?

By the way, are there any drawbacks to using the pre-built linux release provided by the upstream repository? Could we skip the build steps and simply download and package LosslessCut-linux-x64.tar.bz2 ? I successfully built it using this manifest: https://gist.github.com/holubv/ac46a48692b6df5a4e02e0426db87427

@muelli, would it make sense to switch to the upstream pre-built binaries?

@tjanez
Copy link
Copy Markdown
Author

tjanez commented Dec 27, 2025

I tried to build it using @catsout 's build tools flatpak/flatpak-builder-tools#252 which successfully generated generated-sources.json. However, the build step that replaces urls in yarn.lock with local files failed due to a different lock file format. LosslessCut uses yarn4 which is incompatible with the flatpak-node/flatpak-yarn.js plugin.

@catsout, do you plan on updating the flatpak-node/flatpak-yarn.js plugin to support Yarn4?

I see that @catsout has created a new pull request: flatpak/flatpak-builder-tools#494 that adds support for Yarn Berry (i.e. Yarn version 2+) to flatpak-builder-tools.
It replaces flatpak/flatpak-builder-tools#252 which we was used by this PR previously.

@catsout, do you have an example on how to use the new Yarn provider?
Or more practically, how should this PR be reworked to work with the code in flatpak/flatpak-builder-tools#494?

@catsout
Copy link
Copy Markdown

catsout commented Dec 27, 2025

@muelli
Copy link
Copy Markdown
Collaborator

muelli commented Dec 31, 2025

#87

@muelli muelli closed this Dec 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants