Unable to fetch git dependency. #645

levydsa · 2024-06-12T03:26:01Z

levydsa
Jun 12, 2024

When trying to build a package with git dependency this error happens. Any easy way to solve this?

[workspace]
members = [
  # ...
]
resolver = "2"

[workspace.dependencies]
#...
libsql = { git = "https://github.com/levydsa/libsql.git", package = "libsql", branch = "nix-patch" }
# ...

error: builder for '/nix/store/gnj648xgsqgl1niy7h8cfzjhsdgm80r8-qed-web-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   failed to clone into: /build/source/.cargo-home/git/db/libsql-32cf909f52753506
       >
       > Caused by:
       >   network failure seems to have happened
       >   if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
       >   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
       >
       > Caused by:
       >   failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
       For full logs, run 'nix log /nix/store/gnj648xgsqgl1niy7h8cfzjhsdgm80r8-qed-web-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/34ffbvvbk4j7950mnmbcjr4n4ma389h2-qed-web-0.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4l3w200bxs0bcfmjd3a7c9hq3gfiyqrb-image-root.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zmcmqlwl1i9fcyvndmsmvllr34dh76h5-docker-layer-qed-web.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9062fnmyjbydk024fpqgihln0i2lzzb8-docker-image-qed-web.tar.gz.drv' failed to build

When I set the net.git-fetch-with-cli env var and add the git package to the build. The output is similar, but with another error (maybe related to networking again?).

warning: Git tree '/home/dante/src/repos/qed' is dirty
error: builder for '/nix/store/7lsgp812cyad4bm63mrdgrf54mpiq24f-qed-web-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       >   failed to load source for dependency `libsql`
       >
       > Caused by:
       >   Unable to update https://github.com/levydsa/libsql.git?branch=nix-patch
       >
       > Caused by:
       >   failed to clone into: /build/source/.cargo-home/git/db/libsql-32cf909f52753506
       >
       > Caused by:
       >   process didn't exit successfully: `git fetch --force --update-head-ok 'https://github.com/levydsa/libsql.git' '+refs/heads/nix-patch:refs/remotes/origin/nix-patch'` (exit status: 128)
       For full logs, run 'nix log /nix/store/7lsgp812cyad4bm63mrdgrf54mpiq24f-qed-web-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/7wbivdxsvfr80s74snxxm4h06j76374l-qed-web-0.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ni89x9aa01bx62ba172ark5cj9427bvs-image-root.drv' failed to build
error: 1 dependencies of derivation '/nix/store/d25i2a7af7rpmzbw45q9hbjwsbjbkxm1-docker-layer-qed-web.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xif6czwm2rih9cjw9wigdsm9ms9zbna7-docker-image-qed-web.tar.gz.drv' failed to build

packages.qed-web = craneLib.buildPackage {
  src = fileSetForCrate ./crates/qed-web;
  cargoToml = ./crates/qed-web/Cargo.toml;
  cargoExtraArgs = "-p ${packages.qed-web.pname}";

  strictDeps = true;

  LD_LIBRARY_PATH = "${pkgs.openssl.out}/lib";
  CARGO_NET_GIT_FETCH_WITH_CLI="true";

  nativeBuildInputs = with pkgs; [
    git
    pkg-config
    openssl
  ];

  buildInputs = with pkgs; [
    openssl
  ];
};

I was doing this to try to debug another problem when trying to build with the libsql-ffi dependency. Maybe I can make another issue later? I'll try to figure it out on my own, but this issue is a bit overwhelming to me. No idea what's happening.

Answered by ipetkov

Jun 15, 2024

The problem is that build script will attempt to copy bindgen.rs to OUT_DIR multiple times. Here's what happens:

Sources are vendored in a derivation which goes into the /nix/store/...
Nix changes the permissions of the files to be readonly (i.e. r--, usually cargo keeps source files stored as rw-)
When the vendored sources are linked to the deps/real derivations they maintain the same permission bits
The build script will successfully copy bindgen.rs to OUT_DIR the first time, but the file will maintain its read-only r-- permissions
A second attempt to overwrite the file fails since std::fs::copy appears to try to update the file in place (which fails due to no write-permissions)

If yo…

View full answer

ipetkov · 2024-06-12T04:20:15Z

ipetkov
Jun 12, 2024
Maintainer

Hi @levydsa! First question that comes to mind is your Cargo.lock file up to date? Running cargo update should make sure that the git entry shows up and can be picked up during vendoring

0 replies

levydsa · 2024-06-12T17:13:19Z

levydsa
Jun 12, 2024
Author

It makes a lot of sense. I don't remenber updating the Cargo.lock before running the nix build.
Now, cargo update fails in a similar way. Maybe I have to set something up?

$ cargo update
    Updating crates.io index
    Updating git repository `https://github.com/levydsa/libsql.git`
error: failed to get `libsql` as a dependency of package `qed-web v0.1.0 (/home/dante/src/repos/qed/crates/qed-web)`

Caused by:
  failed to load source for dependency `libsql`

Caused by:
  Unable to update https://github.com/levydsa/libsql.git?branch=nix-patch

Caused by:
  failed to fetch into: /home/dante/.local/share/cargo/git/db/libsql-32cf909f52753506

Caused by:
  network failure seems to have happened
  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  SSL error: unknown error; class=Ssl (16)

$ CARGO_NET_GIT_FETCH_WITH_CLI="true" cargo update
    Updating crates.io index
    Updating git repository `https://github.com/levydsa/libsql.git`
remote: Enumerating objects: 243527, done.
remote: Counting objects: 100% (128/128), done.
remote: Compressing objects: 100% (104/104), done.
error: 1315 bytes of body are still expected5 MiB | 1004.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
error: failed to get `libsql` as a dependency of package `qed-web v0.1.0 (/home/dante/src/repos/qed/crates/qed-web)`

Caused by:
  failed to load source for dependency `libsql`

Caused by:
  Unable to update https://github.com/levydsa/libsql.git?branch=nix-patch

Caused by:
  failed to fetch into: /home/dante/.local/share/cargo/git/db/libsql-32cf909f52753506

Caused by:
  process didn't exit successfully: `git fetch --force --update-head-ok 'https://github.com/levydsa/libsql.git' '+refs/heads/nix-patch:refs/remotes/origin/nix-patch'` (exit status: 128)

cargo run worked lol. I made a typo, it's already "ammended" it in the nix-patch branch. Just to show the repos getting fetched.

$ cargo run
    Updating git repository `https://github.com/levydsa/libsql.git`
     Locking 7 packages
      Adding libsql v0.4.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql-ffi v0.3.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql-hrana v0.2.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql-rusqlite v0.31.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql-sqlite3-parser v0.12.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql-sys v0.6.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
      Adding libsql_replication v0.4.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
   Compiling libsql-ffi v0.3.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
   Compiling libsql-sqlite3-parser v0.12.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
   Compiling libsql-hrana v0.2.0 (https://github.com/levydsa/libsql.git?branch=nix-patch#10cb6996)
error: expected `;`, found keyword `let`
   --> /home/dante/.local/share/cargo/git/checkouts/libsql-32cf909f52753506/10cb699/libsql-ffi/build.rs:102:57
    |
102 |     dbg!((format!("{dir}/{bindgen_rs_path}"), out_path))
    |                                                         ^ help: add `;` here
103 |
104 |     let mut cfg = cc::Build::new();
    |     --- unexpected token

error: could not compile `libsql-ffi` (build script) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

When I fixed the semicolon, everything seems to run fine. But when trying to build...

$ nix build .#container
warning: Git tree '/home/dante/src/repos/qed' is dirty
error: builder for '/nix/store/fng144bdb79s083j49d4x6p6m93fmwvs-cargo-git.drv' failed with exit code 1;
       last 10 log lines:
       > source root is source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Updating Autotools / GNU config script to a newer upstream version: ./libsql-ffi/bundled/SQLite3MultipleCiphers/admin/build-aux/config.sub
       > Updating Autotools / GNU config script to a newer upstream version: ./libsql-sqlite3/config.sub
       > Updating Autotools / GNU config script to a newer upstream version: ./libsql-ffi/bundled/SQLite3MultipleCiphers/admin/build-aux/config.guess
       > Updating Autotools / GNU config script to a newer upstream version: ./libsql-sqlite3/config.guess
       > Running phase: installPhase
       > cp: cannot stat '/build/source/libsql-sqlite3/ext/crr/rs/sqlite-rs-embedded/sqlite3_capi/deps/sqlite3.h': No such file or directory
       > cp: cannot stat '/build/source/libsql-sqlite3/ext/crr/rs/sqlite-rs-embedded/sqlite3_capi/deps/sqlite3ext.h': No such file or directory
       For full logs, run 'nix log /nix/store/fng144bdb79s083j49d4x6p6m93fmwvs-cargo-git.drv'.
error: 1 dependencies of derivation '/nix/store/2yxgnbj59alh04bjikka6pyzkib5nmp1-linkLockedDeps.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6rpzy95kb32qq9l9qs9xhjd3vfkw6a5z-vendor-cargo-deps.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kcbfdjzvkya0l7v7k25mm52m566dka7b-qed-web-0.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4ah1mx85jj4n9namp1n4gyzpfvwddg2d-image-root.drv' failed to build
error: 1 dependencies of derivation '/nix/store/pm4ibhzrgvnvivig3yw5jz1c99va5jdw-docker-layer-qed-web.drv' failed to build
error: 1 dependencies of derivation '/nix/store/3i4v3099fv4z9lbzi6b59q9h7yjgxfls-docker-image-qed-web.tar.gz.drv' failed to build

Now I'm lost. I know that the build.rs for libsql-ffi is broken when trying to build with crane from the cargo registry. But this error is new. I don't know why there is a symlink in upstream libsql that points to nothing. I'm going insane.

0 replies

levydsa · 2024-06-12T22:16:33Z

levydsa
Jun 12, 2024
Author

Ok, now if fixed it by just copying the files that should be symlinked, just to make things work. Now I can debug the initial problem. It seems that libsql-ffi is trying to modify files that are off the nix store.

warning: Git tree '/home/dante/src/repos/qed' is dirty
error: builder for '/nix/store/w1i4hsyy5bxvjxs11n8r4kacq0bx2mrs-qed-web-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       >
       >   --- stderr
       >   [/nix/store/h23s6yzi0w9m9h0bm606fysg5wmq4bk4-vendor-cargo-deps/2207e6162cffb926680ad9d4345ae20306b4e0c2c08a080ea970acf2486365a6/libsql-ffi-0.3.0/build.rs:101:5] (format!("{dir}/{bindgen_rs_path}"), out_path) = (
       >       "/nix/store/h23s6yzi0w9m9h0bm606fysg5wmq4bk4-vendor-cargo-deps/2207e6162cffb926680ad9d4345ae20306b4e0c2c08a080ea970acf2486365a6/libsql-ffi-0.3.0/bundled/bindings/bindgen.rs",
       >       "/build/source/target/release/build/libsql-ffi-c266b42decff3ceb/out/bindgen.rs",
       >   )
       >   thread 'main' panicked at /nix/store/h23s6yzi0w9m9h0bm606fysg5wmq4bk4-vendor-cargo-deps/2207e6162cffb926680ad9d4345ae20306b4e0c2c08a080ea970acf2486365a6/libsql-ffi-0.3.0/build.rs:102:65:
       >   called `Result::unwrap()` on an `Err` value: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
       >   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/w1i4hsyy5bxvjxs11n8r4kacq0bx2mrs-qed-web-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/75ggrf973z2n75lx35x87qqmm9nknhka-qed-web-0.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xn4h04xnpij284jqdiqfi95lhc35lms3-image-root.drv' failed to build
error: 1 dependencies of derivation '/nix/store/5apj1agb28qwd6v3x635418g8iz343my-docker-layer-qed-web.drv' failed to build
error: 1 dependencies of derivation '/nix/store/5barcbggjvgw1nzkz6q86ldnzi85g3yx-docker-image-qed-web.tar.gz.drv' failed to build

https://github.com/levydsa/libsql/blob/74a8df8304a3fd99cee4373e2bce0d9955d5e43b/libsql-ffi/build.rs#L99-L103

But it seems to be trying to write to $OUT_DIR, which is correct? idk

https://github.com/levydsa/libsql/blob/74a8df8304a3fd99cee4373e2bce0d9955d5e43b/libsql-ffi/build.rs#L20-L21

0 replies

levydsa · 2024-06-14T07:08:22Z

levydsa
Jun 14, 2024
Author

@ipetkov shoud I open another issue and close this one?

0 replies

ipetkov · 2024-06-14T16:39:27Z

ipetkov
Jun 14, 2024
Maintainer

If you have a flake that reproduces the issue you can post it here

0 replies

levydsa · 2024-06-14T19:43:24Z

levydsa
Jun 14, 2024
Author

Clone it from here https://github.com/levydsa/qed.

0 replies

ipetkov · 2024-06-15T16:51:38Z

ipetkov
Jun 15, 2024
Maintainer

The problem is that build script will attempt to copy bindgen.rs to OUT_DIR multiple times. Here's what happens:

Sources are vendored in a derivation which goes into the /nix/store/...
Nix changes the permissions of the files to be readonly (i.e. r--, usually cargo keeps source files stored as rw-)
When the vendored sources are linked to the deps/real derivations they maintain the same permission bits
The build script will successfully copy bindgen.rs to OUT_DIR the first time, but the file will maintain its read-only r-- permissions
A second attempt to overwrite the file fails since std::fs::copy appears to try to update the file in place (which fails due to no write-permissions)

If you go through some trouble to make the vendored sources writable the build continues past this point. Hence my advice is to patch the build script to either not copy the file multiple times, or to attempt to delete the file before copying.

Hope this helps!

3 replies

levydsa Jun 15, 2024
Author

Thank you so much! I'll send a PR to upstream.

dpc Jun 15, 2024

rust-lang/rust#115982 might be relevant

levydsa Jun 18, 2024
Author

@ipetkov PR got merged :) tursodatabase/libsql#1478

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to fetch git dependency. #645

{{title}}

Replies: 7 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unable to fetch git dependency. #645

levydsa Jun 12, 2024

Replies: 7 comments · 3 replies

ipetkov Jun 12, 2024 Maintainer

levydsa Jun 12, 2024 Author

levydsa Jun 12, 2024 Author

levydsa Jun 14, 2024 Author

ipetkov Jun 14, 2024 Maintainer

levydsa Jun 14, 2024 Author

ipetkov Jun 15, 2024 Maintainer

levydsa Jun 15, 2024 Author

dpc Jun 15, 2024

levydsa Jun 18, 2024 Author

levydsa
Jun 12, 2024

Replies: 7 comments 3 replies

ipetkov
Jun 12, 2024
Maintainer

levydsa
Jun 12, 2024
Author

levydsa
Jun 12, 2024
Author

levydsa
Jun 14, 2024
Author

ipetkov
Jun 14, 2024
Maintainer

levydsa
Jun 14, 2024
Author

ipetkov
Jun 15, 2024
Maintainer

levydsa Jun 15, 2024
Author

levydsa Jun 18, 2024
Author