cargo / rust include_bytes support

[radupopa369]

Is your feature request related to a problem? Please describe.
Would be nice if crane could help fixing
include_bytes does not work as expected withbuildRustPackage for git dependencies

Describe the solution you'd like
Crane would be able to build rust nix packages that use include_bytes

Describe alternatives you've considered
Traditional cargo build is the only working solution at the moment.

Additional context

[johnbchron]

Most of the time this is because your non-rust file didn't make it through your nix source filter. Can you share your flake?

[radupopa369]

@johnbchron Thank you very much for your fast reply. Below are the links to our nix files.
https://github.com/movementlabsxyz/movement/blob/nix-issue/rustBuildPackage/flake.nix
https://github.com/movementlabsxyz/movement/blob/nix-issue/rustBuildPackage/nix/m1-da-light-node.nix

we also tried to use crane, see PR here
https://github.com/movementlabsxyz/movement/pull/82/files

[radupopa369]

NixOS/nixpkgs#312122

[radupopa369]

Most of the time this is because your non-rust file didn't make it through your nix source filter. Can you share your flake?

@johnbchron Thank you very much for your suggestion.
Do you know how would we be able to make include_bytes pass the xis source filter? We already tried this with no luck

[ipetkov]

@radupopa369 please take a look at the following

• https://crane.dev/source-filtering.html
• https://crane.dev/patching_dependency_sources.html

Hope that helps!

[radupopa369]

@radupopa369 please take a look at the following

• https://crane.dev/source-filtering.html
• https://crane.dev/patching_dependency_sources.html

Hope that helps!

I had a look at https://crane.dev/source-filtering.html and tried to implement something like explained there.
I am also try with no filters at all, just build the package and still does not work.

This is the nix file https://github.com/movementlabsxyz/movement/blob/l-monninger/crane/nix/m1-da-light-node.nix
And this main flake.nix https://github.com/movementlabsxyz/movement/blob/l-monninger/crane/flake.nix

error: builder for '/nix/store/grbzp5nw5r8r81819kqzdbcj2ci08gjd-cargo-package-deps-0.3.0.drv' failed with exit code 101;
       last 25 log lines:
       >     Checking crossbeam-channel v0.5.12
       >     Checking signal-hook-mio v0.2.3
       >     Checking move-prover-boogie-backend v0.1.0 (https://github.com/movementlabsxyz/aptos-core?rev=e9b42128f8ed51e90d06beec72d32797693ab66c#e9b42128)
       >     Checking aptos-vm-types v0.0.1 (https://github.com/movementlabsxyz/aptos-core?rev=e9b42128f8ed51e90d06beec72d32797693ab66c#e9b42128)
       >     Checking move-errmapgen v0.1.0 (https://github.com/movementlabsxyz/aptos-core?rev=e9b42128f8ed51e90d06beec72d32797693ab66c#e9b42128)
       >     Checking idna v0.3.0
       >    Compiling include_dir_macros v0.7.3
       >     Checking psl-types v2.0.11
       >    Compiling convert_case v0.4.0
       >    Compiling buildtime-helpers v0.3.0 (/private/tmp/nix-build-cargo-package-deps-0.3.0.drv-0/source/util/buildtime/buildtime-helpers)
       >     Checking move-prover v0.1.0 (https://github.com/movementlabsxyz/aptos-core?rev=e9b42128f8ed51e90d06beec72d32797693ab66c#e9b42128)
       > error: couldn't read /nix/store/n4nwzi7zk4lwzjb7zsw9pv3cnicqbk5a-vendor-cargo-deps/f147b819d1df9a957b2332449ad706885b2333990e683ed621e739653534eac8/move-prover-0.1.0/src/../../../../aptos-move/framework/src/aptos-natives.bpl: No such file or directory (os error 2)
       >    --> /nix/store/n4nwzi7zk4lwzjb7zsw9pv3cnicqbk5a-vendor-cargo-deps/f147b819d1df9a957b2332449ad706885b2333990e683ed621e739653534eac8/move-prover-0.1.0/src/cli.rs:774:33
       >     |
       > 774 |                   template_bytes: include_bytes!(
       >     |  _________________________________^
       > 775 | |                     "../../../../aptos-move/framework/src/aptos-natives.bpl"
       > 776 | |                 )
       >     | |_________________^
       >     |
       >     = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       >     Checking publicsuffix v2.2.3
       > error: could not compile `move-prover` (lib) due to 1 previous error
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/grbzp5nw5r8r81819kqzdbcj2ci08gjd-cargo-package-deps-0.3.0.drv'.
error: 1 dependencies of derivation '/nix/store/yvfsr6k2n3jbfrkhmh31v8svsxzzvbw3-m1-da-light-node-0.3.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/x9r9as2qvaahi79hxj42rmpph2j74j2g-m1-da-light-node-0.1.0.drv' failed to build

[ipetkov]

@radupopa369 source filtering needs to be configured when the issue is with your project's sources. In this case it appears to be an issue compiling a dependency crate, in which case you will need to either patch the problem yourself (e.g. via https://crane.dev/patching_dependency_sources.html ) or submit a fix upstream!

[andyjsbell]

@ipetkov I was hitting the same problem. Are the vendor dependencies then filtered to contain only *.rs, Cargo.toml and Cargo.lock as it looks like the bpl file here is filtered out, just a thought.

[ipetkov]

@andyjsbell we follow the same procedures to vendor git dependencies that cargo vendor performs. If the project cannot be built using a vanilla (i.e. not as a Nix derivation) invocation of cargo vendor >> .cargo/config.toml && cargo build ... then the issue either needs to be resolved upstream or it needs to be patched as I've linked above.

There are no source filters that are run when vendoring dependencies.

[radupopa369]

@ipetkov Would be interesting in discussing a bug bounty reward to help us fix this bug?
We would need some help with nix and probably setup https://www.cachix.org/ for our CI/CD pipeline
Here is my email: [email protected] . Please reach out to me if interested.
I will delete the contact in a couple of days :)

[ipetkov]

@radupopa369 no, not at this time, unfortunately