Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Find all crates in git repositories, matching cargo's behavior #399

Merged
merged 1 commit into from
Apr 6, 2024
Merged

Find all crates in git repositories, matching cargo's behavior #399

merged 1 commit into from
Apr 6, 2024

Conversation

sgvictorino
Copy link
Contributor

@sgvictorino sgvictorino commented Feb 29, 2024

Right now, get_git_repo_packages only finds git dependencies that are:

  • their repo's root crate (Cargo.toml or */Cargo.toml)
  • in the dependency or workspace member trees of those root crates

Otherwise, get_git_package_sources throws a KeyError.

I found this with eframe:

package.name = "uses-eframe"
dependencies.eframe.git = "https://github.com/emilk/egui"
cargo generate-lockfile # succeeds
flatpak-cargo-generator.py Cargo.lock -d # KeyError: 'eframe'
Full Debug Log

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:root:depends-on-eframe has no source
INFO:root:Loading packages from https://github.com/emilk/egui
DEBUG:root:Loading workspace member ./crates/ecolor/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui_demo_app/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui_demo_lib/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui_extras/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui_glow/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui_plot/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui-wgpu/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui-winit/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/egui/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/emath/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./crates/epaint/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/confirm_exit/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/test_inline_glow_paint/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/test_viewports/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/keyboard_events/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/user_attention/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/custom_font_style/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/custom_3d_glow/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/custom_plot_manipulation/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/screenshot/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/serial_windows/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/hello_world_simple/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/custom_font/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/multiple_viewports/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/file_dialog/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/custom_window_frame/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/hello_world_par/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/images/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/hello_world/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/puffin_profiler/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Loading workspace member ./examples/save_plot/Cargo.toml in /home/me/.cache/flatpak-cargo/https_github.com_emilk_egui
DEBUG:root:Packages in https://github.com/emilk/egui:
{
    "ecolor": "crates/ecolor",
    "egui_demo_app": "crates/egui_demo_app",
    "egui_demo_lib": "crates/egui_demo_lib",
    "egui_extras": "crates/egui_extras",
    "egui_glow": "crates/egui_glow",
    "egui_plot": "crates/egui_plot",
    "egui-wgpu": "crates/egui-wgpu",
    "egui-winit": "crates/egui-winit",
    "egui": "crates/egui",
    "emath": "crates/emath",
    "epaint": "crates/epaint",
    "confirm_exit": "examples/confirm_exit",
    "test_inline_glow_paint": "examples/test_inline_glow_paint",
    "test_viewports": "examples/test_viewports",
    "keyboard_events": "examples/keyboard_events",
    "user_attention": "examples/user_attention",
    "custom_font_style": "examples/custom_font_style",
    "custom_3d_glow": "examples/custom_3d_glow",
    "custom_plot_manipulation": "examples/custom_plot_manipulation",
    "screenshot": "examples/screenshot",
    "serial_windows": "examples/serial_windows",
    "hello_world_simple": "examples/hello_world_simple",
    "custom_font": "examples/custom_font",
    "multiple_viewports": "examples/multiple_viewports",
    "file_dialog": "examples/file_dialog",
    "custom_window_frame": "examples/custom_window_frame",
    "hello_world_par": "examples/hello_world_par",
    "images": "examples/images",
    "hello_world": "examples/hello_world",
    "puffin_profiler": "examples/puffin_profiler",
    "save_plot": "examples/save_plot"
}
INFO:root:Adding package ecolor from https://github.com/emilk/egui
INFO:root:Adding package eframe from https://github.com/emilk/egui
INFO:root:Adding package egui from https://github.com/emilk/egui
INFO:root:Adding package egui-wgpu from https://github.com/emilk/egui
INFO:root:Adding package egui-winit from https://github.com/emilk/egui
INFO:root:Adding package egui_glow from https://github.com/emilk/egui
INFO:root:Adding package emath from https://github.com/emilk/egui
INFO:root:Adding package epaint from https://github.com/emilk/egui
Traceback (most recent call last):
  File "/tmp/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 470, in 
    main()
  File "/tmp/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 463, in main
    generated_sources = asyncio.run(generate_sources(load_toml(args.cargo_lock),
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/tmp/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 417, in generate_sources
    for pkg in await asyncio.gather(*pkg_coros):
  File "/tmp/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 367, in get_package_sources
    return await get_git_package_sources(package, git_repos)
  File "/tmp/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 326, in get_git_package_sources
    git_pkg = git_repo['commits'][commit][name]
KeyError: 'eframe'

Currently, `get_git_repo_packages` only finds dependencies that are
in the dep or workspace member trees of their repo's root
`Crate.toml` or `*/Cargo.toml`.
This can cause a KeyError in `get_git_package_sources`.
Copy link
Contributor

@zecakeh zecakeh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked quickly Cargo's source code and it seems that it does recurse to find all Cargo.tomls, without a care about what the workspace says.

Workspace members are used for two things: if there is a workspace = true in one of the member definitions, it allows to fetch, and for cargo commands that can apply to a whole workspace.

This is a bit simplistic, for 2 things:

  • It adds the parent workspace to all packages, even if they are not a member of it. It should be fine though, except maybe in possible very weird cases where there are several workspaces in a tree and the package is not a member of the workspace that is the closest ancestor. My take is that this can probably be overlooked.
  • It doesn't recurse to get package dependencies, which should also be fine since Cargo.lock should list them all.

I tested this with the same Cargo.lock as #405, without any issue. I also successfully tested that this fixes #322 (because it doesn't recurse into the dependencies).

@hfiguiere
Copy link
Collaborator

Thanks for the help reviewing this.

@hfiguiere hfiguiere merged commit 3d4ac94 into flatpak:master Apr 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cargo Rust cargo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants