Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove git2 dependency #934

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

patrickjcasey
Copy link
Contributor

@patrickjcasey patrickjcasey commented Feb 18, 2025

STILL A WIP

Overview

This PR is split into 5 different commits to make it easier to review and revert, if there is a problem in the future. The commits are as follows:

  1. replace git2-based fetch implementation with gix-based implementation
  2. replace git2-based clone implementation with gix-based implementation
  3. replace git2-based checkout implementation with gix-based implementation
  4. remove git2 dependency and remaining git2 code (logging shim, rustls transport shim...)
  5. integrate gix functionality with user facing output, so progress is visible to user

Comparison of User Facing Output

TODO: record gifs comparing main and this branch when resolving git stuff

Build Timings

Note: All benchmarks were performed on an Ubuntu 24.04 machine with 8 CPUs and 32 GB of RAM

cargo build --workspace

Benchmark Command

cd /tmp && \
hyperfine --runs 5 \
    --prepare 'cd ~/workspace/hipcheck/main && cargo clean' \
    'cd ~/workspace/hipcheck/main && RUSTC_WRAPPER="" cargo build --workspace'  \
    --prepare 'cd ~/workspace/hipcheck/patrickjcasey/remove-git2-dependency && cargo clean' \
    'cd ~/workspace/hipcheck/patrickjcasey/remove-git2-dependency  &&  RUSTC_WRAPPER="" cargo build --workspace'

Results

image

cargo build --workspace --release

Benchmark Command

cd /tmp && \
hyperfine --runs 5 \
    --prepare 'cd ~/workspace/hipcheck/main && cargo clean' \
    'cd ~/workspace/hipcheck/main && RUSTC_WRAPPER="" cargo build --workspace --release'  \
    --prepare 'cd ~/workspace/hipcheck/patrickjcasey/remove-git2-dependency && cargo clean' \
    'cd ~/workspace/hipcheck/patrickjcasey/remove-git2-dependency  &&  RUSTC_WRAPPER="" cargo build --workspace --release'

Results

image

Benefits

  • removed git2 and openssl dependencies
  • standardized on gix for git operations, as it is already in use with mitre/git plugin
  • considerable build time reduction
  • TODO: compare binary artifact size of hc

Manual testing performed

Since this change impacts the way repos are retrieved and/or updated, I performed the following manual tests to verify the new implementation is working correctly:

hc run requiring a clone

rm -rf ~/.cache/hipcheck/clones/github/rustls/rustls
./target/debug/hc check --policy ./config/local.Hipcheck.kdl https://github.com/rustls/rustls
# manually checked commit in  ~/.cache/hipcheck/clones/github/rustls/rustls matches commit hash of tip of main

hc run with existing repo, fetch & pull needed

cd  ~/.cache/hipcheck/clones/github/rustls/rustls
git reset --hard HEAD~10
./target/debug/hc check --policy ./config/local.Hipcheck.kdl https://github.com/rustls/rustls
# manually checked commit in  ~/.cache/hipcheck/clones/github/rustls/rustls matches commit hash of tip of main

Update an existing local repo

TODO

Update an existing repo with multiple local branches

TODO

Test PyPI Package without version

rm -rf ~/.cache/hipcheck/clones/github/tqdm
./target/debug/hc check --policy ./config/local.Hipcheck.kdl -t pypi tqdm
# manually checked commit in  ~/.cache/hipcheck/clones/github/tqdm/tqdm matches commit hash of latest version tag on github (4.67.1)

Test PyPI package with version

rm -rf ~/.cache/hipcheck/clones/github/tqdm
./target/debug/hc check --policy ./config/local.Hipcheck.kdl -t pypi [email protected]
# manually checked commit in  ~/.cache/hipcheck/clones/github/tqdm/tqdm matches commit hash of 4.50.0 tag on github

Test NPM Package without version

rm -rf ~/.cache/hipcheck/clones/github/tqdm
./target/debug/hc check --policy ./config/local.Hipcheck.kdl -t npm express
# manually checked commit in  ~/.cache/hipcheck/clones/github/expressjs/express matches commit hash of latest version tag on github (5.0.1)

Test NPM Package without version

rm -rf ~/.cache/hipcheck/clones/github/tqdm
./target/debug/hc check --policy ./config/local.Hipcheck.kdl -t npm [email protected]
# manually checked commit in  ~/.cache/hipcheck/clones/github/expressjs/express matches commit hash of 4.21.0 tag on github

@patrickjcasey patrickjcasey linked an issue Feb 18, 2025 that may be closed by this pull request
@patrickjcasey patrickjcasey added this to the 3.12.0 milestone Feb 18, 2025
@patrickjcasey patrickjcasey force-pushed the patrickjcasey/remove-git2-dependency branch from 880c8ae to a1dcf86 Compare February 20, 2025 16:36
@patrickjcasey patrickjcasey force-pushed the patrickjcasey/remove-git2-dependency branch from c77e930 to bf0943d Compare February 21, 2025 00:30
@alilleybrinker alilleybrinker changed the title WIP WIP: Remove git2 dependency Feb 25, 2025
@alilleybrinker alilleybrinker changed the title WIP: Remove git2 dependency Remove git2 dependency Feb 25, 2025
@alilleybrinker
Copy link
Collaborator

Updated title and removed WIP from it since it's already marked as a draft and thus can't be accidentally merged, even by myself or Julian.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Status: Todo
Development

Successfully merging this pull request may close these issues.

remove git2 dependency
2 participants