Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: tag in fetchFromGitHub support #312

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

gepbird
Copy link

@gepbird gepbird commented Dec 23, 2024

Mainly opening this to avoid duplicated work as I see more people want this feature.

I don't have much time right now, but I can probably finish it in a day.

Closes #304, NixOS/nixpkgs#367562

@gepbird gepbird marked this pull request as draft December 23, 2024 09:52
Flake lock file updates:

• Updated input 'flake-parts':
    'github:hercules-ci/flake-parts/506278e768c2a08bec68eb62932193e341f55c90?narHash=sha256-hgmguH29K2fvs9szpq2r3pz2/8cJd2LPS%2Bb4tfNFCwE%3D' (2024-11-01)
  → 'github:hercules-ci/flake-parts/205b12d8b7cd4802fbcb8e8ef6a0f1408781a4f9?narHash=sha256-4pDvzqnegAfRkPwO3wmwBhVi/Sye1mzps0zHWYnP88c%3D' (2024-12-04)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/0a14706530dcb90acecb81ce0da219d88baaae75?narHash=sha256-55e1JMAuYvHZs9EICprWgJ4RmaWwDuSjzJ5K7S7zb6w%3D' (2024-11-17)
  → 'github:NixOS/nixpkgs/7e4a1594489d41bf8e16046b28e14a0e264c9baa?narHash=sha256-YsLK4ZiGY5CZmmgzsfU76OHVUTDeZJgirKzNO%2Bet0UQ%3D' (2024-12-21)
• Updated input 'treefmt-nix':
    'github:numtide/treefmt-nix/746901bb8dba96d154b66492a29f5db0693dbfcc?narHash=sha256-vK%2Ba09qq19QNu2MlLcvN4qcRctJbqWkX7ahgPZ/%2BmaI%3D' (2024-10-30)
  → 'github:numtide/treefmt-nix/65712f5af67234dad91a5a4baee986a8b62dbf8f?narHash=sha256-MMi74%2BWckoyEWBRcg/oaGRvXC9BVVxDZNRMpL%2B72wBI%3D' (2024-12-20)
@Mic92
Copy link
Owner

Mic92 commented Dec 23, 2024

Looks good. Take your time. I won't get to it, but I will at least timely merge it.

@gepbird
Copy link
Author

gepbird commented Dec 23, 2024

The test passes but there may be some things that the tests didn't catch (eg. 3f6932b), I will try to find these later.

The fetchers still use only rev (or only version_number sometimes?), adding tag support for them is out of the scope of this PR. This leaves the code base in a bit inconsistent state where sometimes we check version.rev or version.tag when the version came from the nix eval, but for versions coming from fetchers we still only do new_version.rev.

Another weird thing I found is: this might be already fixed in nixpkgs/master, I couldn't reproduce my issue. I'll investigate soon.
Edit: Apparently after the fetchgit tag PR was merged, nix-update was not crashing, I was on a wrong base when I tested it. However if we only update the lockfile, the commit message has issues like saying the diff is from None to a version.

@gepbird
Copy link
Author

gepbird commented Dec 23, 2024

I did some refactors to make it clearer when we are talking about a "rev or a tag" for the old/not-updated package and called it old_rev_tag. I think I caught all instances that previously checked for a rev (excluding versions coming from fetchers as explained above).

I think this is ready, but more manual testing wouldn't hurt :)

@gepbird gepbird marked this pull request as ready for review December 23, 2024 17:23
@xiaoxiangmoe
Copy link

xiaoxiangmoe commented Dec 25, 2024

@gepbird Can you add both tag and rev support?

For example:

{
  buildGoModule,
  fetchFromGitHub,
  lib,
  nix-update-script,
}:
buildGoModule rec {
  pname = "lipo-go";
  version = "0.9.2";
  env = {
    GIT_VERSION = version;
    GIT_REVISION = "b7b34565565e3cde8037d1b5ee95dd2bb3579ef1";
  };
  src = fetchFromGitHub {
    owner = "konoui";
    repo = "lipo";
    tag = "v${version}";
    rev = env.GIT_REVISION;
    hash = "sha256-FW2mOsshpXCTTjijo0RFdsYX883P2cudyclRtvkCxa0=";
  };

  passthru.updateScript = nix-update-script { };

  vendorHash = "sha256-7M6CRxJd4fgYQLJDkNa3ds3f7jOp3dyloOZtwMtCBQk=";

  postPatch = ''
    # remove the test that requires access permit to /bin
    sed -i '/bin := filepath.Join/a info, err := os.Stat(bin);if err != nil || info.Mode().Perm()&0444 != 0444 { continue }' pkg/lipo/archs_test.go
  '';

  buildPhase = ''
    make build VERSION=$GIT_VERSION REVISION=$GIT_REVISION BINARY=$out/bin/lipo
  '';

  meta = {
    description = "This lipo is designed to be compatible with macOS lipo, written in golang.";
    homepage = "https://github.com/konoui/lipo";
    license = lib.licenses.mit;
    maintainers = with lib.maintainers; [ xiaoxiangmoe ];
  };
}

@gepbird
Copy link
Author

gepbird commented Dec 25, 2024

@xiaoxiangmoe can you explain that use case? I don't see a reason for using both, because if they differ, which should be fetched?

And there's an eval failure on (a 4 day old) nixpkgs:

       error: fetchFromGitHub requires one of either `rev` or `tag` to be provided (not both).                                                                                                                    

@xiaoxiangmoe
Copy link

xiaoxiangmoe commented Dec 25, 2024

Some app build will need both GIT_VERSION and GIT_REVISION for log in sentry, show version and so on. So we need get both tag and rev.

For example https://github.com/konoui/lipo

./result/bin/lipo -version                                       
0.9.2/b7b34565565e3cde8037d1b5ee95dd2bb3579ef1

@xiaoxiangmoe
Copy link

xiaoxiangmoe commented Dec 25, 2024

error: fetchFromGitHub requires one of either `rev` or `tag` to be provided (not both).    

I think we should support both for it.

Maybe someday tag will change (there are precedents for this in history) and rev is always immutable and relyable.

If tag and rev is not same, we can put a warning for users.

@xiaoxiangmoe
Copy link

xiaoxiangmoe commented Dec 25, 2024

Here is another example
https://github.com/NixOS/nixpkgs/blob/a3484e1559a2e63670aef012b0afd1217a9e011b/pkgs/by-name/af/affine/package.nix#L46

I can't use nix-update-script because I can't update version with GITHUB_SHA both. And sentry relies on the env GITHUB_SHA.

@gepbird
Copy link
Author

gepbird commented Dec 25, 2024

Some app build will need both GIT_VERSION and GIT_REVISION for log info, show version and so on. So we need get both tag and rev.

For example konoui/lipo

./result/bin/lipo -version                                       
0.9.2/b7b34565565e3cde8037d1b5ee95dd2bb3579ef1

In this package some REV (COMMIT) and TAG is being patched from the version and src.rev attributes. Or this go package seems like a better example that overrides the commit (rev) and version (tag).

error: fetchFromGitHub requires one of either `rev` or `tag` to be provided (not both).    

I think we should support both for it.

Maybe someday tag will change (there are precedents for this in history) and rev is always immutable and relyable.

If tag and rev is not same, we can put a warning for users.

I doubt it would change, any fetcher like fetchFromGitHub should only have one source, either a tag (that is just a named rev) or a rev. And validating whether the rev and the tag is the same would add extra overhead.

@gepbird
Copy link
Author

gepbird commented Dec 25, 2024

Here is another example NixOS/nixpkgs@a3484e1/pkgs/by-name/af/affine/package.nix#L46

I can't use nix-update-script because I can't update version with GITHUB_SHA both. And sentry relies on the env GITHUB_SHA.

This makes sense, but seems like a rare use case and implementing it wouldn't be easy, so it's out of the scope of this PR, sorry. After fetching the new version, we'd need to find the related hard coded commit SHA (this step would probably lead to a lot of false positives and breaking packages), then fetch the new commit SHA and replace the old one.

If you really need to update the commit SHA automatically, you can write an update script that utilises nix-update to update the version, src.hash and vendorHash, then you can probably obtain the commit SHA with GitHub's API and replace it with sed.

Feel free to open an issue on nixpkgs or start a discourse thread for help with packaging or making the update script and ping me :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

crash with fetchFromGitHub tag
3 participants