Pass auth token to all indexd calls #21

quinnwai · 2025-07-11T20:45:35Z

Description

Because we wanted to lock down indexd so that file names aren't accessible via API, we need to pass in an auth token to all gen3 API calls when registering files. Made use of addGen3AuthHeader to do this

How Has This Been Tested?

Setup

# download lfs
brew install git-lfs
git lfs install --skip-smudge 

# build git-drs from source w/ custom gen3-client dependency
git clone https://github.com/bmeg/git-drs.git
cd git-drs
git checkout bugfix/index-rbac
git submodule update --init
go build

# make the executable accessible
export PATH=$PATH:$(pwd)
cd ..

# clone repo
git clone https://source.ohsu.edu/CBDS/git-drs-test-repo.git
cd git-drs-test-repo

# initialize repo with configs, auth, git hooks
git drs init --profile calypr-dev --apiendpoint https://calypr-dev.ohsu.edu/ --cred /path/to/credentials.json

Testing Items

Query files: git drs query <did> (check https://calypr-dev.ohsu.edu/index/index when logged in to find a DID to use)
git pull -I /path/to/existing/file
add, commit, and push a new file

Types of Changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

I have added tests to cover my changes.
All new and existing tests passed.

…till 404ing

…ansfer

…of go-git bc slow

…/drs-list

* Adding a git-drs list command to list files found on DRS server * Adding paging and auth token to drs list command * Fixing issues related to PR * show 403 error when expired creds * update gen3-client module dependency * create structure to return errors within channel --------- Co-authored-by: quinnwai <[email protected]> Co-authored-by: Liam Beckman <[email protected]>

…into bugfix/indexd-rbac-2

lbeckman314

Overview ⚙️

Fantastic work with git-drs — initialization, querying, pulling, and pushing all working as expected! 🎉

Testing Steps ✔️

1. Install Git-LFS ✔️

➜ which git-lfs
/opt/homebrew/bin/git-lfs

➜ git lfs --version
git-lfs/3.5.1 (GitHub; darwin arm64; go 1.22.1)

➜ git lfs install --skip-smudge
Updated Git hooks.
Git LFS initialized.

2. Build git-drs (with custom `gen3-client` dependency) ✔️

➜ cd git-drs

➜ gh pr checkout 21
branch 'bugfix/indexd-rbac' set up to track 'origin/bugfix/indexd-rbac'.
Switched to a new branch 'bugfix/indexd-rbac' 

➜ git show
commit 18f2331: pass in caller handling no records found

➜ git submodule update --init
Submodule path 'cdis-data-client': checked out '80a54c1430d2fd30a2779f301b62426213418629'

➜ go install

➜ which git-drs
$HOME/go/bin/git-drs

➜ git drs
Git DRS provides the benefits of Git-LFS file management using DRS for seamless integration with Gen3 servers

Usage:
  git-drs [command]

Available Commands:
  download    Download file using file object ID
  help        Help about any command
  init        initialize required setup for git-drs
  list        List DRS entities from server
  precommit   pre-commit hook to create DRS objects
  query       Query DRS server by DRS ID
  transfer    register LFS files into gen3 during git push
  version     Get version

Flags:
  -h, --help   help for git-drs

Use "git-drs [command] --help" for more information about a command.

3. Clone Test Repo ✔️

➜ git clone [email protected]:CBDS/git-drs-test-repo.git
Cloning into 'git-drs-test-repo'...

➜ cd git-drs-test-repo

# initialize repo with configs, auth, git hooks
➜ git drs init --profile calypr-dev --apiendpoint https://calypr-dev.ohsu.edu/ --cred $HOME/.gen3/credentials.json.calypr-dev
.drs/lfs/objects already present in .gitignore
Git DRS initialized successfully!

4. Query Files ✔️

Query files: git drs query (check https://calypr-dev.ohsu.edu/index/index when logged in to find a DID to use)

Example DID Record (`gdc-esca`)

{
  "records": [
    {
      "acl": [],
      "authz": [
        "/programs/gdc/projects/esca"
      ],
      "did": "0067e750-00d8-5a8c-ba9d-d61eeb357667",
      "file_name": "gdcdata/0dd32ab6-7a2b-470e-80c3-e312107bebc4/0dd32ab6-7a2b-470e-80c3-e312107bebc4/TCGA-V5-A7RB-10A-01D-A88W-36.WholeGenome.RP-1657.bai",
      "size": 9681992,
      "urls": [
        "s3://gdcdata/0dd32ab6-7a2b-470e-80c3-e312107bebc4/0dd32ab6-7a2b-470e-80c3-e312107bebc4/TCGA-V5-A7RB-10A-01D-A88W-36.WholeGenome.RP-1657.bai"
      ],
    }
  ]
}

➜ git drs query 0067e750-00d8-5a8c-ba9d-d61eeb357667
{
  "id": "0067e750-00d8-5a8c-ba9d-d61eeb357667",
  "name": "gdcdata/0dd32ab6-7a2b-470e-80c3-e312107bebc4/0dd32ab6-7a2b-470e-80c3-e312107bebc4/TCGA-V5-A7RB-10A-01D-A88W-36.WholeGenome.RP-1657.bai",
  "self_uri": "drs://PREFIX:0067e750-00d8-5a8c-ba9d-d61eeb357667",
  "size": 9681992,
  "mime_type": "application/json",
  "checksums": [
    {
      "checksum": "9ff01a19a213b464effdaed4f0d07824-2",
      "type": "etag"
    }
  ],
  "access_methods": [
    {
      "type": "s3",
      "access_url": {
        "url": "s3://gdcdata/0dd32ab6-7a2b-470e-80c3-e312107bebc4/0dd32ab6-7a2b-470e-80c3-e312107bebc4/TCGA-V5-A7RB-10A-01D-A88W-36.WholeGenome.RP-1657.bai",
        "headers": null
      },
      "access_id": "s3"
    }
  ]
}

6. Pull Files ✔️

Tip

An asterisk (*) after the OID indicates a full object, a minus (-) indicates an LFS pointer. (git lfs ls-files --help)

git lfs pull -I /path/to/existing/file

➜ ls -lh
134 test.bam

➜ git lfs ls-files
d0c9352744 - test.bam    <---- Pointer

➜ git lfs pull -I test.bam

➜ git lfs ls-files
d0c9352744 * test.bam    <---- Full Object

➜ ls -lh
500M test.bam

7. Push Files ✔️

add, commit, and push a new file

➜ mkfile 1g example.bam

➜ ls -lh example.bam
1.0G example.bam

➜ git lfs track example.bam
Tracking "example.bam"

➜ git add .gitattributes

➜ git add example.bam

➜ git lfs ls-files
49bc20df15 * example.bam    <---- Full Object

➜ git commit -m "test: add example bam file"
[main 315392f] test: add example bam file
 2 files changed, 4 insertions(+)
 create mode 100644 example.bam

➜ tree .drs/lfs/objects
.drs/lfs/objects
├── 49
│   └── bc
│       └── 49bc20d...a14   <---- Newly created object
...

9 directories, 4 files

➜ git push
Uploading LFS objects: 100% (1/1), 0 B | 0 B/s, done.
To source.ohsu.edu:CBDS/git-drs-test-repo.git
   cf59014..3ffa50c  main -> main

8. Verify New Record

Tip

The DRS ID below was found from manually searching for the filename (example.bam) on Indexd (calypr-dev.ohsu.edu/index/index)

Could git-drs somehow provide it instead?

➜ git drs query 731f00e8-8ca2-520b-8729-42ccc91c7b69
{
  "id": "731f00e8-8ca2-520b-8729-42ccc91c7b69",
  "name": "example.bam",
  "self_uri": "drs://PREFIX:731f00e8-8ca2-520b-8729-42ccc91c7b69",
  "size": 1073741824,
  "mime_type": "application/json",
  "checksums": [
    {
      "checksum": "49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14",
      "type": "sha256"
    }
  ],
  "access_methods": [
    {
      "type": "s3",
      "access_url": {
        "url": "s3://cbds/731f00e8-8ca2-520b-8729-42ccc91c7b69/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14",
        "headers": null
      },
      "access_id": "s3"
    }
  ]
}

Transfer Log

.drs/transfer.log

2025/07/22 12:02:01 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 12:02:01 Received message: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:download remote:origin]
2025/07/22 12:02:01 Handling init: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:download remote:origin]
2025/07/22 12:02:01 Responding to init with empty object
2025/07/22 12:02:01 Received message: map[action:<nil> event:download oid:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 size:%!!(MISSING)s(float64=5.24288e+08)]
2025/07/22 12:02:01 Handling download event: map[action:<nil> event:download oid:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 size:%!!(MISSING)s(float64=5.24288e+08)]
2025/07/22 12:02:01 requested download of file oid d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173
2025/07/22 12:02:01 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173
2025/07/22 12:02:02 Downloading file for OID d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 from DRS object: &{Id:783e44ed-a7b3-5656-94c8-c80aea771917 Name:test.bam SelfURI:drs://PREFIX:783e44ed-a7b3-5656-94c8-c80aea771917 Size:524288000 CreatedTime: UpdatedTime: Version: MimeType:application/json Checksums:[{Checksum:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 Type:sha256}] AccessMethods:[{Type:s3 AccessURL:{URL:s3://cbds/783e44ed-a7b3-5656-94c8-c80aea771917/d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 Headers:[]} AccessID:s3 Cloud: Region: Avalible: Authorizations:<nil>}] Contents:[] Description: Aliases:[]}
2025/07/22 12:02:02 Downloading file with oid d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173, access ID: s3, file name: test.bam
2025/07/22 12:02:02 using endpoint: https://calypr-dev.ohsu.edu/ga4gh/drs/v1/objects/783e44ed-a7b3-5656-94c8-c80aea771917/access/s3

2025/07/22 12:02:02 added auth header
2025/07/22 12:02:02 got a response
2025/07/22 12:02:02 unmarshaled response into DRS AccessURL
2025/07/22 12:02:02 Got signed URL for OID d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173: &{URL:https://aced-storage.ohsu.edu/cbds/783e44ed-a7b3-5656-94c8-c80aea771917/d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173?user_id=1&username=beckmanl%!o(MISSING)hsu.edu&client_id=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cbds-user%!F(MISSING)20250722%!F(MISSING)us-east-1%!F(MISSING)s3%!F(MISSING)aws4_request&X-Amz-Date=20250722T190202Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=da4b851aee7da48df685fdf09983f220f1bd653857ae3b187fb9f8d067c32c46 Headers:[]}
2025/07/22 12:02:32 Getting DRS objects from indexd
2025/07/22 12:02:36 Getting DRS objects from indexd
2025/07/22 12:07:10 Getting DRS objects from indexd
2025/07/22 12:07:26 Getting DRS objects from indexd
2025/07/22 12:07:56 Getting DRS objects from indexd
2025/07/22 12:10:51 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 12:10:51 Received message: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:download remote:origin]
2025/07/22 12:10:51 Handling init: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:download remote:origin]
2025/07/22 12:10:51 Responding to init with empty object
2025/07/22 12:10:51 Received message: map[action:<nil> event:download oid:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 size:%!!(MISSING)s(float64=5.24288e+08)]
2025/07/22 12:10:51 Handling download event: map[action:<nil> event:download oid:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 size:%!!(MISSING)s(float64=5.24288e+08)]
2025/07/22 12:10:51 requested download of file oid d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173
2025/07/22 12:10:51 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173
2025/07/22 12:10:51 Downloading file for OID d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 from DRS object: &{Id:783e44ed-a7b3-5656-94c8-c80aea771917 Name:test.bam SelfURI:drs://PREFIX:783e44ed-a7b3-5656-94c8-c80aea771917 Size:524288000 CreatedTime: UpdatedTime: Version: MimeType:application/json Checksums:[{Checksum:d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 Type:sha256}] AccessMethods:[{Type:s3 AccessURL:{URL:s3://cbds/783e44ed-a7b3-5656-94c8-c80aea771917/d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 Headers:[]} AccessID:s3 Cloud: Region: Avalible: Authorizations:<nil>}] Contents:[] Description: Aliases:[]}
2025/07/22 12:10:51 Downloading file with oid d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173, access ID: s3, file name: test.bam
2025/07/22 12:10:51 using endpoint: https://calypr-dev.ohsu.edu/ga4gh/drs/v1/objects/783e44ed-a7b3-5656-94c8-c80aea771917/access/s3

2025/07/22 12:10:51 added auth header
2025/07/22 12:10:51 got a response
2025/07/22 12:10:51 unmarshaled response into DRS AccessURL
2025/07/22 12:10:51 Got signed URL for OID d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173: &{URL:https://aced-storage.ohsu.edu/cbds/783e44ed-a7b3-5656-94c8-c80aea771917/d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173?user_id=1&username=beckmanl%!o(MISSING)hsu.edu&client_id=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cbds-user%!F(MISSING)20250722%!F(MISSING)us-east-1%!F(MISSING)s3%!F(MISSING)aws4_request&X-Amz-Date=20250722T191052Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=785dc04c2e1eae0e14819bd4320bb9161be5f20750fb68970643ab66669a06fb Headers:[]}
2025/07/22 12:12:17 Download for OID d0c93527446e7180e8164e2bdbd12504d39a8cab18b7f0b1013cf4053fa8b173 complete
2025/07/22 12:12:18 Received message: map[event:terminate]
2025/07/22 12:12:18 terminate event received: map[event:terminate]
2025/07/22 12:12:18 ~~~~~~~~~~~~~ COMPLETED: custom transfer ~~~~~~~~~~~~~
2025/07/22 12:28:08 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 12:28:08 Received message: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 12:28:08 Handling init: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 12:28:08 Responding to init with empty object
2025/07/22 12:28:08 Received message: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 12:28:08 Handling upload event: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 12:28:08 Got UploadMessage: {Event:upload Oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Size:1073741824 Path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Action:<nil>}

2025/07/22 12:28:08 register file started for oid: 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 12:28:08 error registering indexd record: error getting indexd object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: error reading DRS object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: open .drs/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: no such file or directory
2025/07/22 12:28:08 Error registering file: error registering indexd record: error getting indexd object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: error reading DRS object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: open .drs/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: no such file or directory
2025/07/22 12:28:08 creating response message with oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 12:29:22 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 12:29:38 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 13:28:17 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 13:28:17 Received message: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 13:28:17 Handling init: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 13:28:17 Responding to init with empty object
2025/07/22 13:28:17 Received message: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 13:28:17 Handling upload event: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 13:28:17 Got UploadMessage: {Event:upload Oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Size:1073741824 Path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Action:<nil>}

2025/07/22 13:28:17 register file started for oid: 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 13:28:17 error registering indexd record: error getting indexd object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: error reading DRS object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: open .drs/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: no such file or directory
2025/07/22 13:28:17 Error registering file: error registering indexd record: error getting indexd object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: error reading DRS object for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: open .drs/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14: no such file or directory
2025/07/22 13:28:17 creating response message with oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 14:45:10 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 14:45:10 Update to DRS objects started
2025/07/22 14:45:11 git lfs ls-files output
2025/07/22 14:45:11 Repo Name: git-drs-test-repo
2025/07/22 14:45:11 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 14:45:11 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:01:05 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:01:05 Update to DRS objects started
2025/07/22 15:01:06 git lfs ls-files output
2025/07/22 15:01:06 Repo Name: git-drs-test-repo
2025/07/22 15:01:06 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 15:01:06 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:01:43 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:01:43 Update to DRS objects started
2025/07/22 15:01:43 git lfs ls-files output
2025/07/22 15:01:43 Repo Name: git-drs-test-repo
2025/07/22 15:01:43 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 15:01:43 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:09:34 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:09:34 Update to DRS objects started
2025/07/22 15:09:34 git lfs ls-files output
2025/07/22 15:09:34 Repo Name: git-drs-test-repo
2025/07/22 15:09:34 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 15:09:34 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:09:46 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:09:46 Update to DRS objects started
2025/07/22 15:09:46 git lfs ls-files output
2025/07/22 15:09:46 Repo Name: git-drs-test-repo
2025/07/22 15:09:46 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 15:09:46 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:11:06 ~~~~~~~~~~~~~ START: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:11:06 Update to DRS objects started
2025/07/22 15:11:06 git lfs ls-files output
2025/07/22 15:11:06 Repo Name: git-drs-test-repo
2025/07/22 15:11:06 Creating DRS objects for staged files: [.gitattributes example.bam]
2025/07/22 15:11:06 GET request created for indexd: https://calypr-dev.ohsu.edu/index/index?hash=sha256:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:11:06 Processing staged file: example.bam, OID: 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14, DRS ID: 731f00e8-8ca2-520b-8729-42ccc91c7b69

2025/07/22 15:11:06 Adding to DRS Objects: example.bam -> 731f00e8-8ca2-520b-8729-42ccc91c7b69
2025/07/22 15:11:06 Created .drs/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 for file example.bam
2025/07/22 15:11:06 ~~~~~~~~~~~~~ COMPLETED: pre-commit ~~~~~~~~~~~~~
2025/07/22 15:11:39 ~~~~~~~~~~~~~ START: custom transfer ~~~~~~~~~~~~~
2025/07/22 15:11:39 Received message: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 15:11:39 Handling init: map[concurrent:%!!(MISSING)s(bool=false) concurrenttransfers:%!!(MISSING)s(float64=8) event:init operation:upload remote:origin]
2025/07/22 15:11:39 Responding to init with empty object
2025/07/22 15:11:39 Received message: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 15:11:39 Handling upload event: map[action:<nil> event:upload oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 size:%!!(MISSING)s(float64=1.073741824e+09)]
2025/07/22 15:11:39 Got UploadMessage: {Event:upload Oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Size:1073741824 Path:/Users/beckmanl/code/git-drs-test-repo/.git/lfs/objects/49/bc/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Action:<nil>}

2025/07/22 15:11:39 register file started for oid: 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:11:39 retrieved IndexdObj: {"authz":["/programs/cbds/projects/git_drs_test"],"did":"731f00e8-8ca2-520b-8729-42ccc91c7b69","file_name":"example.bam","form":"object","hashes":{"sha256":"49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14"},"size":1073741824,"urls":["s3://cbds/731f00e8-8ca2-520b-8729-42ccc91c7b69/49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14"]}
2025/07/22 15:11:39 POST request created for indexd: https://calypr-dev.ohsu.edu/index/index
2025/07/22 15:11:40 POST successful: 200 OK
2025/07/22 15:11:40 GET for DRS ID successful: 731f00e8-8ca2-520b-8729-42ccc91c7b69
2025/07/22 15:13:08 creating response message with oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14
2025/07/22 15:13:08 Complete message: {Event:complete Oid:49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 Path:example.bam}
2025/07/22 15:13:08 Upload for oid 49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 complete
2025/07/22 15:13:08 Received message: map[event:terminate]
2025/07/22 15:13:08 terminate event received: map[event:terminate]
2025/07/22 15:13:08 ~~~~~~~~~~~~~ COMPLETED: custom transfer ~~~~~~~~~~~~~
2025/07/22 15:18:15 Getting DRS objects from indexd

lbeckman314

Approved with latest commit (18f2331) and testing, thanks Quinn Wai! ✅

kellrott and others added 30 commits June 3, 2025 11:43

Experimenting with integrating patterns from git-lfs

2960410

Adding DRS query test to code

6e251ce

Starting to outline the DRS/indexd client support

ab2437f

1st draft DRS query and download, make sure to setup .drsconfig

28179ff

make DownloadFile more loosely coupled

e6fb636

drafted README requirements

08425c3

typos

9493117

clarify requirements, provide git lfs vs git drs comparison

5912b71

1st draft: create hash -> drs map w/ precommit

66a370f

use uuid

5a50247

working draft for indexd writing

a537b3f

parse gen3 project info from config, drafted upload-single refactor s…

6b512a8

…till 404ing

successful push using oid (sha) as filename

46749c2

improve download to localize to .git/lfs/objects

d147178

creation date stub

50cdbdd

update gen3-client location to be within directory

bfc5713

logs

c185674

fix build warning

be629da

add cdis-data-client submodule

a1755fc

refactor precommit and transfer into cli

7b89b86

implement init to create pre-commit hook and git config for custom tr…

7b841dd

…ansfer

update branch for submodules

a49db2b

enable init to pass in the right keys

fa2249a

delete indexd record if drs object registration (ie file upload) fails

0219557

first pass refactor drs map into separate objects in to .drs directory

5a39eb6

fully deprecate drs-map usage in favor of .drs/objects

7c1231d

update only staged files // delete final drsmap usages // revert use …

95147c7

…of go-git bc slow

move .drsconfig to .drs/config

1faab96

clean up commands

183de90

remove urls from config, pull from gen3Profile

0bd25ae

kellrott and others added 5 commits July 1, 2025 16:41

Adding a git-drs list command to list files found on DRS server

e89f652

Adding paging and auth token to drs list command

7063aac

Merge remote-tracking branch 'origin/main' into feature/drs-list

c6360fe

first draft fixed with rbac

74ade96

Merge remote-tracking branch 'origin/bugfix/indexd-rbac' into feature…

d524556

…/drs-list

quinnwai requested a review from lbeckman314 July 12, 2025 00:56

kellrott and others added 2 commits July 14, 2025 15:51

Merge remote-tracking branch 'origin/main' into feature/drs-list

fd8091f

show 403 error when expired creds

a6c6122

quinnwai removed the request for review from lbeckman314 July 15, 2025 22:38

quinnwai and others added 12 commits July 15, 2025 16:44

tidy go mod

f104938

update commit to log on failure to gen3 configure during init

c9aff0f

Fixing issues related to PR

bb203d8

show 403 error when expired creds

bec366a

update gen3-client module dependency

416395c

caller handles no records found

7baafa9

caller handles no records found

8634a32

ensure indexd passes back errors in updateDrsObjects

33086e2

Merge branch 'bugfix/indexd-rbac' of https://github.com/bmeg/git-drs …

70f2797

…into bugfix/indexd-rbac-2

fix comment

fc7c33e

pass in caller handling no records found

18f2331

quinnwai requested review from lbeckman314 and matthewpeterkort and removed request for matthewpeterkort July 21, 2025 18:38

lbeckman314 requested changes Jul 22, 2025

View reviewed changes

lbeckman314 approved these changes Jul 22, 2025

View reviewed changes

quinnwai and others added 2 commits July 22, 2025 16:18

readme changes

9ed0533

Merge branch 'main' into bugfix/indexd-rbac

bff9484

quinnwai merged commit 501b169 into main Jul 22, 2025
1 check passed

quinnwai deleted the bugfix/indexd-rbac branch July 22, 2025 23:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pass auth token to all indexd calls #21

Pass auth token to all indexd calls #21

Uh oh!

quinnwai commented Jul 11, 2025 •

edited by lbeckman314

Loading

Uh oh!

lbeckman314 left a comment •

edited

Loading

Uh oh!

lbeckman314 left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pass auth token to all indexd calls #21

Pass auth token to all indexd calls #21

Uh oh!

Conversation

quinnwai commented Jul 11, 2025 • edited by lbeckman314 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Setup

Testing Items

Types of Changes

Checklist:

Uh oh!

lbeckman314 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Overview ⚙️

Testing Steps ✔️

1. Install Git-LFS ✔️

2. Build git-drs (with custom gen3-client dependency) ✔️

3. Clone Test Repo ✔️

4. Query Files ✔️

6. Pull Files ✔️

7. Push Files ✔️

8. Verify New Record

Transfer Log

Uh oh!

lbeckman314 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

quinnwai commented Jul 11, 2025 •

edited by lbeckman314

Loading

lbeckman314 left a comment •

edited

Loading

2. Build git-drs (with custom `gen3-client` dependency) ✔️

lbeckman314 left a comment •

edited

Loading