Code

GH action to generate report#199

Code

Open

asmacdo wants to merge 132 commits intomaindandi/dandi-hub:mainfrom cron-data-usage-reportdandi/dandi-hub:cron-data-usage-reportCopy head branch name to clipboard

+642-63

Commits

Commits on Jan 17, 2025

• Inital commit to add GH action to generate report

  asmacdo

  committed
  30e9cbf

  Copy full SHA for 30e9cbf

• Assume Jupyterhub Provisioning Role

  asmacdo

  committed
  3bcba91

  Copy full SHA for 3bcba91

• Fixup: indent

  asmacdo

  committed
  b5cdcf3

  Copy full SHA for b5cdcf3

• Rename job

  asmacdo

  committed
  6e118cc

  Copy full SHA for 6e118cc

• Add assumed role to update-kubeconfig

  asmacdo

  committed
  5062f08

  Copy full SHA for 5062f08

• No need to add ProvisioningRole to masters

  asmacdo

  committed
  d21a3a9

  Copy full SHA for d21a3a9

• Deploy a pod to the cluster, and schedule with Karpenter

  asmacdo

  committed
  403028f

  Copy full SHA for 403028f

• Fixup: correct path to pod manifest

  asmacdo

  committed
  92b9925

  Copy full SHA for 92b9925

• Fixup again ugh, rename file

  asmacdo

  committed
  478a31f

  Copy full SHA for 478a31f

• Delete Pod even if previous step times out

  Show description for 9db914e
  asmacdo

  committed
  9db914e

  Copy full SHA for 9db914e

• Hack out initial du

  asmacdo

  committed
  8458d01

  Copy full SHA for 8458d01

• tmp comment out job deployment, test dockerhub build

  asmacdo

  committed
  7999455

  Copy full SHA for 7999455

• Fixup hyphens for image name

  asmacdo

  committed
  d2e65de

  Copy full SHA for d2e65de

• Write file to output location

  asmacdo

  committed
  5e9e7df

  Copy full SHA for 5e9e7df

• use kubectl cp to retrieve report

  asmacdo

  committed
  d33973c

  Copy full SHA for d33973c

• Combine run blocks to use vars

  asmacdo

  committed
  98fecbc

  Copy full SHA for 98fecbc

• Mount efs and pass arg to du script

  asmacdo

  committed
  40ae0e8

  Copy full SHA for 40ae0e8

• Comment out repo pushing, lets see if the report runs

  asmacdo

  committed
  4c978f7

  Copy full SHA for 4c978f7

• Restrict job to asmacdo for testing

  asmacdo

  committed
  6bd7b82

  Copy full SHA for 6bd7b82

• Sanity check. Just list the directories

  asmacdo

  committed
  73c3e80

  Copy full SHA for 73c3e80

• Job was deployed, but never assigned to node, back to sanity check

  asmacdo

  committed
  685dfb1

  Copy full SHA for 685dfb1

• change from job to pod

  asmacdo

  committed
  f6afefc

  Copy full SHA for f6afefc

• deploy pod to same namespace as pvc

  asmacdo

  committed
  6dad759

  Copy full SHA for 6dad759

• Use ns in action

  asmacdo

  committed
  3a33937

  Copy full SHA for 3a33937

• increase timeout to 60s

  Show description for 1ffb1c9
  asmacdo

  committed
  1ffb1c9

  Copy full SHA for 1ffb1c9

• fixup: image name in manifest

  asmacdo

  committed
  58e0753

  Copy full SHA for 58e0753

• increase timeout to 150

  Show description for 6767755
  asmacdo

  committed
  6767755

  Copy full SHA for 6767755

• override entrypoint so i can debug with exec

  asmacdo

  committed
  cbf951e

  Copy full SHA for cbf951e

• bound /home actually meant path was /home/home/asmacdo

  asmacdo

  committed
  59eb045

  Copy full SHA for 59eb045

• Create output dir prior to writing report

  asmacdo

  committed
  db140d5

  Copy full SHA for db140d5

• pod back to job

  asmacdo

  committed
  f90176a

  Copy full SHA for f90176a

• Fixup use the correct job api

  asmacdo

  committed
  c31ccdd

  Copy full SHA for c31ccdd

• Add namespace to pod retrieval

  asmacdo

  committed
  3ee9d9f

  Copy full SHA for 3ee9d9f

• write directly to pv to test job

  asmacdo

  committed
  d7f81ba

  Copy full SHA for d7f81ba

• fixup script fstring

  asmacdo

  committed
  0856baa

  Copy full SHA for 0856baa

• no retry on failure, we were spinning up 5 pods, lets just fail 1 time

  asmacdo

  committed
  5301b1b

  Copy full SHA for 5301b1b

• Fixup backup limit job not template

  asmacdo

  committed
  7384274

  Copy full SHA for 7384274

• Initial report

  asmacdo

  committed
  8e81e38

  Copy full SHA for 8e81e38

• disable report

  Show description for cb5db49
  asmacdo

  committed
  cb5db49

  Copy full SHA for cb5db49

• deploy ec2 instance directly

  asmacdo

  committed
  5d188a7

  Copy full SHA for 5d188a7

• Update AMI image

  asmacdo

  committed
  2f39e9c

  Copy full SHA for 2f39e9c

• update sg and subnet

  asmacdo

  committed
  3a21106

  Copy full SHA for 3a21106

• terminate even if job fails

  asmacdo

  committed
  6a54da0

  Copy full SHA for 6a54da0

• debug: print public ip

  asmacdo

  committed
  87075fb

  Copy full SHA for 87075fb

• explicitly allocate public ip for ec2 instance

  asmacdo

  committed
  48c7f35

  Copy full SHA for 48c7f35

• Add WIP scripts

  asmacdo

  committed
  743359e

  Copy full SHA for 743359e

• rm old unused

  asmacdo

  committed
  0ba12f2

  Copy full SHA for 0ba12f2

• initial commit of scripts

  asmacdo

  committed
  2893ab2

  Copy full SHA for 2893ab2

• clean up launch script

  asmacdo

  committed
  5ef8f80

  Copy full SHA for 5ef8f80

• make scripe executable

  asmacdo

  committed
  b02720e

  Copy full SHA for b02720e

• fixup cleanup script

  asmacdo

  committed
  ae98909

  Copy full SHA for ae98909

• add a name to elastic ip (for easier manual cleanup)

  asmacdo

  committed
  7e80e4a

  Copy full SHA for 7e80e4a

• Exit on fail

  asmacdo

  committed
  f2a4116

  Copy full SHA for f2a4116

• Add permission for aws ec2 wait instance-status-ok

  asmacdo

  committed
  6ffef17

  Copy full SHA for 6ffef17

• Upload scripts to instance

  asmacdo

  committed
  20cc085

  Copy full SHA for 20cc085

• explicitly return

  asmacdo

  committed
  76477df

  Copy full SHA for 76477df

• output session variables to file

  Show description for b38ded1
  asmacdo

  committed
  b38ded1

  Copy full SHA for b38ded1

• modify cleanup script to retrieve instance from temporary file

  asmacdo

  committed
  f795570

  Copy full SHA for f795570

• All ec2 persmissions granted

  Show description for f8a92b2
  asmacdo

  committed
  f8a92b2

  Copy full SHA for f8a92b2

• Add EFS mount (hardcoded)

  asmacdo

  committed
  e9726df

  Copy full SHA for e9726df

• No pager for termination

  asmacdo

  committed
  c6e92f9

  Copy full SHA for c6e92f9

• force pseudo-terminal, otherwise hangs after yum install

  asmacdo

  committed
  17d77cd

  Copy full SHA for 17d77cd

• Add doublequotes to variable usage for proper expansion

  asmacdo

  committed
  2246af5

  Copy full SHA for 2246af5

• Fixup -t goes on ssh, not scp

  asmacdo

  committed
  b49b7b5

  Copy full SHA for b49b7b5

• Mount as a single command, since we dont have access to pty

  asmacdo

  committed
  584ac4d

  Copy full SHA for 584ac4d

• add todos for manual steps

  asmacdo

  committed
  4a700e5

  Copy full SHA for 4a700e5

• Disable job for now

  Show description for 6339924
  asmacdo

  committed
  6339924

  Copy full SHA for 6339924

• Update AMI to ubuntu

  Show description for 17130ef
  asmacdo

  committed
  17130ef

  Copy full SHA for 17130ef

• Roll back to AL 2023

  Show description for cc29df5
  asmacdo

  committed
  cc29df5

  Copy full SHA for cc29df5

• drop gzip, just write json

  Show description for 295361c
  asmacdo

  committed
  295361c

  Copy full SHA for 295361c

• include target dir in relative paths

  asmacdo

  committed
  a667c04

  Copy full SHA for a667c04

• Second script will not produce user report, but directory stats json

  asmacdo

  committed
  a91beb0

  Copy full SHA for a91beb0

• inital algorithm hackout

  asmacdo

  committed
  9371982

  Copy full SHA for 9371982

• Clean up and refactor for simplicity

  asmacdo

  committed
  8cead5a

  Copy full SHA for 8cead5a

• Add basic tests

  asmacdo

  committed
  86a7c72

  Copy full SHA for 86a7c72

• test multiple directories in root

  asmacdo

  committed
  fc1cab1

  Copy full SHA for fc1cab1

• comment about [:-1]

  asmacdo

  committed
  2308aed

  Copy full SHA for 2308aed

• support abspaths

  asmacdo

  committed
  84754fe

  Copy full SHA for 84754fe

• [DATALAD RUNCMD] blacken

  Show description for a1427ac
  asmacdo

  committed
  a1427ac

  Copy full SHA for a1427ac

• test propagation with files in all dirs

  asmacdo

  committed
  16e4890

  Copy full SHA for 16e4890

• Write files to disk as they are inspected

  Show description for 528833d
  asmacdo

  committed
  528833d

  Copy full SHA for 528833d

• Comment out column headers in output

  asmacdo

  committed
  3c0e7f7

  Copy full SHA for 3c0e7f7

• Write all fields for every file

  Show description for 260c69d
  asmacdo

  committed
  260c69d

  Copy full SHA for 260c69d

• Convert to reading tsv

  asmacdo

  committed
  87dd8ca

  Copy full SHA for 87dd8ca

• Fixup: update test to match tsv-read data

  asmacdo

  committed
  e0e0a32

  Copy full SHA for e0e0a32

• update for renamed script

  asmacdo

  committed
  41aaa2a

  Copy full SHA for 41aaa2a

• install pip

  asmacdo

  committed
  25e27eb

  Copy full SHA for 25e27eb

• install parallel

  asmacdo

  committed
  204b70e

  Copy full SHA for 204b70e

• install dependencies in launch script

  asmacdo

  committed
  64d653e

  Copy full SHA for 64d653e

• Output to tmp, accept only 1 arg, target dir

  asmacdo

  committed
  6475f11

  Copy full SHA for 6475f11

• add up sizes

  asmacdo

  committed
  b67c063

  Copy full SHA for b67c063

• print useful info as index is created

  asmacdo

  committed
  3241473

  Copy full SHA for 3241473

• dont fail if output dir exists

  asmacdo

  committed
  f4eb101

  Copy full SHA for f4eb101

• Create a report dict with only relevant stats

  asmacdo

  committed
  13e0e75

  Copy full SHA for 13e0e75

• output data reports

  asmacdo

  committed
  a7e6991

  Copy full SHA for a7e6991

• Remove unused

  asmacdo

  committed
  845df00

  Copy full SHA for 845df00

Commits on Jan 23, 2025

• rm redundant

  asmacdo

  committed
  f283ad4

  Copy full SHA for f283ad4

• WIP: run against all users, construct a totals tsv

  asmacdo

  committed
  2311c92

  Copy full SHA for 2311c92

• use constants better

  asmacdo

  committed
  b65df42

  Copy full SHA for b65df42

• keep track of errors separately so we can depend on file data to be clean

  asmacdo

  committed
  474bc18

  Copy full SHA for 474bc18

Commits on Jan 27, 2025

• stats[username] is total, lets keep that and not worry about others for now

  asmacdo

  committed
  fd44a50

  Copy full SHA for fd44a50

• fixup tests

  asmacdo

  committed
  a7bafd1

  Copy full SHA for a7bafd1

• Drop (comment) full directory counting

  Show description for b712d8c
  asmacdo

  committed
  b712d8c

  Copy full SHA for b712d8c

Commits on Jan 28, 2025

• Add success message

  asmacdo

  committed
  8ef8918

  Copy full SHA for 8ef8918

• add report generation readme

  asmacdo

  committed
  5c3c858

  Copy full SHA for 5c3c858

• remove development artifacts

  asmacdo

  committed
  147d947

  Copy full SHA for 147d947

• Update .github/scripts/launch-ec2.sh

  Show description for 5f5a1ac
  

  asmacdo

  and

  yarikoptic

  authored
  5f5a1ac

  Copy full SHA for 5f5a1ac

• rm todo

  asmacdo

  committed
  31969a2

  Copy full SHA for 31969a2

• Add report generation logic back in

  asmacdo

  committed
  a42e1c4

  Copy full SHA for a42e1c4

Commits on Jan 29, 2025

• enh: WIP count nwb_files, bids_datasets, zarr_files

  asmacdo

  committed
  8d4f014

  Copy full SHA for 8d4f014

• fixup: rename generate statistics

  asmacdo

  committed
  2fd7afe

  Copy full SHA for 2fd7afe

• fixup: tests pass

  asmacdo

  committed
  1399bc4

  Copy full SHA for 1399bc4

• fixup

  asmacdo

  committed
  25ebaee

  Copy full SHA for 25ebaee

• propegate bids_datasets

  asmacdo

  committed
  6197b81

  Copy full SHA for 6197b81

• test bids_datasets propegate

  asmacdo

  committed
  ca63cdf

  Copy full SHA for ca63cdf

• WIP: wishful refactor

  asmacdo

  committed
  aae6d1b

  Copy full SHA for aae6d1b

• Refactor: DirectoryStats class

  asmacdo

  committed
  9d693f6

  Copy full SHA for 9d693f6

• add user cache counter

  asmacdo

  committed
  27981b7

  Copy full SHA for 27981b7

• fixup tests for OO refactor

  asmacdo

  committed
  0ed52d3

  Copy full SHA for 0ed52d3

• Add usercache count test

  asmacdo

  committed
  1ce1ca9

  Copy full SHA for 1ce1ca9

• Fixup clean and comment

  asmacdo

  committed
  f07eea0

  Copy full SHA for f07eea0

• Write to json + minor refactor

  asmacdo

  committed
  b076840

  Copy full SHA for b076840

• Add zarr and nwb counters

  asmacdo

  committed
  4d143bb

  Copy full SHA for 4d143bb

• blacken

  asmacdo

  committed
  e34c14f

  Copy full SHA for e34c14f

• bf: handle nested username/.cache

  asmacdo

  committed
  2391fd5

  Copy full SHA for 2391fd5

• Only write summary

  asmacdo

  committed
  c1db06b

  Copy full SHA for c1db06b

• Use increment consistently

  asmacdo

  committed
  f0efbdb

  Copy full SHA for f0efbdb

• bf: accept abspaths, but output relative paths

  asmacdo

  committed
  a5d909b

  Copy full SHA for a5d909b

• Set paths for real usage

  asmacdo

  committed
  cf1ef5c

  Copy full SHA for cf1ef5c

• bf: handle case where there are no files

  asmacdo

  committed
  1e3e6f5

  Copy full SHA for 1e3e6f5

• fixup: set paths for real usage 2

  asmacdo

  committed
  7664cac

  Copy full SHA for 7664cac

• Print user starting for usage sanity

  asmacdo

  committed
  b4cc442

  Copy full SHA for b4cc442