Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add s3 storage artifact route and ui integration of it #2827

Conversation

Gkrumbach07
Copy link
Member

@Gkrumbach07 Gkrumbach07 commented May 20, 2024

closes: https://issues.redhat.com/browse/RHOAIENG-7081

Description

  • created a new /api/storage GET route that takes bucket and namespace along with the path as a query param and gives back what is in that s3 bucket.
  • using that in compare runs markdown compare and in artifacts list uri link
  • added preview and peek file size warnings

Screenshot 2024-06-05 at 4 32 42 PM
Screenshot 2024-06-05 at 4 12 18 PM
Screenshot 2024-06-05 at 4 11 55 PM
Screenshot 2024-06-05 at 4 05 31 PM

How Has This Been Tested?

  • go to artifacts list page and navigate to pipelines with output artifacts
    (reach out to @Gkrumbach07 i you need a test cluster for this PR)

Test Impact

no tests add bc this is a backend route, will add util tests

Request review criteria:

Self checklist (all need to be checked):

  • The developer has manually tested the changes and verified that the changes work
  • Commits have been squashed into descriptive, self-contained units of work (e.g. 'WIP' and 'Implements feedback' style messages have been removed)
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has added tests or explained why testing cannot be added (unit or cypress tests for related changes)

If you have UI changes:

  • Included any necessary screenshots or gifs if it was a UI change.
  • Included tags to the UX team if it was a UI/UX change (find relevant UX in the SMEs section).

After the PR is posted & before it merges:

  • The developer has tested their solution on a cluster by using the image produced by the PR to main

cc @yannnz

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress This PR is in WIP state label May 20, 2024
@Gkrumbach07
Copy link
Member Author

keeping this as WIP until security review is done. I will also add tests in the meantime

@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch 2 times, most recently from 540e26b to cb9c8a0 Compare May 22, 2024 17:29
@Gkrumbach07
Copy link
Member Author

/retest

Copy link

codecov bot commented May 22, 2024

Codecov Report

Attention: Patch coverage is 65.21739% with 72 lines in your changes missing coverage. Please review.

Project coverage is 78.75%. Comparing base (f7c9979) to head (c5ecff0).
Report is 6 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2827      +/-   ##
==========================================
- Coverage   78.83%   78.75%   -0.09%     
==========================================
  Files        1113     1116       +3     
  Lines       23586    23717     +131     
  Branches     5940     5972      +32     
==========================================
+ Hits        18595    18679      +84     
- Misses       4991     5038      +47     
Files Coverage Δ
frontend/src/concepts/areas/const.ts 100.00% <ø> (ø)
frontend/src/concepts/areas/types.ts 100.00% <100.00%> (ø)
...pelines/apiHooks/mlmd/useGetEventsByExecutionId.ts 100.00% <100.00%> (+6.25%) ⬆️
.../src/concepts/pipelines/content/artifacts/utils.ts 100.00% <100.00%> (ø)
...eRuns/metricsSection/PipelineRunArtifactSelect.tsx 2.77% <ø> (ø)
.../pipelinesDetails/pipeline/PipelineTaskDetails.tsx 76.19% <ø> (+2.27%) ⬆️
...ipelinesDetails/pipelineRun/PipelineRunDetails.tsx 81.25% <100.00%> (-0.29%) ⬇️
...Details/pipelineRun/SelectedNodeInputOutputTab.tsx 93.33% <ø> (-0.22%) ⬇️
...ails/pipelineRun/artifacts/ArtifactNodeDetails.tsx 70.00% <100.00%> (ø)
...ipelineRun/artifacts/ArtifactNodeDrawerContent.tsx 70.58% <ø> (ø)
... and 15 more

... and 11 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7c9979...c5ecff0. Read the comment docs.

@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from cb9c8a0 to 55e65d4 Compare May 23, 2024 12:28
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from 55e65d4 to 57e0c6d Compare June 3, 2024 23:28
@Gkrumbach07 Gkrumbach07 changed the title WIP: Add s3 storage artifact route and ui integration of it Add s3 storage artifact route and ui integration of it Jun 3, 2024
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress This PR is in WIP state label Jun 3, 2024
@Gkrumbach07
Copy link
Member Author

@andrewballantyne I took this out of WIP, the safety check for the storage status is used, and the feature flag blocks the endpoint

@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from 57e0c6d to fac6d6c Compare June 4, 2024 12:31
Copy link
Member

@andrewballantyne andrewballantyne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you would like to talk about permission flows more, let me know

backend/src/routes/api/storage/storageUtils.ts Outdated Show resolved Hide resolved
backend/src/routes/api/storage/storageUtils.ts Outdated Show resolved Hide resolved
backend/src/routes/api/storage/index.ts Outdated Show resolved Hide resolved
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch 2 times, most recently from 8fd6e8f to 04a64fd Compare June 4, 2024 15:33
@andrewballantyne andrewballantyne removed the request for review from uidoyen June 4, 2024 19:32
@Gkrumbach07 Gkrumbach07 changed the title Add s3 storage artifact route and ui integration of it [WIP] Add s3 storage artifact route and ui integration of it Jun 4, 2024
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress This PR is in WIP state label Jun 4, 2024
@Gkrumbach07 Gkrumbach07 changed the title [WIP] Add s3 storage artifact route and ui integration of it Add s3 storage artifact route and ui integration of it Jun 4, 2024
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress This PR is in WIP state label Jun 4, 2024
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch 2 times, most recently from 09d0651 to c598110 Compare June 6, 2024 16:07
@openshift-merge-robot openshift-merge-robot added the needs-rebase PR needs to be rebased label Jun 6, 2024
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from c598110 to 740b2d4 Compare June 6, 2024 16:10
@openshift-merge-robot openshift-merge-robot removed the needs-rebase PR needs to be rebased label Jun 6, 2024
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from 740b2d4 to cbf9cfe Compare June 6, 2024 17:24
@Gkrumbach07
Copy link
Member Author

/retest

@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from cbf9cfe to be12313 Compare June 7, 2024 12:50
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from be12313 to 872a099 Compare June 7, 2024 14:17
Copy link
Member

@andrewballantyne andrewballantyne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, let's go with this implementation. We'll be looking at replacing this logic with more structured backend soon, so this is temporary.

We need loading spinners for the artifact peek loading & we definitely need to consider why the Artifact page does not have the peek -- it's awkward that the direct source (the artifact page itself) of the data doesn't have it but the embedded sources (on the runs page) do...

/hold

Not sure if you're addressing the loading spinners now or later, unhold it if you'll follow up in another PR.

@openshift-ci openshift-ci bot added do-not-merge/hold This PR is hold for some reason lgtm approved labels Jun 7, 2024
chore: Update ArtifactPreview component to show loading spinner during fetch

chore: Update ArtifactPreview component to show loading spinner during fetch
@Gkrumbach07 Gkrumbach07 force-pushed the task/RHOAIENG-7081-spike-download-s3-and-minio-from-the-dashboard-ba branch from 872a099 to c5ecff0 Compare June 7, 2024 14:43
@openshift-ci openshift-ci bot removed the lgtm label Jun 7, 2024
Copy link
Member

@andrewballantyne andrewballantyne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

/unhold

@openshift-ci openshift-ci bot added lgtm and removed do-not-merge/hold This PR is hold for some reason labels Jun 7, 2024
Copy link
Contributor

openshift-ci bot commented Jun 7, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewballantyne

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 34187a9 into opendatahub-io:main Jun 7, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants