Skip to content

🎨 replace project_id and node_id with appropriate labels when exporting #7508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

GitHK
Copy link
Contributor

@GitHK GitHK commented Apr 10, 2025

What do these changes do?

When exporting data, project_id is replaced with the project's name while node_id is replaced with the node's label.
Also, special character / is escaped with _.

Related issue/s

How to test

Dev-ops checklist

@GitHK GitHK self-assigned this Apr 10, 2025
@GitHK GitHK added this to the Pauwel Kwak milestone Apr 10, 2025
Copy link

codecov bot commented Apr 10, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.00%. Comparing base (f35d515) to head (2f54a14).

❗ There is a different number of reports uploaded between BASE (f35d515) and HEAD (2f54a14). Click for more details.

HEAD has 29 uploads less than BASE
Flag BASE (f35d515) HEAD (2f54a14)
unittests 30 1
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #7508       +/-   ##
===========================================
- Coverage   87.43%   69.00%   -18.44%     
===========================================
  Files        1740      739     -1001     
  Lines       67389    34628    -32761     
  Branches     1143      170      -973     
===========================================
- Hits        58924    23895    -35029     
- Misses       8145    10675     +2530     
+ Partials      320       58      -262     
Flag Coverage Δ
integrationtests 65.14% <ø> (+<0.01%) ⬆️
unittests 88.03% <100.00%> (+1.41%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 77.24% <ø> (-8.16%) ⬇️
agent ∅ <ø> (∅)
api_server ∅ <ø> (∅)
autoscaling ∅ <ø> (∅)
catalog ∅ <ø> (∅)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 78.55% <ø> (-12.74%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 89.08% <ø> (-1.04%) ⬇️
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage 88.03% <100.00%> (+0.44%) ⬆️
webclient ∅ <ø> (∅)
webserver 59.13% <ø> (-26.75%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f35d515...2f54a14. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@GitHK GitHK added the a:storage issue related to storage service label Apr 10, 2025
@GitHK GitHK marked this pull request as ready for review April 10, 2025 13:31
@GitHK GitHK requested a review from sanderegg as a code owner April 10, 2025 13:31
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just wondering why you did not re-use the function that is already around?

).get_project_id_and_node_id_to_names_map(
project_uuids=_get_project_ids(user_selecton={x[0] for x in source_object_keys})
)

archive_entries: ArchiveEntries = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about re-using this??

def update_display_fields(self, id_name_mapping: dict[str, str]) -> None:
        display_path = f"{self.path}"
        for old, new in id_name_mapping.items():
            display_path = display_path.replace(old, urllib.parse.quote(new, safe=""))
        self.display_path = Path(display_path)

instead of _replace_node_id_project_id_in_path ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did not convince me about this is:

  • I require different escaping here
  • does not run in linear time

What I like about it:

  • only replaces what it finds (no risk of not finding anything)

The actual think that annoys me is the non liner runtime. If a user exports one or more projects, the more nodes there are the worst this gets.
This will scale with the number of files times the number of nodes + projects instead of just the number of files

@GitHK GitHK requested a review from sanderegg April 11, 2025 09:34
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:storage issue related to storage service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants