Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add delete segment data update action #7435

Merged
merged 14 commits into from
Dec 13, 2023
Merged

Add delete segment data update action #7435

merged 14 commits into from
Dec 13, 2023

Conversation

frcroth
Copy link
Member

@frcroth frcroth commented Nov 13, 2023

URL of deployed dev instance (used for testing):

Steps to test:

  • Delete segments in WK. You can ensure that they are deleted visually and via fossildb.

TODOs:

  • Frontend: Add remove action to context menu

Issues:


(Please delete unneeded items, merge only when none are left open)

@frcroth frcroth changed the title Add delete segment data update action WIP: Add delete segment data update action Nov 13, 2023
@frcroth frcroth self-assigned this Nov 13, 2023
@philippotto philippotto self-assigned this Nov 14, 2023
@philippotto
Copy link
Member

The front-end part should be done now. I noticed that the RAM allocation of the JVM went up quite a bit (and stayed there) during testing. Not sure whether this is just typical JVM behavior or if there is a memory leak somewhere?

Do we have any measures in place to avoid huge dataset modifications? Especially in combination with a fallback layer, this could quickly become a very long operation, I assume.

@philippotto philippotto removed their assignment Nov 14, 2023
@frcroth frcroth changed the title WIP: Add delete segment data update action Add delete segment data update action Nov 15, 2023
@frcroth frcroth requested a review from fm3 November 15, 2023 08:17
@frcroth frcroth marked this pull request as ready for review November 15, 2023 08:22
@fm3
Copy link
Member

fm3 commented Nov 15, 2023

Hmm good point, this feature is not really prepared for fallback segmentation layers. In fact, currently, the backend will throw an error if this update action is used on a tracing without a segment index. This should definitely be checked in the frontend first.

When we add a segment index with fallback layer functionality, we might have to rethink this whole thing. Otherwise we would potentially update tens of thousands of buckets without meaning to…

@frcroth do you have any ideas about the memory consumption? My assumption would be that the jvm GC should take care of this once there is some memory pressure, but I haven’t looked into this in depth. If you know anything rightaway, let’s discuss it, but don’t spend too much effort researching this now unless we see actual problems. This is likely also related to our general numeric arrays problems (we will want to avoid allocating objects for each element in the future)

@frcroth
Copy link
Member Author

frcroth commented Nov 15, 2023

@frcroth do you have any ideas about the memory consumption? My assumption would be that the jvm GC should take care of this once there is some memory pressure, but I haven’t looked into this in depth. If you know anything rightaway, let’s discuss it, but don’t spend too much effort researching this now unless we see actual problems. This is likely also related to our general numeric arrays problems (we will want to avoid allocating objects for each element in the future)

I also think that is the cause because the code uses the expensive UnsignedInteger objects.

Copy link
Member

@fm3 fm3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backend LGTM and works :) true, it’s slow, but I’d say that’s #3810

Request for the Frontend: this option should only be enabled in case the volume tracing has a segment index.

@philippotto
Copy link
Member

Request for the Frontend: this option should only be enabled in case the volume tracing has a segment index.

What about fallback data? Should I forbid this case, too? Otherwise, once we have segment indices for fallback data, this feature will be automatically available (and probably lead to problems).

@philippotto philippotto self-assigned this Nov 22, 2023
@fm3
Copy link
Member

fm3 commented Nov 22, 2023

Hmm, yes, let’s forbid it there as well. We can revisit this as part of #7132 or later

@philippotto
Copy link
Member

Hmm, yes, let’s forbid it there as well. We can revisit this as part of #7132 or later

Ok, done.

Let me know if you need something else from me!

Copy link
Member

@fm3 fm3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@frcroth
Copy link
Member Author

frcroth commented Nov 27, 2023

@philippotto Does this need frontend review? If so, can you please assign a reviewer?

@philippotto
Copy link
Member

Ok, I requested a review now.

Copy link
Contributor

@MichaelBuessemeyer MichaelBuessemeyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR enabling to 🗑️ a segment.

The code looks good to me 👍. I saw the "problem" with having to hand the resolve callback of a Promise via the action to the saga, but could not come up with a better idea :/. So let's keep it that way.

Just an idea that popped into my mind: The deletion of a segment is like a 3d bucket fill with id 0. Would it be possible to implement this as a feature with the overwrite id as a parameter, so it might replace the current 3d bucket fill done by the worker? But, this is a premium feature, isn't it? And maybe we do not want our users to wait for every bucket fill?

Testing results:

  • I'd add the name of the segment to the warning modal shown before deletion of the data.
  • I am aware that the feature is meant to only delete the (voxel) data of the segment, but after the deletion, the segment in the list is empty, which feels weird to me: Having a segment without any data in the list. I would kinda expect the segment being deleted from the list (although the menu item clearly is named something like "delete data") -> feel free to argue 😄

@fm3 Could you please check whether the feature works by check the fossil-db like felix suggested (In case you think this is testing step is worth it)? I do not know how to do this.

@fm3
Copy link
Member

fm3 commented Dec 4, 2023

the segment in the list is empty

I agree, this is not ideal. I’d say the frontend should always send both the deleteSegmentData and deleteSegment (from list) update action

Could you please check whether the feature works by check the fossil-db

Looking good 👍

@philippotto
Copy link
Member

Just an idea that popped into my mind: The deletion of a segment is like a 3d bucket fill with id 0. Would it be possible to implement this as a feature with the overwrite id as a parameter, so it might replace the current 3d bucket fill done by the worker? But, this is a premium feature, isn't it? And maybe we do not want our users to wait for every bucket fill?

I think, this is a good idea, but nothing for this PR. We have to be careful about not putting too much workload on the server...

I'd add the name of the segment to the warning modal shown before deletion of the data.

Done.

the segment in the list is empty

I agree, this is not ideal. I’d say the frontend should always send both the deleteSegmentData and deleteSegment (from list) update action

I only agree partly. If the user segmented something incorrectly and they want to start from scratch, chances are that they already set a custom name and color for the segment. When automatically deleting the entry from the list, these properties are gone, too. I'd rather delete too little than too much. However, I also see where you are coming from. That's why I pushed a trade-off: After the deletion, a success toast is shown which allows to also delete the segment from the list with one click.

image

I would rather have used a checkbox in the confirm-modal, but this proved to be a bit too difficult.

I think, this trade-off is fine for a feature which is probably rather rarely used. I hope you agree 🤞

@philippotto
Copy link
Member

I think, it's fine to merge. My last change was quite small and I just doublechecked it.

@frcroth frcroth merged commit 5a9ecee into master Dec 13, 2023
2 checks passed
@frcroth frcroth deleted the delete-segment branch December 13, 2023 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Delete segment
4 participants