-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate how to tackle cropping timestamps in raw data #369
Comments
One simple idea mentioned in a Slack thread by myself:
|
The timestamps contain either zeros/ones or the values close to the limit of the data type (65535). In normalisation we seem to avoid division by zero, but then We might also want to alert the user that the data contains "too many" zeroes of values close to the precision limit. May be counting those elements in raw data won't be a big burden? |
quick follow-up. May be we shouldn't terminate a job but warn the user of the percentage of those values in the raw data. I'd say the levels higher than 5% should be alarming. |
You may have already discussed this further, or I may be unaware of other details, so you may not need to reply. Relating to your comment about a loader template:
Would you mean using the standard loader with an alternative preview value? Or standard loader with an additional parameter to specify the exclusion on time stamp? or a slightly different loader? If it was the extra parameter suggestion, would it be something like this: 'crop_timestamp' Relating to Daniil's comment:
It does sound like a piece of feedback the user would want to be aware of if they weren't already. |
@jessicavers I was meaning using the standard loader with a value for the preview:
detector_y:
start: 30 The suggestion was trying to provide the "simplest" solution in that it doesn't do anything clever, nor does it require a new feature or code changes (like an additional parameter or a different loader would). Simply a pre-written loader config for a beamline that can be copy+pasted. |
I think we shouldn't really do anything in addition from the loader point of view. We just need to make sure that the pipeline won't be exited based on some nan's / inf's in some pixels. I think previously it was due to |
Possibly #267, though I'm not 100% sure if that's the one you mean? |
Thank you for clarifying A loader template, using the standard loader, with a specific preview value. A further suggestion, maybe including a message/recommendation within the verbose comment for the preview parameter? Drawbacks would be that it adds extra text. I'm also not sure that many users would know that -vv exists or would notice. Or the possibility of a warning message, "I12 users should be aware that there may be a timestamp present. If so, we recommend preview value: " - perhaps slightly long. |
Okay, yes, that sounds good |
closing this as not planned for now and #372 should help partially. But, as I said earlier, the presence of nans/infs shouldn't break the run. If you remember in Savu the zeros were even intentionally converted to nans in the recon so that Dawn could do better scaling when visualising the images. Not that I like it... |
Some beamlines collect data such that there is a timestamp in the top left corner of projections. Leaving in this information can cause the processing of methods to behave unexpectedly (for example, incorrect normalisation performed), so it is best to crop out this region by cropping the
detector_y
dimension in some way.This issue serves as a discussion place for how to go about doing this in a way that users would find easy.
The text was updated successfully, but these errors were encountered: