Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate thread safety #48

Open
tbirdso opened this issue Aug 10, 2023 · 6 comments
Open

Investigate thread safety #48

tbirdso opened this issue Aug 10, 2023 · 6 comments

Comments

@tbirdso
Copy link
Collaborator

tbirdso commented Aug 10, 2023

From #44 (comment):

When I run tests in parallel with ctest -j8, I get:

  9 - IOOMEZarrNGFF_readSubregion (Subprocess aborted)

Serially, they pass.

I have been able to reproduce these errors with the -j<n> flag with different tests failing. In particular I've noticed errors where Tensorstore appears to fail to read different image dimensions. There may be an issue with store contexts not being unique among threads or processes.

Tasks

  • Add a test to verify that using two OMEZarrNGFFImageIO instances with two ImageReaders of different dimensions in sequence does not result in an error + metadata is read correctly
  • Investigate failures in running tests in parallel
  • Add a test to verify that running multiple threads with one OMEZarrNGFFImageIO instance and ImageReader per thread with different data sources / regions does not result in failures
  • If thread/process/context safety issues are identified, start by documenting the issue ("multiple instances are not allowed") or seeing if we can enforce one context at a time.
  • If safety issues are identified, revisit and determine how/whether we can refactor for thread/process/context safety.
@tbirdso
Copy link
Collaborator Author

tbirdso commented Aug 10, 2023

fyi @thewtex @dzenanz

@dzenanz
Copy link
Member

dzenanz commented Aug 10, 2023

This is a good identification of what to look at next.

@dzenanz dzenanz self-assigned this Aug 30, 2023
@dzenanz
Copy link
Member

dzenanz commented Aug 30, 2023

After all the contents are added to in-memory zip file, the buffer size is 47447. After closeZip() is called, the buffer size increases to the correct value of 47685. This leads me to suspect that destructor is not invoked sometimes.

I will now try to fix this.

@tbirdso
Copy link
Collaborator Author

tbirdso commented Aug 30, 2023

Thanks @dzenanz , that is in reference to the in-memory zip issue correct? #36

@dzenanz
Copy link
Member

dzenanz commented Aug 30, 2023

Yes.

@dzenanz
Copy link
Member

dzenanz commented Aug 30, 2023

I now realize that this discussion should be moved to that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants