Add support for Windows Hosts #4
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When Fiji directory is placed in a directory inwhere there is a whitespace, " ", the python external command worker file will fail to run, as it will treat any characters following whitespace as command arguments.
On Windows the multiprocessing library relies on the spawn backend will import (i.e. execute) the main module at start. While this is (negligibly) slower, it is safer. An if __name__ == '__main__': guard in the external command will avoid creating itself recursively, resolving the RuntimeError. On Linux, the code will still default to the "fork" backend, which is not available on Windows. This appears to be correct in the 2D and 3D xanes module, but without test data I am unable to verify.
On Windows hosts with 64 logical cores or more, multiprocessing library will error out because python will exceed the Windows limitation of _winapi.WaitForMultipleObjects. When a host machine with an invalid configuration (64 logical processors and windows OS) is detected, limit processes created down to 40. While resources online recommend 60 or 61 to account for any existing python processes, occasionally, I've experimentally found that Tomopy(?) will not always clean up, so a buffer of 20 threads was given to have a consistently stable environment for batch jobs. As to the performance hit, I've found it to not be significant as the processor is still nearly fully utilized (All 32 physical cores are fully saturated on our AMD host, so therefore all 64 logical cores report a near 100% utilization) (This may differ between Intel's hyperthreading, and AMD's SMT, but that is out of the scope of this fix).
This resolves the current version conflict for the ImageJ package in anaconda when setting the python version to 3.10 on a windows host Additionally, adds that windows support was added by myself, in the future if any other issues from a windows host come up, I am happy to help address them
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following changes enable TXM_Sandbox to be run on Windows hosts (Tested on Windows 11, Windows 10) smoothly and efficiently.
Detailed writeups of problem solutions and commented codes are available in the individual commits that make up the pull request. Primarily the issues that were solved are enabling whitespace in external command path, windows multiprocessing support using the correct syntax for "spawn" as opposed to "fork" (Linux hosts unchanged), windows limitation fixes: _winapi.WaitForMultipleObjects maximum of 63 is now respected, and all physical cores continue to be saturated) (Linux hosts again unchanged).
Lastly, a small addendum is added to the README.md with the commands that are needed to install TXM_Sandbox in Windows through anaconda without the ImageJ dependency conflict on Python 3.10.
Adapted for the Kisailus BNM in house data processing computer running Windows 10 Pro 22H2. On our copy of the software, I've "installed" TXM_Sandbox as a "native" app with a desktop icon in order to facilitate usage of the program by undergraduates without CS experience. If this is a feature you'd like other researchers to share I could additionally parameterize it and submit it in a future pull request.