-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ValueError: No performance file performance_all_0.json found: your network has failed. #88
Comments
Hi @ZhaooooooYUE , WORC makes use of fastr to execute all jobs in an experiment, which gives the reports you showed on which jobs have successfully executed and failed. You have to debug the failed experiment to figure out which specific jobs have failed for which reasons, e.g., the Python error of a specific script. In this case, your classification results has failed because your feature extraction jobs have failed, and we need to figure out the reason that happened. See this part of the WORC manual on debugging a failed network: https://worc.readthedocs.io/en/latest/static/user_manual.html#debugging. If you follow that, which means you end up with running a command similar to Hope that clears things up: feel free to share the exact error / output of the fastr trace here if you need help. Kind regards, Martijn |
Dear Martijn, Thank you for taking the time to respond to my inquiry. Your reply has been very helpful in allowing me to better understand the issue. Based on the information you provided regarding fastr trace, I inspected the code and found that it is able to generate "feature" files but not "performance" files. I attempted to run the fastr trace command:
However, the error message provided is not specific enough to determine the exact cause of the error. I have attached a screenshot of the error message for your reference. I would be very grateful if you could assist me once again in identifying the specific cause of the error. Thank you again for your time and support. Sincerely, |
You are almost there, you should trace back the error to a single sample in a sink, which corresponds to single job, to get exact errors. Hence in this case that would be
From the looks of it however, it seems also some of the feature computations failed, since you get 23 failed samples in the features_train_MRI_0_pyradiomics sink. Hence could you also run such a fastr trace on a failed sample from that sink, something like
You can check which samples failed with
Some samples appear in multiple sinks, so you will see multiple failed jobs, but based on the IDs of your samples you should be able to figure out which are the ones that failed in the features sink. If you could share the output of these, ideally in text instead of screenshots, we will see the error and I can help you further! |
Dear Martijn, ===== JOB WORC_MDD_05293___calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0___HC_0012_0 ===== ----- ERRORS -----
No process information: Output data: Status history: It seemed that Pyradiomics didn't calculate the features, but the provided information are not useful to solve it. I also checked that all subjects didn't have the calculated features from Pyradimics, but succedd in PREDICT. |
Dear Martijn, Thank you for your prompt response, which has helped me to better troubleshoot the current issue. Following your guidance, I have gathered the following information. I have attached the relevant code and error logs to this email for your reference:
Your help would be greatly appreciated. Please let me know if you require any additional information. Sincerely, |
Great, now we see the actual error in the PyRadiomics Python job. For your info, the command you see in this report is exactly what was being executed on the command line, hence you can run that to replicate the error, or debugging is you change something. The error you got very much reminds me of a previous error in PyRadiomics 3.0.0, see AIM-Harvard/pyradiomics#592 and my fix in https://worc.readthedocs.io/en/latest/static/faq.html#error-file-lib-site-packages-numpy-lib-function-base-py-line-4406-in-delete-keep-obj-false-indexerror-arrays-used-as-indices-must-be-of-integer-or-boolean-type. That issue was fixed in PyRadiomics version 3.0.1, which I personally would highly recommend over both 3.0.0 and 3.1.0 which is also sometimes buggy. So I would recommend: check the PyRadiomics version to see if that could be the issue, if it's not 3.0.1. go to that version, and rerun the calling command of this job instead of the whole WORC experiment to see if that helped:
@Jing-Bin-Lab your issue seems slightly different as you see from the fastr traces. Fully agreed that the provided information is not descriptive enough to solve it, I've send a message to the fastr developer to ask if he can help and improve these kind of messages. So it's also a bit guessing for me. The FastrExecutableNotFoundError seems to hint that maybe pyradiomics cannot be called as an executable: can you check that? You could see if that specific job has actually run, and copy the calling command to execute it on the command line to hopefully get a more descriptive error. Unfortunately for you it's not in the fastr trace you did, but should be in the file /home/bjing/Desktop/MDD/WORC_1_MDD_05293/calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0/HC_0012_0/fastr_stdout.txt. If that's not it: when running the experiment, do you see earlier jobs failing, e.g., images, segmentations, segmentix, that may cause this job to have been cancelled? |
following information:
When I run the code, I receive the following error:
ValueError: No performance file performance_all_0.json found: your network has failed.
I have also noticed that the following jobs have failed or been cancelled:
-1-: calcfeatures_train_pyradiomics_Pyradiomics_1_0_MRI_0___MDD_0012_0 with status JobState.failed
-2-: featureconverter_train_pyradiomics_Pyradiomics_1_0_MRI_0___MDD_0012_0 with status JobState.cancelled
I am not sure what is causing these errors and would be grateful for any assistance you could provide. I have checked the documentation but have not been able to find a solution.
Would you be able to suggest any steps I can take to resolve this issue? I would be happy to provide any additional information that may be helpful.
Thank you for your time and consideration.
The text was updated successfully, but these errors were encountered: