Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jobs with multiple output files do not have proper naming with batch inputs #1190

Open
kyrieb-ekat opened this issue Jul 15, 2024 · 6 comments
Assignees
Labels
help-wanted Help :^ Priority: HIGH Rodan Client Issues pertaining to Rodan client Type: FIX

Comments

@kyrieb-ekat
Copy link

When you're processing a lot of images through the classifying job, the layers it separates and outputs won't be renamed "job name - Layer X" etc, as it does when you process single images. This makes sifting through and renaming relevant layers somewhat problematic if you need to search and use specific layers for another job, etc. Examples attached below (Image 1, multiple images with layers separated, image 2 a single image with layers separated and successfully renamed).

Screenshot 2024-07-15 at 9 51 21 AM
Screenshot 2024-07-15 at 9 51 50 AM

@homework36 homework36 self-assigned this Jul 16, 2024
@homework36
Copy link

What do you think should be the expected naming practice for this?

@kyrieb-ekat
Copy link
Author

Probably would just need to do what it does for the single upload (the BaRa3 example), where it just puts the file name at the front of "Fast Pixelwise... - Layer X", unless you think abridging it might be more helpful/useful.

@homework36
Copy link

homework36 commented Jul 16, 2024

Okay. I'll check and fix it.
It's not a problem within classifying. All jobs have this same behavior.

@homework36 homework36 changed the title Classifying job won't always output correct titles jobs with multiple output files do not have proper naming with batch inputs Jul 17, 2024
@homework36
Copy link

homework36 commented Sep 9, 2024

Some notes for myself:
File paths and naming are handled by Django upload_to which should work fine and we do not do anything else manually in terms of naming or renaming output files.
Looks like when there're multiple outputs for one inputs, outputs[opt_name]["is_list"] is still False (opt_name is the job name according to logger msg). I'm not sure but it seems to me that it should be a list when there're multiple outputs. There're a few lines in RodanTaskType __init__ that seems to do something related but we never actually get into the else block of the if attrs.get("_abstract") is True statement.
Also looking at the output.py in model (Django) now.

@homework36 homework36 added the help-wanted Help :^ label Sep 13, 2024
@homework36
Copy link

homework36 commented Sep 13, 2024

I believe naming of output files is handled here

class package_results(Task):

However, I cannot print out file names etc using logger.info for this with the following setting

import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("rodan")

I did get it working when it is placed right below the class but not from the if statement for mode 0 and 1.
According to this part of code, output file names should always contain job name and output port name. I wonder if it's a problem with how we handle multiple input resources for the same input port and some info gets lost.

@homework36
Copy link

Just took another look at package_results.
The two packaging modes work as follows:
Mode 0:

tmp_dir/
  Job1 - OutputPortA/
    Resource1.txt
    Resource2.txt
  Job2 - OutputPortB/
    ResourceList1/
      00.jpg
      01.jpg

Mode 1:

tmp_dir/
  Resource1/
    Job1 - OutputPortA.txt
  Resource2/
    Job2 - OutputPortB.txt
  ResourceList1/
    Job2 - OutputPortB.list/
      00.jpg
      01.jpg

However, none of the directory structures are reflected on the resources page. We need to check (1) is this is really used by Rodan and (2) how is this show up with Rodan client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help-wanted Help :^ Priority: HIGH Rodan Client Issues pertaining to Rodan client Type: FIX
Projects
None yet
Development

No branches or pull requests

2 participants