Data Structure: ONNXRuntime Performance Test

[agemagician]

Hello,

I want to run "perftest" on a complete dataset, as shown here:
https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest

Currently, I am creating each sample in a separate folder as:

--ModelName
    --test_data_set_0
        --input_0.pb
    --test_data_set_n
        --input_0.pb
    --model.onnx

Is this the correct way or I should change it to:

--ModelName
    --test_data_set_0
        --input_0.pb
        --input_n.pb
    --model.onnx

As far as I understood, each folder should have only a single example, and if I have multiple inputs, it should be in a single folder.

It will be great if you can clarify this point.

Thanks.

[snnn]

Please see the examples at: https://github.com/onnx/models .

[agemagician]

Thanks for your reply.
I have already seen it before, but it is still not clear to me.

It will be great if you can explain it.

[snnn]

If the module has multiple inputs, they should be all in the same folder.

[agemagician]

Thanks a lot for the clarification, but I am interested in running the performance for a complete dataset.
Each sample has a single input and a single output.
In this case, I must put each sample in a separate folder ? , i.e.:

--ModelName
    --test_data_set_0
        --input_0.pb
    --test_data_set_n
        --input_0.pb
    --model.onnx

[snnn]

Sorry it is not supported. For advanced usages, I suggest you onboarding your model to https://github.com/mlcommons/inference. You don't need to put your code in their repo, but you can reuse their test framework, especially the loadgen lib. They are more professional.