Best way for large scale batch inference MMSeg TensorRT model?

[davodogster]

Hello MMDeploy community,
I trained an MMSeg model and converted it to ONNX and then TRT and I was able to do successful inference on a single image using mmdeploy.apis.inference_model.

Is it possible to do large scale batch inference with an MMSeg TensorRT model? I am using a custom Test Time augmentation function so I would like the batch size to be a minimum of 8. I have 100,000s of images to run through the pipeline on a GPU (and I plan to run it on AWS).

It appears there are at least 3 options (for batch=1?) like Model Converter, SDK, Python API. Thanks, Sam

@RunningLeon

[davodogster]

@lvhan028 I tried this in SDK Segmentation with my onnx model but it failed.

SDK inference looks very complex compared to inference_model in python

[lvhan028]

According to the log, loading SDK model failed.
Can you run tree ../mmdeploy/workdir and paste the result here?
Also could you share the command which is used to convert the MMSeg model?

[davodogster]

Hi @lvhan028

Here is the tree command:

Command to convert the mmseg model to onnx is here:
python ~/workspace/mmdeploy/tools/deploy.py \ ~/workspace/mmdeploy/configs/mmseg/segmentation_onnxruntime_dynamic.py \ /workspace/mounted_dir/output/FM_binary-mmseg-pspnet50/fcn_unet.py \ /workspace/mounted_dir/output/FM_binary-mmseg-pspnet50/latest.pth \ /workspace/mounted_dir/data/FM1-indexed-split/test/AT24_1000_3948__7.jpg \ --test-img /workspace/mounted_dir/data/FM1-indexed-split/test/AT24_1000_3948__7.jpg

Thanks, Sam

[lvhan028]

please append --dump-info in the command. It will generate meta info used by mmdeploy SDK

[davodogster]

@lvhan028 thanks for your reply - good to know about --dump-info.

I ran it with dump info and copied the files into /workdir and have rerun the code. Here is the output:

Can I fix this? Cheers, Sam

To generate the .onnx file I used the cpu docker image docker run -it --rm -v pwd:/workspace/mounted_dir mmdeploy:master-cpu

[lvhan028]

It seems that onnxruntime backend is not built. What's your command to build SDK?
-DMMDEPLOY_TARGET_BACKENDS="ort;trt" should be passed to invoke both onnxruntime backend and tensorrt backend

[davodogster]

Thanks @lvhan028 . My other question is: Can TensorRT SDK be used inside the prebuilt MMdeploy:GPU docker image? Is it possible and easy to configure?

[lvhan028]

Yes. It can.
You can refer to https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/01-how-to-build/build_from_docker.md

[davodogster]

@lvhan028 Thanks! Now the Segmenter is very giving different results to TRTWrapper. Both run inside the same Python environment.

Segmenter only takes NP arrays. The TRTWrapper I am feeding a CUDA Torch Tensor

And is it okay to ignore: ?

[12/22/2022-14:43:37] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.5 but loaded cuBLAS/cuBLAS LT 111.0.3

[lvhan028]

Yes. Warnings can be ignored.
But I don't get your question. Do you mean that the result inferred by mmdepoy_python.Segmentor is different from that inferred by TRTWrapper?

[davodogster]

Yes that's what I mean - the results are very different. It should be predicting 100K+pixels for 0 and 1. I have visualised the image and the tensor and they look the same. Does the TRTWrapper automatically preprocess the image? I assume it does?Thanks, Sam

…

________________________________ From: lvhan028 ***@***.***> Sent: Friday, December 23, 2022 3:25:20 AM To: open-mmlab/mmdeploy ***@***.***> Cc: Sam Davidson ***@***.***>; Author ***@***.***> Subject: Re: [open-mmlab/mmdeploy] Best way for large scale batch inference MMSeg TensorRT model? (Discussion #1457) Yes. Warnings can be ignored. But I don't get your question. Do you mean that the result inferred by mmdepoy_python.Segmentor is different from that inferred by TRTWrapper? — Reply to this email directly, view it on GitHub<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mmlab%2Fmmdeploy%2Fdiscussions%2F1457%23discussioncomment-4476141&data=05%7C01%7Csam.davidson%40scionresearch.com%7C81fe670d75ac4370efc008dae4285cb4%7C912c7b2ae3384788b809a191b86cdea0%7C0%7C0%7C638073159262418580%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VOp9vQg3G1tHJE7tV94hXRj6aT24Dqy4IGpifVfl16g%3D&reserved=0>, or unsubscribe<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK7R4HEU7PZXLO33DETCFUTWORQFBANCNFSM6AAAAAASN7RWI4&data=05%7C01%7Csam.davidson%40scionresearch.com%7C81fe670d75ac4370efc008dae4285cb4%7C912c7b2ae3384788b809a191b86cdea0%7C0%7C0%7C638073159262574775%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=My0K7UHa0PnxB167NtZX5wGgl4I1BeFAFsQ%2BT5RVeLw%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***> This email and any attachments are intended solely for the addressees named above. It may contain information which is legally privileged, confidential or subject to copyright. Any unauthorised review, use, disclosure or distribution is prohibited. If you receive this email in error, please delete it. As part of Scion’s cyber security policy, Scion’s IT systems are subject to ongoing monitoring, activity logging and auditing, and may include 3rd party monitoring on our behalf.

[davodogster]

@RunningLeon Thanks that part seems to have worked!

Ill try next steps now to rebuild SDK and then run the Segmentor

[davodogster]

@RunningLeon Thanks! batch inference is working properly now.

Here is the speed for batch size=8, image shape = 992,992,3 for FCN_Unet model

1. Do you think it's significantly faster than using inference with the .pth model (are there any mmdeploy speed benchmarking tests available) ?
2. Can I pass and return torch cuda tensors to the segmentor() function instead of numpy array? If it will speed up inference. My wrapper function expects tensors as input.

[RunningLeon]

Hi,

1. normally tensorrt should be faster than pytorch, but you still have to test on your device.
2. mmdeploy does not provide this feature of Tensor input to segmentor in Python API.

[davodogster]

@davodogster Hi, sorry for the trouble. Could try to change this line

mmdeploy/csrc/mmdeploy/utils/opencv/opencv_utils.cpp

Line 129 in c3986ce

cv::resize(src, dst, dst.size(), method);

to cv::resize(src, dst, dst.size(), 0, 0, method); and rebuild the mmdeploy from source? You could build in your docker container according to https://github.com/open-mmlab/mmdeploy/blob/v0.12.0/docs/en/01-how-to-build/build_from_source.md

Hi @RunningLeon Will there be a future version of the repo (and docker image) that fixes this cv error which has to be manually changed?

[RunningLeon]

@davodogster Yes, it will be fixed in this PR #1625 https://github.com/open-mmlab/mmdeploy/pull/1625/files#diff-15b6fdbc4249496c473fce836516ea8bdd71bc3e9a9432f4f7dc6454362d61c6