How to use tensorrt in FP16 or Int8 mode?

[xuzhenqi]

Mxnet1.3.0 supports tensorrt for inference, but I do not find any tutorials or examples to show how to inference a model in FP16 or Int8 mode. So is there a way to do it?

[marcoabreu]

Hello, the usage of TensorRT is described in https://github.com/apache/incubator-mxnet/pull/12548/files

The tutorial will be available on our website soon. We are currently encountering some issues with the website and thus paused all publishing of new tutorials.

@KellenSunderland @aaronmarkham @lupesko

[kalyc]

Thanks for opening this issue @xuzhenqi
You can also post this question on discuss.mxnet.io to get further help from the community.
@mxnet-label-bot[Question]

[aaronmarkham]

There is a TensorRT tutorial on the website.

It wasn't cherry-picked for the v1.3.0 release branch so it doesn't show up there... only in master.

[marcoabreu]

Could you show where you found that? Besides with a direct link

[lizhen2017]

The tensorrt compiling process is slow, is it common?

[aaronmarkham]

@marcoabreu It wasn't added to the tutorials index. #12587 fixed that.
@xuzhenqi Please close this issue if you have the resources you were requesting. Thanks!

[KellenSunderland]

@lizhen2017 it can take some time to compile as it does some auto-tuning during engine creation. Would you like to see a feature that would allow you to cache the engine creation? If so I can add it to our backlog.

[lizhen2017]

@KellenSunderland
I'll appreciate if you do so.Thanks for your help!

[lizhen2017]

By the way, how to install mxnet-tensorrt-cu90 from sources, because I find that mxnet-tensorrt pip-installed version just support forwarding well, fucntions such as datasets processing and so on , can't be access to.

[ahmadudin]

Is there any example on how i can use tensorrt on mxnet using c++ ?