No installation available of torch_tensorrt for Jet Pack 6.1

[alexoterno]

Hi,
I want to create an docker image with PyTorch, Torchvision, ONNX Runtime GPU and torch_tensorrt to use TensorRT on the Nvidia Jetson Orin Nano with Jet Pack 6.1.
From your latest docker image name dustynv/l4t-pytorch:r36.4.0, I have managed to install:

• torch (2.5.0)
• torchvision (0.20.0)
• onnxruntime-gpu (1.20.0)
  for cuda 12.6 and Python 3.10.

I run some tests and I'm able to run some torch models or onnx models, both, in the CPU and the GPU.

However, I was not able to install torch_tensorrt following and adapting your code from this Dockerfile for Jet Pack 4.6.
Have some insights to install torch_tensorrt on Jet Pack 6.1 ?

Moreover, I saw torch2trt is already installed in the dustynv/l4t-pytorch:r36.4.0 docker image. I did not really understood the difference between torch2trt and torch_tensorrt. It looks like torch2trt is for edge devices but it is not updated since 8 months. Which one is better to use for Nvidia Jetson Orin Nano with Jet Pack 6.1 and which one has the lowest inference time latency?

Thanks for sharing you docker images

[urbste]

I have been stuck with this as well.

Using
jetson-containers build $(autotag torch_tensorrt)

I get this error (it seems to try to find cuda 12.4 although JP61 comes with 12.6):

Repository rule new_local_repository defined at: /root/.cache/bazel/_bazel_root/90b3e7b2ede136300170e5d134cbac49/external/bazel_tools/tools/build_defs/repo/local.bzl:103:39: in <toplevel> ERROR: An error occurred during the fetch of repository '_main~_repo_rules~cuda': Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/90b3e7b2ede136300170e5d134cbac49/external/bazel_tools/tools/build_defs/repo/local.bzl", line 86, column 29, in _new_local_repository_impl children = _get_dir_path(rctx).readdir() File "/root/.cache/bazel/_bazel_root/90b3e7b2ede136300170e5d134cbac49/external/bazel_tools/tools/build_defs/repo/local.bzl", line 55, column 13, in _get_dir_path fail( Error in fail: The repository's path is "/usr/local/cuda-12.4/" (absolute: "/usr/local/cuda-12.4") but it does not exist or is not a directory. ERROR: no such package '@@_main~_repo_rules~cuda//': The repository's path is "/usr/local/cuda-12.4/" (absolute: "/usr/local/cuda-12.4") but it does not exist or is not a directory. ERROR: /root/.cache/bazel/_bazel_root/90b3e7b2ede136300170e5d134cbac49/external/_main~_repo_rules~tensorrt/BUILD.bazel:53:11: @@_main~_repo_rules~tensorrt//:nvinfer depends on @@_main~_repo_rules~cuda//:cudart in repository @@_main~_repo_rules~cuda which failed to fetch. no such package '@@_main~_repo_rules~cuda//': The repository's path is "/usr/local/cuda-12.4/" (absolute: "/usr/local/cuda-12.4") but it does not exist or is not a directory. ERROR: Analysis of target '//:libtorchtrt' failed; build aborted: Analysis failed INFO: Elapsed time: 7.883s, Critical Path: 0.08s INFO: 1 process: 1 internal. ERROR: Build did NOT complete successfully FAILED:

Next I uncommented all the torch_tensorrt build related code from the Dockerfile you posted.
Then I ran the docker container and tried to build torch_tensorrt manually inside the container.

If I check out the main branch from torch_tensorrt I get the following errors:

ERROR: /tmp/torch_tensorrt/cpp/lib/BUILD:5:10: Linking cpp/lib/libtorchtrt_plugins.so failed: (Exit 1): gcc failed: error executing CppLink command (from target //cpp/lib:libtorchtrt_plugins.so) /usr/bin/gcc @bazel-out/aarch64-opt/bin/cpp/lib/libtorchtrt_plugins.so-2.params

[dusty-nv]

@urbste @alexoterno torch_tensorrt typically requires updates to the build scripts, you may be able to find them in that repo if they have been updated for JetPack 6.1. torch2trt is lighterweight to install but functionally similar - I end up using that more frequently (like in github.com/dusty-nv/clip_trt) and that still is working on the latest.

[urbste]

@dusty-nv thanks for you response. Yeah I thought torch_tensorrt was more actively developed and in addition I was using the torch_tensorrt.compile and torch_tensorrt.save in a more general setting for workstations as well, I did not want to change all my interfaces for compilation, loading and saving for the jetson devices.

But then I might give torch2trt a try :)

[urbste]

Would it maybe work to run the build script with CUDA_VERSION=12.4 jetson-containers build $(autotag torch_tensorrt) ??
Or do you think the issue comes from something else entirely?

[dusty-nv]

I remember trying torch_tensorrt previously with jp6.1, it had not yet been updated upstream, and I just went back to using torch2trt.

I would say torch_tensorrt is good for production, it has broader support. torch2trt originated out of the jetson group (jetbot, nanoOWL, ect) having been a precursor to torch_tensorrt. Those libraries though that require custom update steps have been difficult to keep updated though.