You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The a) example does not work because the default from torch_tensorrt.compile is to use an output_format of ExportedProgram.
This line is no longer valid: trt_exp_program = torch_tensorrt.dynamo.export(trt_gm, images)
torch.jit.trace should not be used when specifying the correct output format. The corrected code should be as follows:
`
model = MyModel().eval().cuda()
inputs = [torch.randn((1, 3, 224, 224)).cuda()]
trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs=inputs, output_format="torchscript") # Output is a torch.fx.GraphModule
torch.jit.save(trt_gm, "trt_model.ts")
Later, you can load it and run inference
model = torch.jit.load("trt_model.ts").cuda()
model(*inputs)
`
The text was updated successfully, but these errors were encountered:
@Skier23@peri044
I get below error when try to save tensorrt compiled model
Traceback (most recent call last):
File "/scratch_space/model.py", line 10, in <module>
torch.jit.save(trt_gm, "trt_model.ts")
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_serialization.py", line 83, in save
m.save(f, _extra_files=_extra_files)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1893, in __getattr__
raise AttributeError(
AttributeError: 'GraphModule' object has no attribute 'save'
can you please help? (I am running inside nvcr.io/nvidia/pytorch:24.08-py3 container)
Also, how to load trt_model.ts on triton inference server? With platform: "pytorch_libtorch" in config.pbtxt config file, it expects model file to be in .pt format in model_repository. Is there a way to save tensorrt compiled file in .pt format?
This page has several issues with the provided examples:
https://pytorch.org/TensorRT/user_guide/saving_models.html
TensorRT/docsrc/user_guide/saving_models.rst
Line 20 in 7f14221
`
model = MyModel().eval().cuda()
inputs = [torch.randn((1, 3, 224, 224)).cuda()]
trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs=inputs, output_format="torchscript") # Output is a torch.fx.GraphModule
torch.jit.save(trt_gm, "trt_model.ts")
Later, you can load it and run inference
model = torch.jit.load("trt_model.ts").cuda()
model(*inputs)
`
The text was updated successfully, but these errors were encountered: