Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 [Bug] Incorrect Documentation page #2706

Open
Skier23 opened this issue Mar 23, 2024 · 1 comment
Open

🐛 [Bug] Incorrect Documentation page #2706

Skier23 opened this issue Mar 23, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@Skier23
Copy link

Skier23 commented Mar 23, 2024

This page has several issues with the provided examples:
https://pytorch.org/TensorRT/user_guide/saving_models.html

  1. In the examples, an optional parameter is passed before a non-optional parameter which is invalid python. This issue also appears here:
  2. The a) example does not work because the default from torch_tensorrt.compile is to use an output_format of ExportedProgram.
  3. This line is no longer valid: trt_exp_program = torch_tensorrt.dynamo.export(trt_gm, images)
  4. torch.jit.trace should not be used when specifying the correct output format. The corrected code should be as follows:
    `
    model = MyModel().eval().cuda()
    inputs = [torch.randn((1, 3, 224, 224)).cuda()]
    trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs=inputs, output_format="torchscript") # Output is a torch.fx.GraphModule
    torch.jit.save(trt_gm, "trt_model.ts")

Later, you can load it and run inference

model = torch.jit.load("trt_model.ts").cuda()
model(*inputs)
`

@Skier23 Skier23 added the bug Something isn't working label Mar 23, 2024
@mjain7
Copy link

mjain7 commented Sep 26, 2024

@Skier23 @peri044
I get below error when try to save tensorrt compiled model

Traceback (most recent call last):
  File "/scratch_space/model.py", line 10, in <module>
    torch.jit.save(trt_gm, "trt_model.ts")
  File "/usr/local/lib/python3.10/dist-packages/torch/jit/_serialization.py", line 83, in save
    m.save(f, _extra_files=_extra_files)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1893, in __getattr__
    raise AttributeError(
AttributeError: 'GraphModule' object has no attribute 'save'

can you please help? (I am running inside nvcr.io/nvidia/pytorch:24.08-py3 container)

Also, how to load trt_model.ts on triton inference server? With platform: "pytorch_libtorch" in config.pbtxt config file, it expects model file to be in .pt format in model_repository. Is there a way to save tensorrt compiled file in .pt format?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants