Dynamic shape index #3039

apbose · 2024-07-25T23:21:11Z

No description provided.

github-actions

There are some changes that do not conform to Python style guidelines:

peri044 · 2024-07-31T21:04:20Z

@apbose is this ready ?

apbose · 2024-08-01T00:04:13Z

@peri044 Right now the dynamic input cases are supported, I will add some test cases for dynamic index. Although the implementation should support it. Also, is there any example script to test it with mistral-7b and SD? I did not test it end to end with the models yet

peri044 · 2024-08-01T19:28:52Z

@peri044 Right now the dynamic input cases are supported, I will add some test cases for dynamic index. Although the implementation should support it. Also, is there any example script to test it with mistral-7b and SD? I did not test it end to end with the models yet

For mistral model, you can use the same example https://github.com/pytorch/TensorRT/blob/llm_examples_main/examples/dynamo/torch_export_llama2.py and swap the model name to mistralai/Mistral-7B-v0.3

peri044 · 2024-08-09T00:18:48Z

py/torch_tensorrt/dynamo/conversion/impl/select.py

+                (
+                    get_shape(
+                        ctx,
+                        target,
+                        source_ir,
+                        name + f"_transpose_tensor_shape_mult_d0_{i}",
+                        transpose_tensor,
+                        i,
+                    )
+                    if transpose_tensor_shape[i] == DYNAMIC_DIM
+                    else transpose_tensor_shape[i]
+                ),


consider storing this in a variable for better readability

peri044 · 2024-08-09T00:19:11Z

py/torch_tensorrt/dynamo/conversion/impl/select.py

+                (
+                    get_shape(
+                        ctx,
+                        target,
+                        source_ir,
+                        name + f"_transpose_tensor_shape_mult_d1_{i}",
+                        transpose_tensor,
+                        i,
+                    )
+                    if transpose_tensor_shape[i] == DYNAMIC_DIM
+                    else transpose_tensor_shape[i]
+                ),


same as above

peri044 · 2024-08-09T00:21:29Z

py/torch_tensorrt/dynamo/conversion/impl/select.py

-                dim_tensor_list[adv_indx_indices[adv_indx_count - 1]],
-                name + "_dim_last",
-            )
+            multiplier = dim_tensor_list[adv_indx_indices[adv_indx_count - 1]]


Does this change imply multiplier is always an ITensor since you were using get_trt_tensor before ?

Yes we do not need get_trt_tensor in this, even in the previous non dynamic cases it was not required since dim_tensor_list already stores a list of dimension tensors

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:09:49.859558+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:10:09.332451+00:00
@@ -532,6 +532,6 @@

                with enable_torchbind_tracing():
                    exp_program = torch.export.export(
                        module, tuple(arg_inputs), kwargs=kwarg_inputs, strict=False
                    )
-                    torch.export.save(exp_program, file_path)
\ No newline at end of file
+                    torch.export.save(exp_program, file_path)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:18:47.939073+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:19:11.393152+00:00
@@ -532,6 +532,6 @@

                with enable_torchbind_tracing():
                    exp_program = torch.export.export(
                        module, tuple(arg_inputs), kwargs=kwarg_inputs, strict=False
                    )
-                    torch.export.save(exp_program, file_path)
\ No newline at end of file
+                    torch.export.save(exp_program, file_path)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:20:46.489048+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:21:12.958728+00:00
@@ -532,6 +532,6 @@

                with enable_torchbind_tracing():
                    exp_program = torch.export.export(
                        module, tuple(arg_inputs), kwargs=kwarg_inputs, strict=False
                    )
-                    torch.export.save(exp_program, file_path)
\ No newline at end of file
+                    torch.export.save(exp_program, file_path)

apbose · 2024-08-16T00:23:31Z

I ran the Mistral-7B-v0.3, it shows me this in the end-

INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:08:12.032557
INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.
ERROR:torch_tensorrt [TensorRT Conversion Context]:1: [defaultAllocator.cpp::allocate::19] Error Code 1: Cuda Runtime (out of memory)
WARNING:torch_tensorrt [TensorRT Conversion Context]:Requested amount of GPU memory (28354543872 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
double free or corruption (!prev)

Looks like it got past the dynamic index case

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:24:39.231642+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-16 00:25:02.079196+00:00
@@ -532,6 +532,6 @@

                with enable_torchbind_tracing():
                    exp_program = torch.export.export(
                        module, tuple(arg_inputs), kwargs=kwarg_inputs, strict=False
                    )
-                    torch.export.save(exp_program, file_path)
\ No newline at end of file
+                    torch.export.save(exp_program, file_path)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-22 23:54:07.789419+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/_compile.py	2024-08-22 23:54:27.515379+00:00
@@ -532,6 +532,6 @@

                with enable_torchbind_tracing():
                    exp_program = torch.export.export(
                        module, tuple(arg_inputs), kwargs=kwarg_inputs, strict=False
                    )
-                    torch.export.save(exp_program, file_path)
\ No newline at end of file
+                    torch.export.save(exp_program, file_path)

peri044 · 2024-08-29T23:00:59Z

TestIndexDynamicConstantConverter tests are failing in CI

apbose self-assigned this Jul 25, 2024

facebook-github-bot added the cla signed label Jul 25, 2024

apbose marked this pull request as draft July 25, 2024 23:21

github-actions bot added the component: tests Issues re: Tests label Jul 25, 2024

github-actions bot requested a review from narendasan July 25, 2024 23:21

apbose force-pushed the dynamic_shape_index branch from 059d09a to 7f9d604 Compare July 26, 2024 00:28

github-actions bot added component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Jul 26, 2024

github-actions bot requested changes Jul 26, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from 7f9d604 to daf1be0 Compare July 30, 2024 02:44

apbose marked this pull request as ready for review July 30, 2024 02:45

apbose requested a review from peri044 July 30, 2024 02:45

peri044 reviewed Aug 9, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from daf1be0 to a5ebd76 Compare August 16, 2024 00:09

github-actions bot requested changes Aug 16, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from 673dd7b to 8fba0f2 Compare August 16, 2024 00:20

github-actions bot requested changes Aug 16, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from 8fba0f2 to 8295117 Compare August 16, 2024 00:24

github-actions bot requested changes Aug 16, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from 8295117 to acd0abe Compare August 22, 2024 23:53

github-actions bot requested changes Aug 22, 2024

View reviewed changes

apbose force-pushed the dynamic_shape_index branch from acd0abe to 73a2fdb Compare August 22, 2024 23:55

Dynamic shape index

5dd825e

apbose force-pushed the dynamic_shape_index branch from 73a2fdb to a64c29a Compare August 29, 2024 18:03

apbose force-pushed the dynamic_shape_index branch from a64c29a to 4cafd01 Compare August 30, 2024 07:46

addressing review comments

5379328

apbose force-pushed the dynamic_shape_index branch from 4cafd01 to 5379328 Compare August 30, 2024 09:16

peri044 merged commit 0f8f23d into main Aug 30, 2024
66 of 67 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic shape index #3039

Dynamic shape index #3039

apbose commented Jul 25, 2024

github-actions bot left a comment

peri044 commented Jul 31, 2024

apbose commented Aug 1, 2024 •

edited

Loading

peri044 commented Aug 1, 2024

peri044 Aug 9, 2024

peri044 Aug 9, 2024

peri044 Aug 9, 2024

apbose Aug 16, 2024

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

apbose commented Aug 16, 2024

github-actions bot left a comment

github-actions bot left a comment

peri044 commented Aug 29, 2024

Dynamic shape index #3039

Dynamic shape index #3039

Conversation

apbose commented Jul 25, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

peri044 commented Jul 31, 2024

apbose commented Aug 1, 2024 • edited Loading

peri044 commented Aug 1, 2024

peri044 Aug 9, 2024

Choose a reason for hiding this comment

peri044 Aug 9, 2024

Choose a reason for hiding this comment

peri044 Aug 9, 2024

Choose a reason for hiding this comment

apbose Aug 16, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

apbose commented Aug 16, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

peri044 commented Aug 29, 2024

apbose commented Aug 1, 2024 •

edited

Loading