feat: engine caching

[zewenli98]

Description

Engine caching feature. More details see: #2957

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes
• I have added the relevant labels to my PR in so that relevant reviewers are notified

[github-actions]

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/dynamo/backend/backends.py	2024-07-16 20:09:24.911867+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/dynamo/backend/backends.py	2024-07-16 20:11:22.623270+00:00
@@ -100,11 +100,13 @@

            gm = post_lowering(gm, sample_inputs)

            logger.debug("Lowered Input graph:\n " + str(gm.graph))

-            torchtrt_inputs = prepare_inputs(torch_inputs, disable_memory_format_check=True)
+            torchtrt_inputs = prepare_inputs(
+                torch_inputs, disable_memory_format_check=True
+            )
            trt_compiled = compile_module(
                gm,
                torchtrt_inputs,
                settings=settings,
            )

[narendasan]

Do you have tests for this?

[peri044]

Added some comments. Functionality looks good. One question - Does the hash change if the module has the same graph but different weights ?

[zewenli98]

Added some comments. Functionality looks good. One question - Does the hash change if the module has the same graph but different weights ?

Nope, all weights are set to 0 before hashing, so if the architectures are the same they will be considered isomorphic

[narendasan]

Added some comments. Functionality looks good. One question - Does the hash change if the module has the same graph but different weights ?

Nope, all weights are set to 0 before hashing, so if the architectures are the same they will be considered isomorphic

I dont really get where we refit for functionality in the code now if we store weight stripped (or zero'd) graphs. Also if we set all weights to 0 what happens in data-dependent cases? Can we detect these cases?

[zewenli98]

I dont really get where we refit for functionality in the code now if we store weight stripped (or zero'd) graphs.

The current code calls refit after interpreter.run() but I plan to move it into interpreter.run().

Also if we set all weights to 0 what happens in data-dependent cases? Can we detect these cases?

The current engine caching happens after partition.
If an input is data dependent, does it generate different TRT engines given different actual inputs?

[narendasan]

LGTM!