🐛 [Bug] Error loading model using Torch TensorRT in Libtorch on Windows

[Mmmyyym]

Environment

• Libtorch 2.5.0.dev (latest nightly) (built with CUDA 12.4)
• CUDA 12.4
• TensorRT 10.1.0.27
• PyTorch 2.4.0+cu124
• Torch-TensorRT 2.4.0
• Python 3.12.8
• Windows 10

Compile Torch-TensorRT with Cmake to generate lib and dll：

Option : Export
If you want to optimize your model ahead-of-time and/or deploy in a C++ environment, Torch-TensorRT provides an export-style workflow that serializes an optimized module. This module can be deployed in PyTorch or with libtorch (i.e. without a Python dependency).

1. Optimize + serialize

import torch
import torch_tensorrt

model = MyModel().eval().cuda() # define your model here
inputs = [torch.randn((1, 3, 224, 224)).cuda()] # define a list of representative inputs here

trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs)
torch_tensorrt.save(trt_gm, "trt.ep", inputs=inputs) # PyTorch only supports Python runtime for an ExportedProgram. For C++ deployment, use a TorchScript file
torch_tensorrt.save(trt_gm, "trt.ts", output_format="torchscript", inputs=inputs)

2. Deploy
   Deployment in C++:

#include "torch/script.h"
#include "torch_tensorrt/torch_tensorrt.h"

auto trt_mod = torch::jit::load("trt.ts");
auto input_tensor = [...]; // fill this with your inputs
auto results = trt_mod.forward({input_tensor});

ERROR
auto trt_mod = torch::jit::load("trt.ts")

Unknown type name '__torch__.torch.classes.tensorrt.Engine':
  File "code/__torch__/torch_tensorrt/dynamo/runtime/_TorchTensorRTModule.py", line 6
  training : bool
  _is_full_backward_hook : Optional[bool]
  engine : __torch__.torch.classes.tensorrt.Engine
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`your text`~~~~~~~~ <--- HERE
  def forward(self: __torch__.torch_tensorrt.dynamo.runtime._TorchTensorRTModule.TorchTensorRTModule,
    input: Tensor) -> Tensor:

[narendasan]

What flags did you use to compile? Usually this issue comes from libtorch_tensorrt getting optimized out because there is not a direct reference in most inference programs. Not sure what the flag would be for MVSC but for GCC you do something like -Wl,--no-as-needed to avoid this issue

[Mmmyyym]

What flags did you use to compile? Usually this issue comes from libtorch_tensorrt getting optimized out because there is not a direct reference in most inference programs. Not sure what the flag would be for MVSC but for GCC you do something like -Wl,--no-as-needed to avoid this issue

Thank you for the suggestion! The linker optimization issue you mentioned was likely the root cause. Following your advice, I tried a different approach—explicitly loading the DLL via LoadLibraryA instead—and it resolved the problem!

[narendasan]

Nice, reopen if there are further issues