Torch-TensorRT

Ahead-of-time compilation of TorchScript / PyTorch JIT for NVIDIA GPUs

Torch-TensorRT is a compiler for PyTorch/TorchScript, targeting NVIDIA GPUs via NVIDIA’s TensorRT Deep Learning Optimizer and Runtime. Unlike PyTorch’s Just-In-Time (JIT) compiler, Torch-TensorRT is an Ahead-of-Time (AOT) compiler, meaning that before you deploy your TorchScript code, you go through an explicit compile step to convert a standard TorchScript program into an module targeting a TensorRT engine. Torch-TensorRT operates as a PyTorch extention and compiles modules that integrate into the JIT runtime seamlessly. After compilation using the optimized graph should feel no different than running a TorchScript module. You also have access to TensorRT’s suite of configurations at compile time, so you are able to specify operating precision (FP32/FP16/INT8) and other settings for your module.

More Information / System Architecture:

• GTC 2020 Talk

Getting Started

• Installation

• Getting Started with C++

• Post Training Quantization (PTQ)

• torchtrtc

• Using Torch-TensorRT Directly From PyTorch

• Deploying Torch-TensorRT Programs

• DLA

Python API Documenation

• torch_tensorrt

• torch_tensorrt.logging

• torch_tensorrt.ptq

• torch_tensorrt.ts

C++ API Documenation

• Namespace torch_tensorrt

• Namespace torch_tensorrt::logging

• Namespace torch_tensorrt::ptq

• Namespace torch_tensorrt::torchscript

Contributor Documentation

• System Overview

• Writing Converters

• Useful Links for Torch-TensorRT Development

Indices

• Operators Supported

• Index

• Search Page