Triton Inference Server Documentation

User Guide

The User Guide describes how to use Triton as an inference solution, including information on how to configure Triton, how to organize and configure your models, how to use the C++ and Python clients, etc.

• QuickStart
  • Install Triton
  • Create Model Repository
  • Run Triton
• Model Repository
  • Cloud Storage
  • File Organization
  • Model Versioning
• Model Configuration
  • Required Model Configuration
    • Maximum Batch Size - Batching and Non-Batching Models
    • Input and Output Tensors
      • Tensor Datatypes
      • Tensor Reshape
      • Shape Tensor
  • Auto-Generate Required Model Configuration
  • Version Policy
  • Instance Groups
    • Specifying Multiple Model Instances
    • CPU and GPU Instances
    • Configuring Rate Limiter
  • Optimization Settings
    • Framework-Specific Optimization
      • ONNX-TensorRT
      • ONNX-OpenVINO
      • TensorFlow-TensorRT
      • TensorFlow-Mixed-Precision
    • NUMA Optimization
  • Scheduling and Batching
    • Default Scheduler - Non-Batching
    • Dynamic Batcher
      • How to Configure Dynamic Batcher
        • Delayed Batching
        • Preferred Batch Size
      • Preserving Request Ordering
      • Priority Levels
      • Queuing Policies
      • Ragged Batching
    • Sequence Batcher
      • Stateful Models
      • Control Inputs
      • Implicit State - Stateful Inference Using a Stateless Model
      • Sequence Scheduling Strategies
        • Direct
        • Oldest
    • Rate Limiter
  • Model Warmup
  • Inference Request/Response Cache
• Model Pipeline
  • Model Ensemble
  • Business Logic Scripting (BLS)
• Model Management
  • Explicit Model Loading and Unloading
  • Modifying the Model Repository
• Metrics
• Framework Custom Operations
  • TensorRT
  • TensorFlow
  • PyTorch
  • ONNX
• Client Libraries and Examples
  • C++ HTTP/GRPC Libraries
  • Python HTTP/GRPC Libraries
  • Java HTTP Library
  • GRPC Generated Libraries
    • go
    • Java/Scala
    • Javascript
• Performance Analysis
  • Performance Tuning Guide
  • Optimization
  • Model Analyzer
  • Performance Analyzer
  • Inference Request Tracing
• Jetson and JetPack

Developer Guide

The Developer Guide describes how to build and test Triton and also how Triton can be extended with new functionality.

• Build
• Protocols and APIs.
• Backends
• Repository Agents
• Test