Contents:

• TorchServe
• Troubleshooting Guide
• Performance Guide
• Batch Inference with TorchServe
• Code Coverage
• Advanced configuration
• Custom Service
• TorchServe default inference handlers
• Logging in Torchserve
• TorchServe Metrics
• Model Zoo
• Request Envelopes
• Running TorchServe
• TorchServe model snapshot
• TorchServe on Windows
• TorchServe on Windows Subsystem for Linux (WSL)
• Torchserve Use Cases
• TorchServe Workflows
• Serving large models with Torchserve
• FAQ’S

Hardware Support:

• AMD Support
  • Supported Versions of ROCm
  • Installation
    • Selecting Accelerators Using HIP_VISIBLE_DEVICES
  • Docker
  • Example Usage
• Apple Silicon Support
  • What is supported
  • Experimental Support
    • Testing
      • Example Resnet-18 Using MPS On Mac M1 Pro
      • Conda Example
• TorchServe on linux aarch64 - Experimental
  • Installation
  • Optimizations
  • Example
  • To Dos
• Running TorchServe with NVIDIA MPS
  • Benchmarks
    • G4 Instance
    • P3 instance
  • Summary
• Intel Extension for PyTorch

Service APIs:

• TorchServe gRPC API
  • Python client example for gRPC APIs
  • GRPC Server Side Streaming
• Inference API
  • API Description
  • Health check API
  • Predictions API
    • curl Example
    • curl Example
  • Explanations API
    • curl example
  • KServe Inference API
    • curl example
  • KServe Explanations API
    • curl example
• Management API
  • Register a model
    • Encrypted model serving
  • Scale workers
  • Describe model
  • Unregister a model
  • List models
  • API Description
  • Set Default Version
  • Token Authorization API
• Metrics API
  • Prometheus server
  • Grafana
• TorchServe REST API
• Workflow Inference API
  • Predictions API
    • curl Example
• Management API
  • Register a workflow
  • Describe workflow
  • Unregister a workflow
  • List workflows

Developer APIs:

• ts package
  • Subpackages
    • ts.metrics package
      • Submodules
      • ts.metrics.dimension module
      • ts.metrics.metric module
      • ts.metrics.metric_collector module
      • ts.metrics.metric_encoder module
      • ts.metrics.metrics_store module
      • ts.metrics.process_memory_metric module
      • ts.metrics.system_metrics module
      • ts.metrics.unit module
      • Module contents
    • ts.model_service package
      • Submodules
      • ts.model_service.model_service module
      • Module contents
    • ts.protocol package
      • Submodules
      • ts.protocol.otf_message_handler module
      • Module contents
    • ts.torch_handler package
      • Subpackages
        • ts.torch_handler.request_envelope package
          • Submodules
          • ts.torch_handler.request_envelope.base module
          • ts.torch_handler.request_envelope.body module
          • ts.torch_handler.request_envelope.json module
          • ts.torch_handler.request_envelope.kserve module
          • ts.torch_handler.request_envelope.kservev2 module
          • Module contents
        • ts.torch_handler.unit_tests package
          • Subpackages
            • ts.torch_handler.unit_tests.models package
              • Submodules
              • ts.torch_handler.unit_tests.models.base_model module
              • Module contents
            • ts.torch_handler.unit_tests.test_utils package
              • Submodules
              • ts.torch_handler.unit_tests.test_utils.mock_context module
              • Module contents
          • Submodules
          • ts.torch_handler.unit_tests.test_base_handler module
          • ts.torch_handler.unit_tests.test_envelopes module
          • ts.torch_handler.unit_tests.test_image_classifier module
          • ts.torch_handler.unit_tests.test_image_segmenter module
          • ts.torch_handler.unit_tests.test_mnist_kf module
          • ts.torch_handler.unit_tests.test_object_detector module
          • Module contents
      • Submodules
      • ts.torch_handler.base_handler module
      • ts.torch_handler.contractions module
      • ts.torch_handler.densenet_handler module
      • ts.torch_handler.image_classifier module
      • ts.torch_handler.image_segmenter module
      • ts.torch_handler.object_detector module
      • ts.torch_handler.text_classifier module
      • ts.torch_handler.text_handler module
      • ts.torch_handler.vision_handler module
      • Module contents
    • ts.utils package
      • Submodules
      • ts.utils.timeit_decorator module
      • ts.utils.util module
      • Module contents
  • Submodules
  • ts.arg_parser module
  • ts.context module
  • ts.model_loader module
  • ts.model_server module
  • ts.model_service_worker module
  • ts.service module
  • ts.version module
  • Module contents
• ts.torch_handler.request_envelope package
  • Submodules
  • ts.torch_handler.request_envelope.base module
  • ts.torch_handler.request_envelope.body module
  • ts.torch_handler.request_envelope.json module
  • ts.torch_handler.request_envelope.kserve module
  • ts.torch_handler.request_envelope.kservev2 module
  • Module contents
• ts.torch_handler package
  • Subpackages
    • ts.torch_handler.request_envelope package
      • Submodules
      • ts.torch_handler.request_envelope.base module
      • ts.torch_handler.request_envelope.body module
      • ts.torch_handler.request_envelope.json module
      • ts.torch_handler.request_envelope.kserve module
      • ts.torch_handler.request_envelope.kservev2 module
      • Module contents
    • ts.torch_handler.unit_tests package
      • Subpackages
        • ts.torch_handler.unit_tests.models package
          • Submodules
          • ts.torch_handler.unit_tests.models.base_model module
          • Module contents
        • ts.torch_handler.unit_tests.test_utils package
          • Submodules
          • ts.torch_handler.unit_tests.test_utils.mock_context module
          • Module contents
      • Submodules
      • ts.torch_handler.unit_tests.test_base_handler module
      • ts.torch_handler.unit_tests.test_envelopes module
      • ts.torch_handler.unit_tests.test_image_classifier module
      • ts.torch_handler.unit_tests.test_image_segmenter module
      • ts.torch_handler.unit_tests.test_mnist_kf module
      • ts.torch_handler.unit_tests.test_object_detector module
      • Module contents
  • Submodules
  • ts.torch_handler.base_handler module
  • ts.torch_handler.contractions module
  • ts.torch_handler.densenet_handler module
  • ts.torch_handler.image_classifier module
  • ts.torch_handler.image_segmenter module
  • ts.torch_handler.object_detector module
  • ts.torch_handler.text_classifier module
  • ts.torch_handler.text_handler module
  • ts.torch_handler.vision_handler module
  • Module contents
• ts.metrics package
  • Submodules
  • ts.metrics.dimension module
  • ts.metrics.metric module
  • ts.metrics.metric_collector module
  • ts.metrics.metric_encoder module
  • ts.metrics.metrics_store module
  • ts.metrics.process_memory_metric module
  • ts.metrics.system_metrics module
  • ts.metrics.unit module
  • Module contents
• ts.model_service package
  • Submodules
  • ts.model_service.model_service module
  • Module contents
• ts.protocol package
  • Submodules
  • ts.protocol.otf_message_handler module
  • Module contents
• serve
  • run_circleci_tests module
  • setup module
  • test package
    • Submodules
    • test.regression_tests module
    • Module contents
  • torchserve_sanity module
  • ts package
    • Subpackages
      • ts.metrics package
        • Submodules
        • ts.metrics.dimension module
        • ts.metrics.metric module
        • ts.metrics.metric_collector module
        • ts.metrics.metric_encoder module
        • ts.metrics.metrics_store module
        • ts.metrics.process_memory_metric module
        • ts.metrics.system_metrics module
        • ts.metrics.unit module
        • Module contents
      • ts.model_service package
        • Submodules
        • ts.model_service.model_service module
        • Module contents
      • ts.protocol package
        • Submodules
        • ts.protocol.otf_message_handler module
        • Module contents
      • ts.torch_handler package
        • Subpackages
          • ts.torch_handler.request_envelope package
            • Submodules
            • ts.torch_handler.request_envelope.base module
            • ts.torch_handler.request_envelope.body module
            • ts.torch_handler.request_envelope.json module
            • ts.torch_handler.request_envelope.kserve module
            • ts.torch_handler.request_envelope.kservev2 module
            • Module contents
          • ts.torch_handler.unit_tests package
            • Subpackages
              • ts.torch_handler.unit_tests.models package
                • Submodules
                • ts.torch_handler.unit_tests.models.base_model module
                • Module contents
              • ts.torch_handler.unit_tests.test_utils package
                • Submodules
                • ts.torch_handler.unit_tests.test_utils.mock_context module
                • Module contents
            • Submodules
            • ts.torch_handler.unit_tests.test_base_handler module
            • ts.torch_handler.unit_tests.test_envelopes module
            • ts.torch_handler.unit_tests.test_image_classifier module
            • ts.torch_handler.unit_tests.test_image_segmenter module
            • ts.torch_handler.unit_tests.test_mnist_kf module
            • ts.torch_handler.unit_tests.test_object_detector module
            • Module contents
        • Submodules
        • ts.torch_handler.base_handler module
        • ts.torch_handler.contractions module
        • ts.torch_handler.densenet_handler module
        • ts.torch_handler.image_classifier module
        • ts.torch_handler.image_segmenter module
        • ts.torch_handler.object_detector module
        • ts.torch_handler.text_classifier module
        • ts.torch_handler.text_handler module
        • ts.torch_handler.vision_handler module
        • Module contents
      • ts.utils package
        • Submodules
        • ts.utils.timeit_decorator module
        • ts.utils.util module
        • Module contents
    • Submodules
    • ts.arg_parser module
    • ts.context module
    • ts.model_loader module
    • ts.model_server module
    • ts.model_service_worker module
    • ts.service module
    • ts.version module
    • Module contents
  • ts_scripts package
    • Submodules
    • ts_scripts.api_utils module
    • ts_scripts.backend_utils module
    • ts_scripts.frontend_utils module
    • ts_scripts.install_dependencies module
    • ts_scripts.install_from_src module
    • ts_scripts.marsgen module
    • ts_scripts.modelarchiver_utils module
    • ts_scripts.print_env_info module
    • ts_scripts.regression_utils module
    • ts_scripts.sanity_utils module
    • ts_scripts.shell_utils module
    • ts_scripts.torchserve_grpc_client module
    • ts_scripts.tsutils module
    • ts_scripts.utils module
    • ts_scripts.validate_model_on_gpu module
    • ts_scripts.workflow_archiver_utils module
    • Module contents