TorchServe

⚠️ Notice: Limited Maintenance

This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.

TorchServe is a performant, flexible and easy to use tool for serving PyTorch models in production.

What’s going on in TorchServe?

• High performance Llama 2 deployments with AWS Inferentia2 using TorchServe

• Naver Case Study: Transition From High-Cost GPUs to Intel CPUs and oneAPI powered Software with performance

• Run multiple generative AI models on GPU using Amazon SageMaker multi-model endpoints with TorchServe and save up to 75% in inference costs

• Deploying your Generative AI model in only four steps with Vertex AI and PyTorch

• PyTorch Model Serving on Google Cloud TPUv5

• Monitoring using Datadog

• Torchserve Performance Tuning, Animated Drawings Case-Study

• Walmart Search: Serving Models at a Scale on TorchServe

• Scaling inference on CPU with TorchServe

• TorchServe C++ backend

• TorchServe with Intel® Extension for PyTorch*

• Grokking Intel CPU PyTorch performance from first principles: a TorchServe case study

• Grokking Intel CPU PyTorch performance from first principles( Part 2): a TorchServe case study

• Case Study: Amazon Ads Uses PyTorch and AWS Inferentia to Scale Models for Ads Processing

• Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker

• Using AI to bring children’s drawings to life

• Model Serving in PyTorch

• Evolution of Cresta’s machine learning architecture: Migration to AWS and PyTorch

• Explain Like I’m 5: TorchServe

• How to Serve PyTorch Models with TorchServe

• How to deploy PyTorch models on Vertex AI

• Quantitative Comparison of Serving Platforms

---

TorchServe Quick Start

Topics: Quick Start

Learn how to install TorchServe and serve models.

Running TorchServe

Topics: Running TorchServe

Indepth explanation of how to run TorchServe

Why TorchServe

Topics: Examples

Various TorchServe use cases

TorchServe GenAI Use Cases

Topics: Use Cases

Showcasing GenAI deployment scenarios and use cases

Performance

Topics: Performance, Troubleshooting

Guides and best practices on how to improve perfromance when working with TorchServe

Metrics

Topics: Metrics, Performance, Troubleshooting

Collecting and viewing Torcherve metrics

1

2