Blog

An efficient decoding Grouped-Query Attention with low-precision KV cache Introduction Generative AI has taken the…

Recently, we demonstrated how FSDP and selective activation checkpointing can be used to achieve 57% MFU…

This post was contributed by Intel AI in partnership with the PyTorch Foundation. In 2017,…

TLDR: We show promising results of up to a 1.46x speedup with <2% drop in accuracy on float32…

Speculative decoding is an optimization technique for inference that makes educated guesses about future tokens…

1.0 Summary We present an optimized Triton FP8 GEMM (General Matrix-Matrix Multiply) kernel TK-GEMM, which…

We are excited to announce the release of ExecuTorch alpha, focused on deploying large language models…

We are excited to announce the release of PyTorch® 2.3 (release note)! PyTorch 2.3 offers…

1.0 Summary We show that by implementing column-major scheduling to improve data locality, we can…

In this blog, we demonstrate the scalability of FSDP with a pre-training exemplar, a 7B…

This post is the fourth part of a multi-series blog focused on how to accelerate…

Overview PyTorch is a Python-based framework for developing deep learning models. It is one of…

TL;DR Leveraging a first principles approach, we showcase a step by step process undertaken to…

We demonstrate how to finetune a 7B parameter model on a typical consumer GPU (NVIDIA…

Multi-model endpoints (MMEs) are a powerful feature of Amazon SageMaker designed to simplify the deployment and operation…