Blog

If you’ve ever trained a large AI model and had it fail with an error…

TL;DR In a joint effort between PyTorch and Nebius, we enabled training DeepSeek-V3 Mixture-of-Experts models…

We are excited to announce the release of PyTorch® 2.11 (release notes)!   The PyTorch 2.11…

Overview We are excited to introduce the highlights of Intel® Core™ Ultra Series 3 processors…

Introduction Over the past year, large language models have rapidly expanded in both scale and…

In this blog post, we present the kernel design of Generalized Dot-Product Attention (GDPA), a…

TL;DR Open source voice models are proliferating, but there's no unified native inference platform for…

TL;DR We recently demonstrated a +30.2% training speedup for Llama4 Scout with equivalent convergence to…

We're excited to announce that PyTorch will have a strong presence at NVIDIA GTC 2026,…

Summary Recently, the PyTorch team released KernelAgent, an open agentic system achieving 100% correctness across…

TL;DR: On Hopper and Blackwell GPUs, FlexAttention now has a FlashAttention-4 backend. We added support…

The world of AI is expanding beyond the cloud, reaching devices that fit in the…

In our previous Quantization-Aware Training (QAT) blog, we introduced the initial QAT flow in TorchAO…

Overview This blog walks through two crucial DeepSpeed updates: (1) a PyTorch-identical backward API that…

Introduction As introduced in a previous blog post, Helion is a high-level DSL that empowers…

We’re excited to share that PyTorch now leverages Pyrefly to power type checking across our…

Summary In this post, we discuss how we optimized the Mamba-2 State-Space Dual (SSD) module…

What is an accumulator in an accelerator's GEMM engine and why does it matter? GPUs…

Why Choose PyTorch for Recommendation System PyTorch has emerged as the de facto framework in…

Recently, the PyTorch team released Helion, a new domain-specific and PyTorch-based language to make the…