Blog

Getting distributed training jobs to run on huge clusters is hard!  This is especially true…

Introduction Normalization methods (LayerNorm/RMSNorm) are foundational in deep learning and are used to normalize values…

Introduction TorchInductor currently supports three autotuning backends for matrix multiplications: Triton, CUTLASS (C++), and cuBLAS.…

If you’ve ever trained a large AI model and had it fail with an error…

TL;DR In a joint effort between PyTorch and Nebius, we enabled training DeepSeek-V3 Mixture-of-Experts models…

We are excited to announce the release of PyTorch® 2.11 (release notes)!   The PyTorch 2.11…

Overview We are excited to introduce the highlights of Intel® Core™ Ultra Series 3 processors…

Introduction Over the past year, large language models have rapidly expanded in both scale and…

In this blog post, we present the kernel design of Generalized Dot-Product Attention (GDPA), a…

TL;DR Open source voice models are proliferating, but there's no unified native inference platform for…

TL;DR We recently demonstrated a +30.2% training speedup for Llama4 Scout with equivalent convergence to…

We're excited to announce that PyTorch will have a strong presence at NVIDIA GTC 2026,…

Summary Recently, the PyTorch team released KernelAgent, an open agentic system achieving 100% correctness across…

TL;DR: On Hopper and Blackwell GPUs, FlexAttention now has a FlashAttention-4 backend. We added support…