Blog

Blog

Deep Dive on the Hopper TMA Unit for FP8 GEMMs

Abstract The Hopper (H100) GPU architecture, billed as the “first truly asynchronous GPU”, includes a…

Adnan Hoque, Less Wright, Chih-Chieh YangJuly 22, 2024

Blog

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large…

Jay Shah and Ganesh Bikshandi, Colfax Research, Ying Zhang, Meta, Vijay Thakkar and Pradeep Ramani, NVIDIA, Tri Dao, TogetherAI and Princeton UniversityJuly 11, 2024

Blog

Learn how to develop Android applications with ExecuTorch and Llama models

This blog is courtesy of the PyTorch team at Arm. More details can be found here.…

ArmJuly 10, 2024

Blog

Accelerated PyTorch inference with torch.compile on AWS Graviton processors

Summary Originally PyTorch, used an eager mode where each PyTorch operation that forms the model…

Sunita NadampalliJuly 9, 2024

Announcements Blog

Announcing Hacker Cup AI Track at NeurIPS 2024

The PyTorch team in partnership with Meta Hacker Cup, and Microsoft Research, are excited to…

PyTorch FoundationJuly 3, 2024

Announcements

Powering the AI Revolution: The PyTorch Documentary

Now live: The official PyTorch Documentary! This film unveils the authentic narrative of PyTorch’s inception, attributing…

The PyTorch FoundationJune 25, 2024

Blog

Training MoEs at Scale with PyTorch

Over the past year, Mixture of Experts (MoE) models have surged in popularity, fueled by…

Brian Chu, Mihir Patel, Less Wright, Vitaliy Chiley, Evan Racah, Wanchao Liang, Iris Zhang, Andrew GuJune 23, 2024

Announcements

🎉 PyTorch Docathon H1 2024 Wrap-up 🎉

We are thrilled to announce the successful completion of the H1 2024 PyTorch Docathon! The…

PyTorch FoundationJune 20, 2024

Blog

Accelerating Neural Network Training with Semi-Structured (2:4) Sparsity

Over the past year, we’ve added support for semi-structured (2:4) sparsity into PyTorch. With just…

Jesse Cai, Daniel Haziza, Supriya RaoJune 20, 2024

Blog

Reducing Model Checkpointing Times by Over 10x with PyTorch Distributed Asynchronous Checkpointing

Summary: With PyTorch distributed’s new asynchronous checkpointing feature, developed with feedback from IBM, we show how…

Meta: Lucas Pasqualin, Less Wright, Iris Zhang (PyTorch), Chien-Chin Huang; IBM Research: Swaminathan Sundararaman, Saransh Gupta, Raghu GantiJune 12, 2024

Announcements

PyTorch Foundation Welcomes New Executive Director

The PyTorch Foundation is excited to welcome Matt White, our new executive director. The PyTorch…

PyTorch FoundationJune 11, 2024

Blog

INT4 Decoding GQA CUDA Optimizations for LLM Inference

An efficient decoding Grouped-Query Attention with low-precision KV cache Introduction Generative AI has taken the…

Sarunya Pumma, Jongsoo Park, Jianyu Huang, Amy Yang, Jaewon Lee, Daniel Haziza, Grigory Sizov, Jeremy Reizenstein, Jeff Johnson, Ying ZhangJune 6, 2024

Announcements

Ready, Set, Contribute: PyTorch Docathon Kickoff H1 2024

The PyTorch Docathon is now live! This event is dedicated to enhancing the quality of…

PyTorch FoundationJune 4, 2024

Case Studies

AI Helps Duolingo Personalize Language Learning

Learning a foreign language was probably one of your goals last year. And the year…

PyTorch FoundationMay 25, 2024

Blog

Maximizing Training Throughput Using PyTorch FSDP and Torch.compile

Recently, we demonstrated how FSDP and selective activation checkpointing can be used to achieve 57% MFU…

Team PyTorch at IBM and Team PyTorch at MetaMay 21, 2024

Blog

Achieving Sustainability Goals with PyTorch and Intel AI

This post was contributed by Intel AI in partnership with the PyTorch Foundation. In 2017,…

PyTorch FoundationMay 15, 2024

Blog

Speeding up ViTs using Block Sparsity

TLDR: We show promising results of up to a 1.46x speedup with <2% drop in accuracy on float32…

FAIR at Meta: Mostafa Elhoushi, Sensors and Systems at Meta Reality Labs Research: Syed Shakib Sarwar, Aaryan Kothapalli, Mia Kasperek, Barbara De Salvo, PyTorch at Meta: Christian Puhrsch, Jesse Cai, Joe Isaacson, Quantsight: Andrew James, Pearu Peterson, Nikita VedeneevMay 14, 2024

Community

Introducing depyf: mastering torch.compile with ease

We are thrilled to introduce depyf, a new project to the PyTorch ecosystem designed to help…

Kaichao YouMay 11, 2024

Community

Deep Learning Energy Measurement and Optimization

This post is authored by Jae-Won Chung, a PhD student at the University of Michigan and…

Jae-Won ChungMay 11, 2024

Announcements Community

Enhancing Deep Learning Workflows: PyTorch Ecosystem Tools

Welcome to the thriving PyTorch ecosystem, where a wealth of tools and libraries await, purpose-built…

PyTorch FoundationMay 11, 2024

Deep Dive on the Hopper TMA Unit for FP8 GEMMs

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Learn how to develop Android applications with ExecuTorch and Llama models

Accelerated PyTorch inference with torch.compile on AWS Graviton processors

Announcing Hacker Cup AI Track at NeurIPS 2024

Powering the AI Revolution: The PyTorch Documentary

Training MoEs at Scale with PyTorch

🎉 PyTorch Docathon H1 2024 Wrap-up 🎉

Accelerating Neural Network Training with Semi-Structured (2:4) Sparsity

Reducing Model Checkpointing Times by Over 10x with PyTorch Distributed Asynchronous Checkpointing

PyTorch Foundation Welcomes New Executive Director

INT4 Decoding GQA CUDA Optimizations for LLM Inference

Ready, Set, Contribute: PyTorch Docathon Kickoff H1 2024

AI Helps Duolingo Personalize Language Learning

Maximizing Training Throughput Using PyTorch FSDP and Torch.compile

Achieving Sustainability Goals with PyTorch and Intel AI

Speeding up ViTs using Block Sparsity

Introducing depyf: mastering torch.compile with ease

Deep Learning Energy Measurement and Optimization

Enhancing Deep Learning Workflows: PyTorch Ecosystem Tools

Docs

Tutorials

Resources

Stay in touch for updates, event info, and the latest news