torchtune Overview

On this page, we’ll walk through an overview of torchtune, including features, key concepts and additional pointers.

What is torchtune?

torchtune is a PyTorch library for easily authoring, fine-tuning and experimenting with LLMs. The library emphasizes 4 key aspects:

• Simplicity and Extensibility. Native-PyTorch, componentized design and easy-to-reuse abstractions

• Correctness. High bar on proving the correctness of components and recipes

• Stability. PyTorch just works. So should torchtune

• Democratizing LLM fine-tuning. Works out-of-the-box on different hardware

torchtune provides:

• Modular native-PyTorch implementations of popular LLMs

• Interoperability with popular model zoos through checkpoint-conversion utilities

• Training recipes for a variety of fine-tuning techniques

• Integration with Hugging Face Datasets for training and EleutherAI’s Eval Harness for evaluation

• Support for distributed training using FSDP

• YAML configs for easily configuring training runs

Excited? To get started, checkout some of our tutorials, including:

• our quickstart guide to finetune your first LLM using torchtune.

• our LoRA tutorial to learn about parameter-efficient finetuning with torchtune.

• our QLoRA tutorial to attain maximal memory efficiency with torchtune.

Key Concepts

As you go through the tutorials and code, there are two concepts which will help you better understand and use torchtune.

Configs. YAML files which help you configure training settings (dataset, model, chekckpoint) and hyperparameters (batch size, learning rate) without modifying code. See the All About Configs deep-dive for more information.

Recipes. Recipes can be thought of as targeted end-to-end pipelines for training and optionally evaluating LLMs. Each recipe implements a training method (eg: full fine-tuning) with a set of meaningful features (eg: FSDP + Activation Checkpointing + Gradient Accumulation + Reduced Precision training) applied to a given model family (eg: Llama2). See the What Are Recipes? deep-dive for more information.

Design Principles

torchtune embodies PyTorch’s design philosophy, especially “usability over everything else”.

Native PyTorch

torchtune is a native-PyTorch library. While we provide integrations with the surrounding ecosystem (eg: Hugging Face Datasets, EleutherAI Eval Harness), all of the core functionality is written in PyTorch.

Simplicity and Extensibility

torchtune is designed to be easy to understand, use and extend.

• Composition over implementation inheritance - layers of inheritance for code re-use makes the code hard to read and extend

• No training frameworks - explicitly outlining the training logic makes it easy to extend for custom use cases

• Code duplication is prefered over unecessary abstractions

• Modular building blocks over monolithic components

Correctness

torchtune provides well-tested components with a high bar on correctness. The library will never be the first to provide a feature, but available features will be thoroughly tested. We provide

• Extensive unit tests to ensure component-level numerical parity with reference implementations

• Checkpoint tests to ensure model-level numerical parity with reference implementations

• Integration tests to ensure recipe-level performance parity with reference implementations on standard benchmarks