LoRA Single Device Finetuning

This recipe supports finetuning on next-token prediction tasks using parameter efficient fine-tuning techniques (PEFT) such as Low Rank Adaptation (LoRA) and Quantized Low Rank Adaptation (QLoRA). These techniques significantly reduce memory consumption during training whilst still maintaining competitive performance.

We provide configs which you can get up and running quickly. Here is an example with Llama 3.1 8B:

Note

You may need to be granted access to the Llama model you’re interested in. See here for details on accessing gated repositories.

# download the model
tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
--output-dir /tmp/Meta-Llama-3.1-8B-Instruct \
--ignore-patterns "original/consolidated.00.pth"

# run the recipe
tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device

You can customize this recipe through the torchtune CLI. For example, when fine-tuning with LoRA, you can adjust the layers which LoRA are applied to:

tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device \
model.lora_attn_modules=“[q_proj,k_proj,v_proj]” \
model.apply_lora_to_mlp=True \
model.lora_rank=64 \
model.lora_alpha=128

For a deeper understanding of the different levers you can pull when using this recipe, see our documentation for the different PEFT training paradigms we support:

• Low Rank Adaptation (LoRA)

• Quantized Low Rank Adaptation (QLoRA)

• Weight-Decomposed Low-Rank Adaptation (DoRA)

Many of our other memory optimization features can be used in this recipe. You can learn more about all of our memory optimization features in our memory optimization overview.

Interested in seeing this recipe in action? Check out some of our tutorials to show off how it can be used:

• Finetuning Llama2 with LoRA

• Finetuning Llama2 with QLoRA

• Fine-tuning Llama3 with Chat Data

• Meta Llama3 in torchtune

• Fine-Tune Your First LLM