LoRA Single Device Finetuning¶
This recipe supports finetuning on next-token prediction tasks using parameter efficient fine-tuning techniques (PEFT) such as Low Rank Adaptation (LoRA) and Quantized Low Rank Adaptation (QLoRA). These techniques significantly reduce memory consumption during training whilst still maintaining competitive performance.
We provide configs which you can get up and running quickly. Here is an example with llama 3.1 8B:
Note
You may need to be granted access to the Llama model you’re interested in. See here for details on accessing gated repositories.
# download the model
tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
--output-dir /tmp/Meta-Llama-3.1-8B-Instruct \
--ignore-patterns "original/consolidated.00.pth"
# run the recipe
tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device
You can customize this recipe through the torchtune CLI. For example, when fine-tuning with LoRA, you can adjust the layers which LoRA are applied to:
tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device \
model.lora_attn_modules=“[q_proj,k_proj,v_proj]” \
model.apply_lora_to_mlp=True \
model.lora_rank=64 \
model.lora_alpha=128
For a deeper understanding of the different levers you can pull when using this recipe, see our documentation for the different PEFT training paradigms we support:
Many of our other memory optimization features can be used in this recipe. You can learn more about all of our memory optimization features in our memory optimization overview.
Interested in seeing this recipe in action? Check out some of our tutorials to show off how it can be used:
