unsloth-lora

Overview

Unsloth optimizes Low-Rank Adaptation (LoRA) by providing 16-bit trainable matrices that allow for efficient fine-tuning without updating all model weights. It supports standard LoRA and Rank-Stabilized LoRA (rsLoRA), utilizing specialized kernels to accelerate training and reduce memory overhead.

When to Use

• When fine-tuning large language models on consumer-grade or limited GPU hardware.
• When aiming to match full fine-tuning performance with significantly lower VRAM usage.
• When specialized scaling (rsLoRA) is required for higher rank stability.

Decision Tree

1. Need to update all weights?
   • Yes: Use [[unsloth-fft]].
   • No: Proceed to LoRA.
2. Using high rank (r > 64)?
   • Yes: Enable use_rslora = True for sqrt(r) scaling.
   • No: Use standard LoRA.
3. Maximizing speed?
   • Yes: Set lora_dropout = 0 to enable internal kernel optimizations.

Workflows