Last month I watched a startup burn $12,000 on A100s trying to full fine-tune Llama 3.1 70B. They needed a domain-specific legal model. Three weeks later they switched to LoRA on 4x RTX 4090s, spent $3,200, and hit 94% of the benchmark score. The model ships to production Tuesday.
The Math That Changes Everything
Full fine-tuning 70B at BF16 demands 560 GB VRAM just for optimizer states. That's 7x H100s or 14x A100 80GB. At $4.50/hr per H100, a 3-epoch run costs $8,000-$12,000. LoRA at rank 64 drops trainable parameters to 0.5%. VRAM collapses to 192 GB — four RTX 4090s at $1.20/hr each. Same run: $2,800. QLoRA 4-bit pushes it to 112 GB — two 48 GB cards or even dual 3090s with offloading.
| Method | Trainable Params | VRAM (BF16) | GPU Config | Est. Cost (3 epochs) |
|---|---|---|---|---|
| Full FT | 70B (100%) | 560 GB | 7x H100 80GB | $9,500 |
| LoRA r=64 | 350M (0.5%) | 192 GB | 4x RTX 4090 24GB | $2,800 |
| QLoRA r=64 | 350M (0.5%) | 112 GB | 2x RTX 6000 Ada 48GB | $1,900 |
| QLoRA r=32 | 175M (0.25%) | 80 GB | 2x A100 40GB | $1,400 |
Why LoRA Works Without Quality Loss
LoRA freezes the base model and injects low-rank adapters (A @ B) into attention layers. The intrinsic dimension of LLM adaptation is surprisingly low — research shows 90% of task information lives in a subspace of rank 8-32. At rank 64 you're over-parameterized. I run r=32 alpha=64 with dropout 0.1 and see zero degradation on MMLU, GSM8K, and custom eval sets versus full FT.
