Why 24GB
A 30B-class model quantized to 4bit occupies roughly 17GB. Adding headroom for context puts the practical threshold at 24GB. A 4GB card cannot load the model at all.
Technical background
Why this configuration
Only the factors that actually decide whether a language model runs well locally.
Why bandwidth matters
Generating each token requires reading the full set of weights from memory, so speed tracks memory bandwidth far more closely than raw compute. The RTX 3090 provides 936 GB/s against 96 GB/s on entry-level cards, a factor of 9.75.
What running locally means
No metered billing, no request caps, and no data leaving the machine. This matters when handling material that cannot be sent outside, or when usage is high enough that API cost becomes a fixed expense.
Power design
The graphics card ships with a 280W power limit applied. Because inference is bound by memory bandwidth, lowering the core clock barely affects generation speed while reducing heat, noise and wear.