NVIDIA GeForce RTX 3080 12GB

NVIDIA · 12GBGDDR6X · Can run 63 models

Manufacturer	NVIDIA
VRAM	12 GB
Memory Type	GDDR6X
Architecture	Ampere
CUDA Cores	8,960
Tensor Cores	280
Bandwidth	912 GB/s
TDP	350W
MSRP	$799
Released	Jan 11, 2022

AI Notes

The RTX 3080 12GB is a decent option for running local AI models. With 12GB of GDDR6X VRAM, it can handle 7B models at full precision and 13B models with quantization. Its older Ampere architecture is slower per core than Ada Lovelace but still delivers solid inference performance.

Compatible Models

Model	Parameters	Best Quant	VRAM Used	Fit	Est. Speed
Qwen 3 0.6B	600M	Q4_K_M	2.5 GB	Runs	~365 tok/s
Qwen 3.5 0.8B	800M	Q4_K_M	1.5 GB	Runs	~608 tok/s
Gemma 3 1B	1B	Q8_0	2 GB	Runs	~456 tok/s
Llama 3.2 1B	1B	Q8_0	3 GB	Runs	~304 tok/s
DeepSeek R1 1.5B	1.5B	Q8_0	3 GB	Runs	~304 tok/s
SmolLM2 1.7B	1.7B	Q8_0	2.7 GB	Runs	~338 tok/s
Gemma 2 2B	2B	Q8_0	4 GB	Runs	~228 tok/s
Gemma 3n E2B	2B	Q4_K_M	3.3 GB	Runs	~276 tok/s
Gemma 4 E2B	2B	Q4_K_M	4 GB	Runs	~228 tok/s
Qwen 3.5 2B	2B	Q4_K_M	3 GB	Runs	~304 tok/s
Llama 3.2 3B	3B	Q8_0	5 GB	Runs	~182 tok/s
StarCoder2 3B	3B	Q4_K_M	3.5 GB	Runs	~261 tok/s
Phi-3 Mini 3.8B	3.8B	Q8_0	5.8 GB	Runs	~157 tok/s
Phi-4 Mini 3.8B	3.8B	Q4_K_M	4.5 GB	Runs	~203 tok/s
Gemma 3 4B	4B	Q4_K_M	5 GB	Runs	~182 tok/s
Gemma 3n E4B	4B	Q4_K_M	4.5 GB	Runs	~203 tok/s
Gemma 4 E4B	4B	Q4_K_M	6 GB	Runs	~152 tok/s
Qwen 3 4B	4B	Q4_K_M	4.5 GB	Runs	~203 tok/s
Qwen 3.5 4B	4B	Q4_K_M	4.5 GB	Runs	~203 tok/s
Yi 1.5 6B	6B	Q4_K_M	5 GB	Runs	~182 tok/s
Codestral Mamba 7B	7B	Q4_K_M	6.9 GB	Runs	~132 tok/s
DeepSeek R1 7B	7B	Q8_0	9 GB	Runs	~101 tok/s
Falcon 3 7B	7B	Q4_K_M	6.8 GB	Runs	~134 tok/s
InternLM 2.5 7B	7B	Q4_K_M	5.5 GB	Runs	~166 tok/s
Mistral 7B	7B	Q8_0	9 GB	Runs	~101 tok/s
OpenChat 3.5 7B	7B	Q4_K_M	6.9 GB	Runs	~132 tok/s
Qwen 2.5 7B	7B	Q8_0	9 GB	Runs	~101 tok/s
Qwen 2.5 Coder 7B	7B	Q8_0	9 GB	Runs	~101 tok/s
Qwen 2.5 VL 7B	7B	Q4_K_M	7 GB	Runs	~130 tok/s
StarCoder2 7B	7B	Q4_K_M	5.5 GB	Runs	~166 tok/s
WizardLM 2 7B	7B	Q4_K_M	6.9 GB	Runs	~132 tok/s
Aya Expanse 8B	8B	Q4_K_M	6.5 GB	Runs	~140 tok/s
Cogito 8B	8B	Q4_K_M	7.5 GB	Runs	~122 tok/s
DeepSeek R1 8B	8B	Q4_K_M	7.5 GB	Runs	~122 tok/s
Dolphin 3 8B	8B	Q4_K_M	6 GB	Runs	~152 tok/s
Granite 3.3 8B	8B	Q8_0	10 GB	Runs	~91 tok/s
Llama 3.1 8B	8B	Q8_0	10 GB	Runs	~91 tok/s
Nemotron 3 Nano 8B	8B	Q4_K_M	7.5 GB	Runs	~122 tok/s
Nous Hermes 2 8B	8B	Q4_K_M	6 GB	Runs	~152 tok/s
Qwen 3 8B	8B	Q4_K_M	7.5 GB	Runs	~122 tok/s
Qwen 3.5 9B	9B	Q4_K_M	7.5 GB	Runs	~122 tok/s
Yi 1.5 9B	9B	Q4_K_M	6.5 GB	Runs	~140 tok/s
Yi Coder 9B	9B	Q4_K_M	8 GB	Runs	~114 tok/s
Falcon 3 10B	10B	Q4_K_M	8.5 GB	Runs	~107 tok/s
Llama 3.2 Vision 11B	11B	Q4_K_M	8.5 GB	Runs	~107 tok/s
Mistral Nemo 12B	12B	Q4_K_M	9.5 GB	Runs	~96 tok/s
DeepSeek R1 14B	14B	Q4_K_M	9.9 GB	Runs	~92 tok/s
Phi-4 14B	14B	Q4_K_M	9.9 GB	Runs	~92 tok/s
Qwen 2.5 14B	14B	Q4_K_M	9.9 GB	Runs	~92 tok/s
Gemma 2 9B	9B	Q8_0	11 GB	Runs (tight)	~83 tok/s
Gemma 3 12B	12B	Q4_K_M	10.5 GB	Runs (tight)	~87 tok/s
Phi-4 Reasoning 14B	14B	Q4_K_M	11 GB	Runs (tight)	~83 tok/s
Qwen 2.5 Coder 14B	14B	Q4_K_M	12 GB	CPU Offload	~23 tok/s
Qwen 3 14B	14B	Q4_K_M	12 GB	CPU Offload	~23 tok/s
StarCoder2 15B	15B	Q8_0	17 GB	CPU Offload	~16 tok/s
InternLM 2.5 20B	20B	Q4_K_M	12 GB	CPU Offload	~23 tok/s
gpt-oss 20B	21B	MXFP4	15 GB	CPU Offload	~18 tok/s
Codestral 22B	22B	Q4_K_M	14.7 GB	CPU Offload	~19 tok/s
Devstral 24B	24B	Q4_K_M	17 GB	CPU Offload	~16 tok/s
Magistral Small 24B	24B	Q4_K_M	17 GB	CPU Offload	~16 tok/s
Mistral Small 3.1 24B	24B	Q4_K_M	18 GB	CPU Offload	~15 tok/s
Gemma 2 27B	27B	Q4_K_M	17.7 GB	CPU Offload	~16 tok/s
Qwen 3.5 35B A3B	35B	Q4_K_M	12 GB	CPU Offload	~23 tok/s

51 model(s) are too large for this hardware.