MPT-7B
MPT-7B is an open-weights language model from MosaicML, released May 5, 2023. It has 7B parameters (dense). On the Epoch Capabilities Index it scores 73.7, rank 244 of 256 rated models.
Check it against your own PC →
Lab
MosaicML
Released
May 5, 2023
Weights
Open unrestricted
Parameters
7B dense
Context
–
Price / M tokens
–
Epoch ECI
73.7 #244
Can you run MPT-7B locally?
The smallest common GPU it runs on is 6 GB of VRAM (with 32 GB of RAM): Q6_K, split GPU / CPU, roughly 8 tokens per second. Estimates assume 32 GB of DDR5 RAM, a 16K context and llama.cpp-style GGUF quantization. Enter your exact hardware in the live app.
| GPU memory | Verdict | Best quant | VRAM | RAM | tok/s |
|---|---|---|---|---|---|
| 6 GB VRAM | Runs, slowly | Q6_K (≈ lossless) | 5.2 GB | 3.3 GB | ~8 |
| 8 GB VRAM | Runs on GPU | Q4_K_M (very good) | 7.0 GB | – | ~37 |
| 12 GB VRAM | Runs on GPU | Q6_K (≈ lossless) | 8.5 GB | – | ~36 |
| 16 GB VRAM | Runs on GPU | Q6_K (≈ lossless) | 8.5 GB | – | ~50 |
| 24 GB VRAM | Runs on GPU | Q6_K (≈ lossless) | 8.5 GB | – | ~89 |
| 32 GB VRAM | Runs on GPU | Q6_K (≈ lossless) | 8.5 GB | – | ~160 |
Benchmark results
| Benchmark (Epoch AI) | Score |
|---|---|
| MMLU | 7.7% |
Links
Models with a similar score
LLaMA-7B · DeepSeek Coder 33B · vicuna-13b-v1.1 · Gemma 2B · CodeQwen1.5-7B · chatglm2-6b · Falcon-7B · StarCoder 2 7B