Granite-4.0-H-Small
Granite-4.0-H-Small is an open-weights language model from IBM, released October 2, 2025. It has 32B parameters (mixture-of-experts, 8.8B active). The context window is 128K tokens.
We’re launching Granite 4, the next generation of IBM language models. Granite 4.0 features a new hybrid Mamba/transformer architecture that greatly reduces memory requirements without sacrificing performance.
Check it against your own PC →
Can you run Granite-4.0-H-Small locally?
The smallest common GPU it runs on is 6 GB of VRAM (with 32 GB of RAM): IQ4_XS, GPU + experts in RAM, roughly 11 tokens per second. Estimates assume 32 GB of DDR5 RAM, a 16K context and llama.cpp-style GGUF quantization. Enter your exact hardware in the live app.
| GPU memory | Verdict | Best quant | VRAM | RAM | tok/s |
|---|---|---|---|---|---|
| 6 GB VRAM | Runs · GPU + CPU experts | IQ4_XS (good) | 5.2 GB | 12.8 GB | ~11 |
| 8 GB VRAM | Runs · GPU + CPU experts | Q4_K_M (very good) | 7.2 GB | 13.4 GB | ~12 |
| 12 GB VRAM | Runs · GPU + CPU experts | Q5_K_M (excellent) | 11.2 GB | 12.6 GB | ~12 |
| 16 GB VRAM | Runs · GPU + CPU experts | Q6_K (≈ lossless) | 15.2 GB | 12.1 GB | ~13 |
| 24 GB VRAM | Runs on GPU | Q4_K_M (very good) | 20.6 GB | – | ~95 |
| 32 GB VRAM | Runs on GPU | Q6_K (≈ lossless) | 27.3 GB | – | ~130 |
Ready-made quantizations on Hugging Face: ibm-granite/granite-4.0-h-small-GGUF · unsloth/granite-4.0-h-small-GGUF · bartowski/ibm-granite_granite-4.0-h-small-GGUF · mradermacher/granite-4.0-h-small-i1-GGUF · taronaeo/Granite-4.0-H-Small-BE-GGUF · ibm-granite/granite-4.0-h-small-FP8
Benchmark results
Epoch AI has not published results for Granite-4.0-H-Small yet. New models are usually evaluated within days to weeks.
Links
Announcement · Hugging Face: ibm-granite/granite-4.0-h-small
More from IBM
granite switch 4.1 8b · granite switch 4.1 30b · Granite 4.1 30B · Granite 4.1 8B · Granite-4.0-H-Tiny · Granite-4.0-H-Micro · Granite-Docling