How to Run Kimi K3 Locally: GGUF Sizes, VRAM & Setup
Unsloth lists Kimi K3 GGUF files from 594 GB to 1,560 GB. Their memory figure is combined RAM plus VRAM, starting at 610 GB.
The short answer
Unsloth's hardware table does not list a single consumer GPU. The smallest row they print is 610 GB of combined RAM plus VRAM for the Dynamic 1-bit S quant, whose file is 594 GB. Their Q8 row is 1.6 TB combined, against a 1,560 GB file. A rental price, a Mac Studio price, and a cluster price were not on that page, so this page does not invent them.
The same guide says a Mac Studio connected to a 128 GB RAM device is one machine they name, and also says the 1-bit quant needs at least 610 GB of RAM. Both sentences are theirs. This page does not turn that into a claim that a 128 GB Mac Studio is enough.
Kimi K3 at a glance
These figures are the Moonshot model card's summary, checked 2026-11-03. The card says 2.8T total parameters, not a rounded 3T.
| Field | Published value |
|---|---|
| Architecture | Mixture-of-Experts |
| Total parameters | 2.8T |
| Activated parameters | 104B |
| Layers | 93, of which 1 is dense |
| Experts | 896 routed, 16 selected per token, plus 2 shared |
| Context length | 1,048,576 tokens |
| Modalities | Text and image |
| Shipped quantization | MXFP4 weights / MXFP8 activations |
Data checked as of 2026-11-03.
GGUF quantization sizes
The sizes below are the GB column on Unsloth's quantization table. Their prose also says UD-Q4_K_XL needs 1.56 TB of RAM or VRAM, while the table's file-size cell is 1,510 GB. Both are quoted. They are not averaged.
| Quant | File size (GB) | What that row says about quality |
|---|---|---|
| UD-IQ1_S | 594 | top-1 agreement 78.875% ± 0.107; perplexity 2.5789 |
| UD-Q2_K_XL | 861.3 | top-1 agreement 90.390% ± 0.077; perplexity 1.7359 |
| UD-Q4_K_XL | 1510 | perplexity 1.4579; top-1 agreement cell was blank on the Unsloth table |
| UD-Q8_K_XL | 1560 | perplexity 1.4581; Unsloth calls this lossless versus the MXFP4 safetensors |
Data checked as of 2026-11-03.
VRAM and RAM requirements
Unsloth's hardware table is labeled as total memory: RAM plus VRAM, or unified memory. It does not split VRAM, system RAM, and disk by context length. Cells they did not publish stay marked that way. 6block separately says an 8×H100 node (640 GB of HBM) cannot hold their IQ1_S files by itself, and that those files need about 515 GB or 550 GB of RAM plus VRAM combined.
| Quant | Context split | VRAM only | Memory figure | Disk |
|---|---|---|---|---|
| Dynamic 1-bit S (UD-IQ1_S) | not published as a separate context length | not published separately | 610 GB combined RAM + VRAM, or unified memory | 594 GB file size in the quantization table |
| Dynamic 1-bit M | not published as a separate context length | not published separately | 665 GB combined | 648.9 GB file size (UD-IQ1_M) |
| Dynamic 2-bit XXS | not published as a separate context length | not published separately | 726 GB combined | 711.1 GB file size (UD-IQ2_XXS) |
| Dynamic 2-bit XL (UD-Q2_K_XL) | not published as a separate context length | not published separately | 880 GB combined | 861.3 GB file size |
| Q8 lossless (UD-Q8_K_XL) | not published as a separate context length | not published separately | 1.6 TB combined | 1,560 GB file size |
Data checked as of 2026-11-03.
Which runtime
Ollama, vLLM, SGLang, and LM Studio were not given a Kimi K3 support statement on the Unsloth guide or the Moonshot card checked 2026-11-03. Those rows are omitted.
| Runtime | GGUF | More than one GPU | Who it is aimed at |
|---|---|---|---|
| Unsloth Studio | Yes. The guide says it searches and downloads GGUF and safetensor models. | The guide says it detects multi-GPU setups and offloads to RAM. | Desktop on macOS, Windows, and Linux, using the published install commands. |
| llama.cpp (Unsloth fork, branch kimi-k3-fullsize-vision) | Yes. The documented command loads unsloth/Kimi-K3-GGUF:UD-IQ1_S. | CUDA is a build flag. The same page says set GGML_CUDA=OFF for CPU, and Metal is on by default for Apple. | People who want the CLI commands Unsloth printed. |
Data checked as of 2026-11-03.
Step by step
The commands are the ones Unsloth printed. This site has not run them.
- Build their llama.cpp fork. The guide checks out branch kimi-k3-fullsize-vision and builds llama-cli, llama-mtmd-cli, llama-server, and llama-gguf-split. Set GGML_CUDA=OFF if you are not using CUDA.
- Official weights are the moonshotai/Kimi-K3 repository. Unsloth's documented GGUF download is
hf download unsloth/Kimi-K3-GGUF --local-dir unsloth/Kimi-K3-GGUF --include "*UD-IQ1_S*". - Pick a quant from the table above. They recommend UD-IQ1_S when you want the size they call the balance point, and they say that choice needs at least 610 GB of RAM.
- Start the CLI they show:
./llama.cpp/llama-cli -hf unsloth/Kimi-K3-GGUF:UD-IQ1_S --temp 1.0 --top-p 0.95. A separate manual-path command points llama-cli at the first UD-IQ1_S shard and an mmproj file. - Unsloth's write-up uses a sample question after launch. Repeating their screenshot is not a result from this site.
- For a desktop client, their other path is
unsloth studio, then open the local URL they print,http://127.0.0.1:8888.
Speed expectations
Reported by Unsloth, not measured here: if the model fits, about 20 tokens per second of generation on B200s, and more than 120 tokens per second of throughput. They do not print the batch size or the context length next to that sentence.
Reported by 6block on an 8×H100 plus 2 TB RAM node, same llama.cpp build: 11.5 tok/s for UD-IQ1_S and 6.8 tok/s for their IQ1_S-XS. They say those two are comparable only inside that write-up.
Troubleshooting
- If RAM plus VRAM is below the file size, Unsloth says the model can still load by using disk, and that it will be much slower.
- 6block says 640 GB of HBM on 8×H100 is not enough for their IQ1_S files alone. They place MoE layers in CPU memory with
--n-cpu-moe 93. - GrEarl says their 528 GiB IQ1_S build does not load in a released llama.cpp, and points at pull request 26185. Unsloth tells you to build a different fork. Those are two different build requirements, not one.
- A KV-cache table by context length was not on the Unsloth page. Nothing is filled in for it here.
- If a download stalls, Unsloth points to Hugging Face Hub and XET debugging rather than a command invented on this page.
FAQ
Is Kimi K3 downloadable?
Moonshot publishes the weights at huggingface.co/moonshotai/Kimi-K3 and says they are released under the Kimi K3 License. Unsloth publishes GGUF files in a separate repository. Read the license on the model card before you download.
How much VRAM do you need?
The Unsloth guide checked on 2026-11-03 does not publish a VRAM-only number. It publishes combined RAM plus VRAM, or unified memory. The smallest dynamic quant in that hardware table is 610 GB for Dynamic 1-bit S, against a 594 GB file.
Is there a free way to run it?
The weights are posted for download. That is not the same as a free hosted API. The Unsloth page does not describe a no-cost hosted endpoint, and the GGUF still needs the combined memory in their table.
Does it need Nvidia chips?
Not according to Unsloth's build instructions. They say to set GGML_CUDA=OFF when you do not have a GPU, and that Metal is on by default for Apple. CUDA is only the flag they use when building with an Nvidia GPU.
Publishing with a local model?
A local model can draft a blog post or a marketing paragraph that still reads stiff. The AI humanizer returns another wording of that draft. The content checker only estimates writing patterns. Neither one changes the model, the token price, or a third-party detector score.
Compare the published prices and GGUF sizes with DeepSeek-V4-Pro-0813
Sources and last updated
Data checked as of 2026-11-03.
- Moonshot Kimi-K3 model card
- Unsloth: How to run Kimi K3 locally
- Unsloth Kimi-K3-GGUF
- Kimi API context caching prices
- Kimi K3 pricing article
- DeepSeek-V4-Pro model card
- DeepSeek API models and pricing
- 6block Kimi-K3-GGUF
- 6block DeepSeek-V4-Pro-0813-GGUF
- GrEarl Kimi-K3 IQ1_S note
Update log
- : First version. Figures are copied from the linked pages. A 4 November 2026 weight release was not stated on those pages, so it is not included.