ai text natural rewriter, ai content checkerAI Humanize Check

How to Run Kimi K3 Locally: GGUF Sizes, VRAM & Setup

Unsloth lists Kimi K3 GGUF files from 594 GB to 1,560 GB. Their memory figure is combined RAM plus VRAM, starting at 610 GB.

The short answer

Unsloth's hardware table does not list a single consumer GPU. The smallest row they print is 610 GB of combined RAM plus VRAM for the Dynamic 1-bit S quant, whose file is 594 GB. Their Q8 row is 1.6 TB combined, against a 1,560 GB file. A rental price, a Mac Studio price, and a cluster price were not on that page, so this page does not invent them.

The same guide says a Mac Studio connected to a 128 GB RAM device is one machine they name, and also says the 1-bit quant needs at least 610 GB of RAM. Both sentences are theirs. This page does not turn that into a claim that a 128 GB Mac Studio is enough.

Kimi K3 at a glance

These figures are the Moonshot model card's summary, checked 2026-11-03. The card says 2.8T total parameters, not a rounded 3T.

Model card summary
FieldPublished value
ArchitectureMixture-of-Experts
Total parameters2.8T
Activated parameters104B
Layers93, of which 1 is dense
Experts896 routed, 16 selected per token, plus 2 shared
Context length1,048,576 tokens
ModalitiesText and image
Shipped quantizationMXFP4 weights / MXFP8 activations

Data checked as of 2026-11-03.

GGUF quantization sizes

The sizes below are the GB column on Unsloth's quantization table. Their prose also says UD-Q4_K_XL needs 1.56 TB of RAM or VRAM, while the table's file-size cell is 1,510 GB. Both are quoted. They are not averaged.

Bar chart of Unsloth file sizes: UD-IQ1_S 594 GB, UD-Q2_K_XL 861.3 GB, UD-Q4_K_XL 1,510 GB, UD-Q8_K_XL 1,560 GB.
Unsloth dynamic GGUF file sizes
QuantFile size (GB)What that row says about quality
UD-IQ1_S594top-1 agreement 78.875% ± 0.107; perplexity 2.5789
UD-Q2_K_XL861.3top-1 agreement 90.390% ± 0.077; perplexity 1.7359
UD-Q4_K_XL1510perplexity 1.4579; top-1 agreement cell was blank on the Unsloth table
UD-Q8_K_XL1560perplexity 1.4581; Unsloth calls this lossless versus the MXFP4 safetensors

Data checked as of 2026-11-03.

VRAM and RAM requirements

Unsloth's hardware table is labeled as total memory: RAM plus VRAM, or unified memory. It does not split VRAM, system RAM, and disk by context length. Cells they did not publish stay marked that way. 6block separately says an 8×H100 node (640 GB of HBM) cannot hold their IQ1_S files by itself, and that those files need about 515 GB or 550 GB of RAM plus VRAM combined.

Combined memory, as published
QuantContext splitVRAM onlyMemory figureDisk
Dynamic 1-bit S (UD-IQ1_S)not published as a separate context lengthnot published separately610 GB combined RAM + VRAM, or unified memory594 GB file size in the quantization table
Dynamic 1-bit Mnot published as a separate context lengthnot published separately665 GB combined648.9 GB file size (UD-IQ1_M)
Dynamic 2-bit XXSnot published as a separate context lengthnot published separately726 GB combined711.1 GB file size (UD-IQ2_XXS)
Dynamic 2-bit XL (UD-Q2_K_XL)not published as a separate context lengthnot published separately880 GB combined861.3 GB file size
Q8 lossless (UD-Q8_K_XL)not published as a separate context lengthnot published separately1.6 TB combined1,560 GB file size

Data checked as of 2026-11-03.

Which runtime

Ollama, vLLM, SGLang, and LM Studio were not given a Kimi K3 support statement on the Unsloth guide or the Moonshot card checked 2026-11-03. Those rows are omitted.

Runtimes named in the Unsloth guide
RuntimeGGUFMore than one GPUWho it is aimed at
Unsloth StudioYes. The guide says it searches and downloads GGUF and safetensor models.The guide says it detects multi-GPU setups and offloads to RAM.Desktop on macOS, Windows, and Linux, using the published install commands.
llama.cpp (Unsloth fork, branch kimi-k3-fullsize-vision)Yes. The documented command loads unsloth/Kimi-K3-GGUF:UD-IQ1_S.CUDA is a build flag. The same page says set GGML_CUDA=OFF for CPU, and Metal is on by default for Apple.People who want the CLI commands Unsloth printed.

Data checked as of 2026-11-03.

Step by step

The commands are the ones Unsloth printed. This site has not run them.

  1. Build their llama.cpp fork. The guide checks out branch kimi-k3-fullsize-vision and builds llama-cli, llama-mtmd-cli, llama-server, and llama-gguf-split. Set GGML_CUDA=OFF if you are not using CUDA.
  2. Official weights are the moonshotai/Kimi-K3 repository. Unsloth's documented GGUF download is hf download unsloth/Kimi-K3-GGUF --local-dir unsloth/Kimi-K3-GGUF --include "*UD-IQ1_S*".
  3. Pick a quant from the table above. They recommend UD-IQ1_S when you want the size they call the balance point, and they say that choice needs at least 610 GB of RAM.
  4. Start the CLI they show: ./llama.cpp/llama-cli -hf unsloth/Kimi-K3-GGUF:UD-IQ1_S --temp 1.0 --top-p 0.95. A separate manual-path command points llama-cli at the first UD-IQ1_S shard and an mmproj file.
  5. Unsloth's write-up uses a sample question after launch. Repeating their screenshot is not a result from this site.
  6. For a desktop client, their other path is unsloth studio, then open the local URL they print, http://127.0.0.1:8888.

Speed expectations

Reported by Unsloth, not measured here: if the model fits, about 20 tokens per second of generation on B200s, and more than 120 tokens per second of throughput. They do not print the batch size or the context length next to that sentence.

Reported by 6block on an 8×H100 plus 2 TB RAM node, same llama.cpp build: 11.5 tok/s for UD-IQ1_S and 6.8 tok/s for their IQ1_S-XS. They say those two are comparable only inside that write-up.

Troubleshooting

  • If RAM plus VRAM is below the file size, Unsloth says the model can still load by using disk, and that it will be much slower.
  • 6block says 640 GB of HBM on 8×H100 is not enough for their IQ1_S files alone. They place MoE layers in CPU memory with --n-cpu-moe 93.
  • GrEarl says their 528 GiB IQ1_S build does not load in a released llama.cpp, and points at pull request 26185. Unsloth tells you to build a different fork. Those are two different build requirements, not one.
  • A KV-cache table by context length was not on the Unsloth page. Nothing is filled in for it here.
  • If a download stalls, Unsloth points to Hugging Face Hub and XET debugging rather than a command invented on this page.

FAQ

Is Kimi K3 downloadable?

Moonshot publishes the weights at huggingface.co/moonshotai/Kimi-K3 and says they are released under the Kimi K3 License. Unsloth publishes GGUF files in a separate repository. Read the license on the model card before you download.

How much VRAM do you need?

The Unsloth guide checked on 2026-11-03 does not publish a VRAM-only number. It publishes combined RAM plus VRAM, or unified memory. The smallest dynamic quant in that hardware table is 610 GB for Dynamic 1-bit S, against a 594 GB file.

Is there a free way to run it?

The weights are posted for download. That is not the same as a free hosted API. The Unsloth page does not describe a no-cost hosted endpoint, and the GGUF still needs the combined memory in their table.

Does it need Nvidia chips?

Not according to Unsloth's build instructions. They say to set GGML_CUDA=OFF when you do not have a GPU, and that Metal is on by default for Apple. CUDA is only the flag they use when building with an Nvidia GPU.

Publishing with a local model?

A local model can draft a blog post or a marketing paragraph that still reads stiff. The AI humanizer returns another wording of that draft. The content checker only estimates writing patterns. Neither one changes the model, the token price, or a third-party detector score.

Rewrite a draft with the AI humanizer

Compare the published prices and GGUF sizes with DeepSeek-V4-Pro-0813

Sources and last updated

Data checked as of 2026-11-03.

Update log

  • : First version. Figures are copied from the linked pages. A 4 November 2026 weight release was not stated on those pages, so it is not included.