In the vllm vs sglang question, there is one difference that matters if your mixture-of-experts model is bigger than your graphics card: SGLang has a way to run it that vLLM’s docs do not describe. SGLang is an open-source inference server. Its pull request 42557, “paged experts”, was opened on 5 October and merged on 19 October 2026. It keeps a fixed number of experts per layer on the GPU, holds all of them in pinned RAM, and copies in only the ones a token needs.
This vllm vs sglang post covers the speed on a 16 GB card, where llama.cpp still wins, and what vLLM’s docs say. My verdict: for one user on a home GPU, paged experts are a real option for models that do not fit. For short prompts or several users at once, llama.cpp is still the better call. Throughput numbers come from the pull requests I link; the ratios are my arithmetic.

What paged experts do that offload flags do not
A mixture-of-experts model has many experts per layer, and each token is routed to only a few. Expert weights are most of the model, so they decide whether it fits.
The old answer is to push weights out of VRAM. SGLang’s PR says its --cpu-offload-gb “streams every offloaded weight to the GPU on every forward pass, whatever the routing.” llama.cpp does it another way: --n-cpu-moe N keeps the experts of some layers in system RAM and computes them on the CPU.
Paged experts do neither. Consecutive tokens reuse most of their experts, so the GPU keeps K slots per layer and the normal kernel runs. The PR says results are unchanged: bit-identical logprobs for bf16 and FP8.
K is a dial: a small K frees memory, a large K buys speed. SGLang sizes it from your memory budget, or you set --paged-experts-num-resident. The switch is --enable-paged-experts.
SGLang vs llama.cpp numbers on a 16 GB card
The PR’s machine is an RTX 5070 Ti with 16 GB, 62 GB of RAM and PCIe 5.0 x16, running Qwen3-30B-A3B. Quantisation is byte-matched: GPTQ int4 against llama.cpp IQ4_XS, block FP8 against Q8_0. The author searched llama.cpp’s --n-cpu-moe in steps of one and counted a setting only if it fit in VRAM. Those llama.cpp runs used --n-cpu-moe and friends, not --moe-cache-mib.
Single-stream decode, tokens per second:
- int4: paged experts 205.7, llama.cpp at its best setting 194.4. That is 5.8 percent more, with K = 92 of 128 experts on the card.
- FP8: paged experts 87.7, llama.cpp 45.2, which is 1.94 times as fast. Here K = 40 of 128, so 31 percent of the experts sit on the card.
- SGLang’s own
--cpu-offload-gb: 16.6 at int4 and 2.9 at FP8. Paging is 12 and 30 times faster than that flag.
The int4 gap is small. The follow-up, pull request 45787, reports the same model at K = 92 on the same card as an end-to-end output rate: 190.3 tok/s at batch size 1, against 205.7 decode in the first PR. That is a 7.5 or 8.1 percent gap depending on which you divide by, more than the 5.8 percent lead. Call int4 close. FP8 is not close.
Where llama.cpp still wins
Short prompts. Median time to first token at int4 on the 16 GB card: 89 ms for paged experts, 68 ms for llama.cpp with --n-cpu-moe 3 and a q8 KV cache. That is 21 ms, or 31 percent, slower. A prompt touches nearly every expert, so paging copies the missing ones once: about 4 GB at K = 92, about 80 ms over PCIe 5.
Long prompts flip it. For a 2,048-token prompt, paged experts take 267 ms against 344 ms at int4, and 443 ms against 984 ms at FP8. The copy is a fixed cost.
Several users at once. With 4 concurrent requests on the 16 GB card, int4 favours paging: 387, 374 and 376 tok/s over three runs against 312. FP8 is 84 against 55. On the 8 GB RTX 4070 laptop it reverses hard: 11 to 12 tok/s for paging against 41 for llama.cpp, because K is only 26 of 128 and each step routes to more experts than fit. The same cliff shows up inside one engine: FP8 on the 16 GB card gives 84 tok/s at K = 40 and 31 at K = 24.
My rule from the tables: few resident experts plus several requests, and paging falls behind. One stream, long prompts, a fast PCIe link and most of the experts living in host RAM, and it wins: that is the FP8 case, 1.94 times. When the model mostly fits, as at int4, the engines sit within 5.8 percent. The laptop on one request is a wash, 30.6 to 32.9 against 23.0 to 32.4.
Which formats you can load
The first PR covered three expert formats: bf16/fp16, block FP8 and GPTQ. PR 45787, opened 22 October and merged 25 October, widens that to twelve. Its table marks nine as new, among them AWQ int4, MXFP4 for gpt-oss, ModelOpt NVFP4 and W8A8 FP8.
The PR’s gpt-oss-20b MXFP4 run, with 16 of 32 experts resident on the same card, gives 107.7 tok/s at batch size 1 and 0.960 on GSM8K against 0.975 unpaged.
The first PR’s scope still holds: one GPU, no tensor or expert parallelism, and a host store that must fit in pinnable RAM. The PR says WSL2 caps pinned memory at about half of system RAM.
vLLM vs SGLang on offload: what the docs say
I read vLLM’s engine arguments page on 13 November 2026. The --cpu-offload-gb entry reads: “this argument can be seen as a virtual way to increase the GPU memory size.” Then: “this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass.”
That is the behaviour SGLang’s PR describes for its own flag, in vLLM’s words, so on offload the vllm vs sglang gap is the routing-aware paging, not the flag. vLLM also lists --offload-group-size, --offload-num-in-group and --offload-prefetch-step, which offload whole layers and prefetch them ahead. The same section has --cpu-offload-params and --offload-params, which pick tensors by name, for example “mlp.experts.w2_weight”, so vLLM can offload expert tensors, but I found no flag that keeps experts resident by routing. vLLM’s own pull request 56177, opened 10 September and still open, not merged, adds a shared GPU expert pool for NVFP4 Marlin experts only. Its body measures Qwen3.8-Flash-Next NVFP4 on one RTX PRO 6000 Blackwell Max-Q at a 48 GiB budget: 64.32 tok/s against 13.15 for the fastest main configuration tested, 4.89 times, vLLM’s own pool against vLLM’s own best offload setup, not against SGLang.
llama.cpp’s expert cache came first
PR 29887, “llama : add a GPU cache for MoE experts kept in host memory”, merged on 7 October, twelve days before SGLang’s paged experts. You enable it with --moe-cache-mib; the PR found 10 percent of the expert size a good setting. PR 30966, merged 7 November, added --moe-cache-cpu-share, which runs missed experts on the CPU, plus scripts/moe-cache-tune.py to find both settings. Release v1.0.0, published 13 November, lists it and adds the moe_cache_cpu_share field.
The PR’s sample run, on a different model and a much larger card, shows why the script exists: 24.2 tok/s with a 6,464 MiB cache and no CPU share, 66.4 with 19,392 MiB and a share of 0.5, and hit rates from 61.5 to 93.5 percent. A cache setting can move speed almost threefold, so any engine-against-engine benchmark is partly a test of who tuned harder.
On 11 November I wrote about the other end of this problem: a 284-billion-parameter model squeezed into 64 GB of RAM with no GPU.
What I would do with this, vLLM or SGLang
With 12 to 16 GB of VRAM, 62 GB of RAM, native Linux and an MoE checkpoint whose experts mostly live in RAM, try paged experts at a few K values and time your own first token. For a chat window with short prompts, or a model that mostly fits, llama.cpp with --moe-cache-mib and its tuning script is the safer first try. For four users at once, neither engine’s offload path is where I would spend an afternoon.
Both pull requests carry their full method, so anyone can re-run them.