DeepInfer is the inference engine DeepSeek says powers its own production services, and on October 16 the company put it on GitHub under the Apache-2.0 licence (deepseek-ai/DeepInfer). An inference engine is the program that loads a model’s weights onto GPUs and turns prompts into tokens, the same job vLLM and SGLang do. This one runs a single model, DeepSeek-V4.1-Flash, on two kinds of hardware: NVIDIA’s newest SM100 GPUs and Huawei’s Ascend 950 chips.
In this post I go over what the DeepInfer inference engine actually contains, what its own README says is missing, and which two features I would copy into any pipeline. My verdict for a startup: it is a great thing to read and a bad thing to deploy this month. Keep vLLM, steal the ideas.
Here is what I did to write this. I cloned the repo at commit 774ce24, read the README, the eval folder and the structured-output example, and followed the batch-invariant flag through the code. I did not build or run anything, because the engine needs a GPU I do not have, and every claim below about speed belongs to someone else.
What do you get in the DeepInfer inference engine?
The clone has about 54,900 lines of Python, C++ and CUDA in csrc/, 16 test files and a vendor/ folder. The kernels it leans on, FlashMLA, DeepGEMM, DeepEP and TileKernels, are DeepSeek projects you can already clone separately.
The feature list is short and specific. Asynchronous scheduling, which overlaps CPU batch planning with GPU work, is on by default. Speculative decoding is switched on with DEEPINFER_ENABLE_MTP_VARLEN=1, and the default draft length is 5 tokens. There is expert parallelism for the mixture-of-experts layers, then batch-invariant inference, deterministic sampling and structured outputs. The last three matter most if you build products on a model.
Setup is a build from source: Linux, GCC 13 or newer, Python 3.12, CUDA 13, then ./build.sh. The default checkpoint, DeepSeek-V4.1-Flash-1017, downloads on first run. The only example is offline inference: you hand it a batch of prompts and wait.
What is missing from the DeepInfer inference engine?
The README says it itself, under a heading I like for its honesty: “Required for a complete inference service (not included in this release).” Three items follow. A KV cache system, which the README does not define and I read as the layer that keeps attention state between requests. Prefill/decode disaggregation, which splits the two phases of generation onto different machines. And load balancing across data-parallel replicas.
Mettons you run a chat product. If I read that first item right, every follow-up message pays for the full conversation again. Without a load balancer, you write the thing that spreads traffic across copies yourself. None of that is exotic, and vLLM and SGLang both have their own versions. DeepSeek has not released its version.
Then the hardware. SM100 is data-centre Blackwell, not something a startup has lying around. A community pull request, #92 from DarkraiHL, is open and not merged as I write this. It ports the engine to Hopper-class H20-3e cards (141 GB each), and the author says 4 GPUs are enough because the expert weights stay in FP4. The author reports a first token in 2.15 seconds for a 128K-token prompt on 8 cards and 3.64 seconds on 4, plus 3,793 total output tokens per second on 8 cards with 64 concurrent requests. Those are the PR author’s numbers, on their own hardware, and I have not checked them.
The same PR says “a comparison with SGLang on the same hardware will follow in a comment.” When I looked, the PR had zero comments. I also searched GitHub for any independent DeepInfer-versus-vLLM or SGLang benchmark and found none. The README has throughput charts, but they are PNG images and I did not transcribe them, so I will not quote them.
One more sentence in the README is worth reading twice: DeepInfer “was originally built primarily on vLLM.” So this is a heavily modified descendant of the engine you probably already run, The open question is what DeepSeek changed.
Batch-invariant inference: the idea worth stealing
Ask a model the same question twice at temperature 0 and you can still get two different answers. The usual cause is batching: your request runs alongside other requests, the GPU adds numbers in a different order depending on who else is in the batch, and floating-point addition is not associative. Tiny differences compound and the output drifts.
DeepInfer has a switch for this. The README says DEEPINFER_BATCH_INVARIANT=1 keeps “model results identical across different batch compositions,” NVIDIA only. I followed the flag into deepinfer/worker/runtime_config.py and it does what you hope: when it is set, the code calls use_deterministic_algorithms(True) on both DeepGEMM and TileKernels. On Ascend it is not supported yet, and the code even asserts the flag is off with a note in Chinese saying to remove that once batch-invariant support is complete.
The catch is hiding in the README. Deterministic sampling needs DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed. Read that middle one again. Variable-length speculative decoding, the thing that makes generation faster, has to be turned off. So reproducible output costs you a speed feature, on purpose, and the README tells you plainly.
Why do I care? Me, I think of a regression test for an LLM feature. Trellis builds tools for Amazon sellers, and a listing audit is the kind of thing a seller runs again next week to see if the score moved. If the same listing can come back with a different score because the batch was busier on Tuesday, you cannot tell a real change from noise. I do not know if this would fix it on the hosted APIs we use today, since batch invariance is something the server has to provide. If we ever ran our own servers, this is the first thing I would test: 40 saved listings, run them 10 times each, count how many outputs differ.
Structured outputs: the boring feature that saves you a retry loop
The other feature I would copy is the constrained decoding. You can force generation to follow a regex, a JSON schema or an EBNF grammar by passing so_args to SamplingParams. The example file has three cases, and I like how mean the test is. The prompts, written in Chinese, ask for a free story or a poem and say not to output JSON or code. The schema forces the answer anyway, and the script asserts the run ended with a normal stop and an end-of-sequence token.
That is the right test, because a constraint that only works when the prompt asks nicely is a suggestion. A model that returns “Sure! Here is the JSON:” followed by JSON, once in two hundred calls, breaks a pipeline at 3 a.m. Grammar-constrained decoding makes that class of bug impossible instead of rare. vLLM and SGLang both have structured outputs too, so this is not new. What I like is seeing it next to determinism in the same engine.
Should a startup run the DeepInfer inference engine?
No, and I will say why in three facts. It runs one model. It runs on hardware most startups do not have. And the pieces that turn an engine into a service are the pieces not shipped. If you already planned to serve DeepSeek-V4.1-Flash on a rack of Blackwell cards, you have a team for this.
For everyone else, the cheap move is the right one. Rent tokens through a gateway and put a hard cap on the spend, the way I did in my LiteLLM budget post. If you do self-host, stay on vLLM, which supports far more models and cards, and read the DeepInfer source for how DeepSeek handles batch invariance and scheduling. If you sell to US government customers, check their rules on Chinese-origin models before anything else. I did not chase the latest order, so do not take that sentence as legal advice.
The repo sat at 10,003 stars when I checked, 4 days after it was created, and a star is a click, not a code review.
The number I am waiting for is boring: the SGLang comparison promised in PR #92. If it appears, and someone repeats it on a second machine, I will run the same prompts through vLLM and report the gap here.
