Local Inference
-
DeepInfer Inference Engine: What DeepSeek Left Out
The DeepInfer inference engine is DeepSeek's open-source Apache-2.0 serving code. What the README gives you, what it leaves out, and what I would steal.
-
MoE Expert Offloading: 125B on a 12GB GPU, With Catches
MoE expert offloading lets a 12GB GPU run a 125B model, but only for one user at a time. What Strata's numbers say, and what I'd run instead.