dshseek
中文

seek://guides/deepinfer

DeepInfer: DeepSeek's Production Inference Framework, Explained for DSH Users

Oct 23, 2026Intermediate← All guides

TL;DR

What DeepSeek's 2026-10-16 DeepInfer release contains, how to build it and run offline inference on DeepSeek-V4.1-Flash-1017, which of its properties matter for agents (batch-invariant inference, deterministic sampling, structured outputs), and the honest gap: no HTTP server ships, so DeepSeek Harness cannot use it as a backend until a serving entrypoint lands.

Key points

  • Per its README, DeepInfer is the high-performance inference framework that powers DeepSeek's production services, originally built primarily on vLLM; this release provides offline inference for one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
  • Build is one exported variable plus one script: `export DEEPINFER_ACCELERATOR=nvidia` (or `ascend`, kept exported at run time) inside an empty Python 3.12 environment, then `./build.sh`; prerequisites are Linux, GCC 13 or newer, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend)
  • `python examples/offline_inference.py` downloads the default checkpoint `deepseek-ai/DeepSeek-V4.1-Flash-1017` into the Hugging Face cache on first run; `DEEPINFER_HF_MODEL_DIR` points it at a local copy instead
  • Three properties matter for agents: batch-invariant inference (`DEEPINFER_BATCH_INVARIANT=1`, NVIDIA only) keeps results identical across batch compositions; deterministic sampling adds `DEEPINFER_ENABLE_MTP_VARLEN=0` and a fixed `seq_seed`; structured outputs constrain generation to a regex, JSON schema or EBNF grammar via `so_args`
  • No HTTP or OpenAI-compatible server ships: per examples/README.md every example is an offline, benchmark, sweep or perplexity script, and the README lists a KV cache system, VE/prefill/decode disaggregation and load balancing as required for a complete service but not included in this release
  • For DSH this means there is nothing to point a custom provider at today; when a serving entrypoint arrives, the DSH side is the ordinary custom-provider flow from the configure-models guide

On 2026-10-16, DeepSeek published DeepInfer under Apache-2.0 — per its README, the high-performance inference framework that powers DeepSeek’s large-scale production services, originally built primarily on vLLM. For DeepSeek Harness users the hook is the checkpoint: DeepInfer’s only supported model is DeepSeek-V4.1-Flash, and DSH v0.2.1-rc.2 added support for its DeepSeek-V4.1-Flash-1017 checkpoint one day after the repository appeared. This guide walks the README and examples/README.md (fetched at write time) and adds the DSH angle: what ships, how to run it, which properties agents benefit from — and what is still missing before any of it can back DSH. (Background on the harness itself: What is DeepSeek Harness?.)

What ships in the release

The README scopes the release to offline inference with the core features needed for serving at scale. In one pass:

  • Kernels from DeepSeek’s other open-source repos — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect — plus custom TileLang kernels in-tree
  • Parallelism: Attention DP / balanced CP, and MoE EP with EPLB
  • DSpark: decoding accelerated with draft tokens, verification scheduled by confidence estimates (DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE caps draft length, default 5)
  • SWA bounded replay: decoder prefill reduced to a bounded suffix (DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 in the default V4.1 settings)
  • Asynchronous scheduling: CPU batch scheduling overlapped with device execution, enabled by default
  • Batch-invariant inference, deterministic sampling, structured outputs — the three this guide returns to below

The environment the README requires:

NVIDIA Ascend
Hardware SM100 GPUs Ascend 950 NPUs
Toolkit CUDA 13 CANN 9.2
DEEPINFER_ACCELERATOR nvidia ascend
Common Linux, GCC ≥ 13, Python 3.12 Linux, GCC ≥ 13, Python 3.12

The performance section reports prefill measured on 4 SM100 GPUs and decode on 64 — each within one scale-up domain — with reproduction scripts under examples/sweep/.

Build it and run offline inference

The README’s build is one exported variable plus one script, inside an empty Python 3.12 environment:

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
python3.12 -m venv .venv && source .venv/bin/activate  # an empty Python 3.12 env, per the README
export DEEPINFER_ACCELERATOR=nvidia   # or: ascend — keep it exported where you run DeepInfer
./build.sh
python examples/offline_inference.py

The first run downloads the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache; to run from a local copy instead, point DEEPINFER_HF_MODEL_DIR at it. Beyond the entry example, examples/README.md documents benchmark.py for throughput/latency points and the sweep scripts that reproduce the README’s pareto charts — and that is the full extent of the surface: every entry point is an offline-inference, benchmark, sweep or perplexity script.

Three properties agent builders should note

Batch-invariant inference

DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) keeps model results identical across different batch compositions — the same prompt produces the same answer whether it shares a batch with one request or one hundred. For anyone running agent fleets behind one engine, that is the property that makes a session’s behavior independent of coincident traffic.

Deterministic sampling

Add DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed in SamplingParams, and identical input tokens with identical sampling parameters produce identical outputs. That is reproducibility at the run level: the same task, prompt and seed replay token for token — the basis a repeatable agent eval needs. The README’s own variables encode a trade: determinism requires MTP off, so DSpark’s draft-token acceleration and deterministic sampling are not available in the same run.

Structured outputs

so_args on SamplingParams constrains generation to a regex, JSON schema or EBNF grammar before decoding begins — payloads a tool call or a state hand-off can rely on without a repair pass. The exact schema_type identifiers live in examples/_structured_output.py.

What is missing before it can back DSH

The README names three things a complete inference service requires that this release does not include: a KV cache system; VE/prefill/decode disaggregation (most required engine functionality is already implemented, it notes); and load balancing across data-parallel replicas for both prefill and decode. The fourth gap is the one DSH feels directly: no HTTP entrypoint ships, so there is no OpenAI-compatible endpoint to register.

Concretely, today’s options run in three lanes:

  • Use DSH against the official DeepSeek API — the supported path, unchanged by this release
  • Run DeepInfer offline on supported hardware to test the engine, measure it, or build on the batch-invariant/deterministic/structured-output properties directly
  • Watch the repository for serving components; when an endpoint lands, the DSH side is the ordinary custom-provider flow from the configure-models guide — base URL, protocol, credential, one model — nothing DSH-specific

Where this fits with the rest of the site

Official references