MTP and DFlash2 at long context on XPU

I was already deep into making Qwen3.8’s MTP head useful on Arc Pro when DFlash2 landed. It had been public for barely two days. I wanted to know if I could get this brand-new draft model running on four B70s without giving up the 1,048,576-token target pool I had built the server around.

I got it working, including the ugly parts: XPU support, tensor parallel candidate selection, a compact draft KV cache, radix reuse, graph lifecycle, and long-context verification. Then I ran it against MTP all the way to a 512K prompt. DFlash2 had just been released, and I had the real checkpoint serving long-context requests on Arc Pro. Getting that complete path to run was the part I cared about most.

I wanted the comparison to match how I use this model: long coding sessions where the repository, tool history, and working notes keep filling the prompt. A speedup at 8K is not much use to me if it disappears later in the same session.

What I ended up with is workload sensitivity rather than one mode winning everywhere. On four Intel Arc Pro B70s at tensor parallel four, MTP was the more consistent default across the prompt set. DFlash2 could lead when its proposal quality was favorable. Both runs use the same AWQ W4A16 target, exact prompt shapes, warm-up policy, cold radix cache, and streaming-decode method. Their greedy completions are byte-identical at every matched shape.

It took much more than two server flags. I found an MTP head built from uninitialized packed tensors. I found an eight-query verify kernel with a severe shape-dependent slowdown over the same KV. I added a native XPU multi-step backend for the MTP draft. For DFlash2, I pinned the source, built a physical sliding-window KV ring, kept graph capture on the target only, and stopped unsupported requests before they reached the worker. Some fixes did nothing. A few made the server slower.

This is independent work I did on my own setup. The DFlash2 checkpoint and mechanism come from Inco. The SGLang XPU integration, memory accounting, lifecycle fixes, and Arc Pro measurements are mine.

Part I: The setup and the comparison method

Running it

This is the image I used:

docker pull rahulunair/sglang-xpu:qwen3.8-27b-20260819

The published repository digest is sha256:12a3ad504d0524dc090bdc7370dd8d369d6b0a3c3c520ea9b8107d21923653ec. After pulling, this command prints the immutable repository reference resolved on the machine:

docker image inspect rahulunair/sglang-xpu:qwen3.8-27b-20260819 \
  --format '{{index .RepoDigests 0}}'

The AWQ target is ulkaa/Qwen3.8-27B-AWQ-INT4, using its 1m revision. DFlash2 adds incoai/Qwen3.8-27B-DFlash2 as the draft checkpoint. The complete copy-paste Docker invocation is in the Qwen3.8-27B Docker Hub recipe. It calls python -m sglang.launch_server directly and does not depend on a wrapper script.

Download both model trees on the host. Pinning the target to its 1m revision is part of reproducing the capacity result:

hf download ulkaa/Qwen3.8-27B-AWQ-INT4 --revision 1m \
  --local-dir "$PWD/models/Qwen3.8-27B-AWQ-INT4-1m"
hf download incoai/Qwen3.8-27B-DFlash2 \
  --local-dir "$PWD/models/Qwen3.8-27B-DFlash2"

The target pool settings are:

--context-length 1048576
--max-total-tokens 1048576
--max-mamba-cache-size 40
--max-running-requests 64

MTP uses the auxiliary head already stored in the target checkpoint:

--speculative-algorithm EAGLE \
--speculative-num-steps 7 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 8

DFlash2 uses one external five-layer draft checkpoint:

--speculative-algorithm DFLASH \
--speculative-draft-model-path /draft \
--speculative-num-draft-tokens 8 \
--speculative-dflash-block-size 8 \
--speculative-draft-window-size 2048

Here is the complete DFlash2 command I used. The render group expression lets the container open the Intel device nodes without assuming one host-specific group id:

docker run -d --name qwen38-dflash2 --restart unless-stopped \
  --device=/dev/dri -v /dev/dri:/dev/dri \
  --group-add video --group-add "$(getent group render | cut -d: -f3)" \
  --ipc=host --shm-size=64g \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 30000:30000 \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  -v "$PWD/models/Qwen3.8-27B-AWQ-INT4-1m:/model:ro" \
  -v "$PWD/models/Qwen3.8-27B-DFlash2:/draft:ro" \
  rahulunair/sglang-xpu:qwen3.8-27b-20260819 \
  python -m sglang.launch_server \
    --model-path /model --served-model-name Qwen3.8-27B \
    --device xpu --tp-size 4 --trust-remote-code --language-only \
    --context-length 1048576 --max-total-tokens 1048576 \
    --max-mamba-cache-size 40 --max-running-requests 64 \
    --chunked-prefill-size 4096 --mem-fraction-static 0.85 --page-size 64 \
    --attention-backend intel_xpu --disable-custom-all-reduce \
    --reasoning-parser qwen3-thinking --tool-call-parser qwen3_coder \
    --strip-thinking-cache --enable-strict-thinking \
    --watchdog-timeout 1800 --skip-server-warmup --enable-cache-report \
    --cuda-graph-config '{"decode":{"backend":"full","bs":[1,2,4,8]},"prefill":{"backend":"disabled"}}' \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path /draft \
    --speculative-num-draft-tokens 8 \
    --speculative-dflash-block-size 8 \
    --speculative-draft-window-size 2048 \
    --host 0.0.0.0 --port 30000

For MTP, remove the /draft mount and replace the five DFlash2 flags at the end with the four MTP flags shown above. No external MTP checkpoint is needed.

The image supplies the XPU defaults I measured: decode INT8 up to M=8, symmetric all-reduce up to 131,072 bytes, single-query speculative verify, target decode graphs for batch sizes [1,2,4,8], and eager DFlash2 draft execution. Radix prefix caching stays on.

Use /health for process readiness. The first ordinary request at a new shape may compile kernels and needs a long client timeout. A cold /health_generate probe became part of the last release failure described later, so I left it out of the public recipe.

What I mean by decode behavior

I evaluated streaming decode after the first token, separately from aggregate output throughput and time to first token. Those quantities answer different questions, especially for long prompts and concurrent requests. The discussion below reports only qualitative comparisons, but it keeps that distinction so a per-user streaming observation is never presented as server throughput.

For the final long-context sweep I used:

four Arc Pro B70 cards, TP4;
AWQ W4A16 target, YaRN factor four, exact 1,048,576-token target pool;
Intel XPU attention, page size 64, chunked prefill 4,096;
target decode graph buckets [1,2,4,8], prefill graphs disabled;
radix enabled in the product but flushed for each cold benchmark cell;
concurrency one, one unreported warm-up, one reported exact-shape prompt;
greedy generation, with output text retained for coherence comparison.

One sample per long-context shape is enough to expose implementation failures, but not enough to claim a general latency distribution. I used a broader prompt set for the shorter-context comparison and keep its conclusion qualitative.

Part II: What I had enabled

The loop both modes use

Before changing the XPU code, I needed a clean picture of what each worker was doing. MTP and DFlash2 both end at target verification, but they arrive there in very different ways.

Ordinary autoregressive decoding runs the target once for each next token. Speculative decoding adds a cheaper proposal path:

The draft proposes a short continuation.
The target scores the proposed causal block in one verification pass.
The runtime accepts the longest valid prefix and commits the target-selected boundary token.
Draft and target state advance to the next step.

The target remains responsible for verification. A broken draft can waste work while generation still looks coherent because every proposal is rejected. That behavior hid the first MTP defect.

MTP and DFlash2 share the target verification path but produce proposals differently. Qwen’s MTP mode runs an in-checkpoint auxiliary layer repeatedly. With seven draft steps it proposes up to eight tokens. DFlash2 runs a separate five-layer sliding-attention backbone once for the block, obtains the target LM-head top-16 candidates, and walks a learned predecessor-successor lattice to choose a path.

mode	SGLang algorithm	proposal mechanism	draft width I used
MTP	`EAGLE`	seven sequential forwards through the in-checkpoint MTP head	8
DFlash2	`DFLASH`	one five-layer block draft plus learned candidate selector	8

Classic DFlash uses the same DFLASH algorithm name but a different model class and selector. The release checks DFlash2DraftModel explicitly so a classic DFlash tree cannot be published under the DFlash2 label.

Speculation changes how many target calls are needed; it does not let the draft commit an unverified token.

The accepted-length metric used by SGLang includes the verified boundary token. An average accepted length of 4 therefore means that one target verification committed about four output tokens, not four correct draft guesses plus another unreported token. That convention matters when dividing cost by useful work.

Why the target output stayed the same

Consider a draft block [d1, d2, d3, d4]. The target evaluates all four rows with a causal mask. Row one sees the committed prefix, row two also sees d1, and so on. Suppose the target’s greedy choices are [d1, d2, x, ...]. The runtime commits d1, d2, and the target token x; d3 and d4 never enter the target KV cache. The next iteration begins after x.

This is why the method can change speed without changing greedy output. Draft tokens are proposals, and the target selects every committed token. I ran these tests at temperature zero, so equality means exact token equality at each position. Sampling requires rejection sampling against target and draft probabilities. SGLang has that machinery too, but I did not measure it here.

What Qwen’s MTP head does

The Qwen3.8 checkpoint contains one multi-token prediction module. SGLang loads it through Qwen3_5ForCausalLMMTP, while EAGLEWorkerV2 owns the speculative loop. The names can be confusing: EAGLE is the serving algorithm selected at launch, and Qwen’s MTP module is the draft network that this worker repeatedly calls.

For one draft position, the module receives two vectors:

the embedding of the token entering the draft step, shape [1, 5120];
the target hidden state at the same boundary, shape [1, 5120].

It applies RMS normalization to each vector independently, concatenates them into [1, 10240], and projects the result back to [1, 5120] with mtp.fc. That fused representation passes through one auxiliary Qwen decoder layer and the target vocabulary head. Top-k one selects the next proposed token.

The worker repeats this dependency seven times. Each call consumes the hidden state produced by the previous draft call, so the seven proposals are sequential. The target then verifies eight positions: the already available boundary position plus seven new proposals. That is why the command uses seven steps and eight draft tokens.

The auxiliary module remains BF16 even though the target transformer is AWQ W4A16. Its parameter names must therefore be excluded from compressed-tensors quantization. Missing that exclusion caused the silent all-zero draft failure described in Part III.

What arrived with DFlash2

The external DFlash2 checkpoint declares DFlash2DraftModel. This is the configuration I used:

component	Qwen3.8-27B DFlash2 value
target feature layers	5, 19, 33, 47, 61
target hidden width	5,120
concatenated feature width	25,600
draft backbone	five sliding-attention layers
sliding window	2,048 tokens
block size	8 positions
selector candidates	top 16 per position
selector rank	256
local convolution	grouped, kernel size 2

During target prefill, SGLang captures the hidden rows from those five target layers. DFlash2 concatenates the rows, projects [5 × 5120] back to width 5,120, and materializes them into the draft model’s private KV storage. The draft checkpoint has neither its own token embedding nor its own LM head. It reuses the target embedding on input and the target vocabulary head when it needs candidate logits.

One decode call lays out eight draft positions. Position zero holds the current verified boundary token. The remaining positions start as mask tokens. The five-layer backbone processes this block with 2,048-token sliding attention. Each layer also applies a learned two-tap grouped convolution around its attention and MLP path. A position mixes its current representation with the immediately preceding position; at the block boundary that preceding value is the final verified state. This local dependency gives later masked positions a cheap signal about the emerging suffix without making eight full sequential draft-model calls.

The target LM head produces candidate logits for each position. On TP4, each rank first selects its local top 16 vocabulary entries. SGLang gathers the 64 rank-local candidates and performs a global top 16 selection, preserving the same candidate set a non-sharded vocabulary head would expose.

The part that makes DFlash2 different

DFlash2 does not independently choose the highest-logit token in every row. Its selector scores transitions between adjacent candidate sets. The research line is described in the DFlash paper, and the DFlash2 selector and convolution changes are explained in Inco’s DFlash2 article. Using Inco’s notation, a transition from candidate a to candidate b at position t has the form:

S_t(a, b) = U_t(b) + dot(A(a) * H(h_t), B(b))

U_t(b) is the ordinary unary score for candidate b. A(a) and B(b) are learned predecessor and successor codebook rows. H(h_t) projects the draft hidden state, and the elementwise product scales the predecessor representation. The interaction lives in rank 256 rather than the 5,120-wide model space.

For top 16, every adjacent pair contributes a 16 by 16 score matrix. Those matrices are computed in parallel. A short sequential walk then chooses the best successor at each position, conditioned on the candidate already chosen. The selector improves path consistency, but it never authorizes a token. The target still verifies the resulting chain and rejects the invalid suffix.

The XPU port uses torch.topk when FlashInfer’s radix top-k operator is absent. That fallback is correct and is the path in this image. A shape-matched XPU top-k kernel is something I still want to test, so I check the model type and candidate shape separately from the kernel used for top-k.

MTP spends seven small sequential forwards to build its chain; DFlash2 spends one block forward plus a learned path selection. Both still run the same target verification step.

Where this lives in SGLang

Both workers begin with a normal target prefill. After that, the control flow is similar: prepare draft inputs, produce candidates, call target verify, accept a prefix, commit recurrent state, and prepare the next draft boundary. The shared worker interface lets both modes use the same target scheduler, KV pool, request batching, and output path.

The state details differ. MTP’s worker carries one auxiliary hidden row through seven draft forwards. DFlash2’s worker carries a compact sliding-window KV ring and five target feature streams. After target verification, it writes the committed target hidden rows back into the correct physical ring slots so the next block begins from verified state. Rejected draft rows are never allowed to become committed history.

I kept losing track of which worker owned which piece of state. Writing it as four responsibilities helped. The model class owns the draft weights and per-layer tensor transforms. The speculative worker owns proposal and acceptance state. The target scheduler owns the resident target pool. The DFlash2 cache path owns only the draft-visible window. The graph and flush failures made sense only after I separated them this way.

Part III: MTP looked fixed until the context grew

Repairing the MTP checkpoint before tuning it

My first MTP server generated readable text but reported almost no useful draft acceptance across code, arithmetic, and prose. Seeing the same failure pattern on unrelated prompts was the clue. A merely weak drafter should have varied with the prompt.

When I logged the candidates, I found token id zero in every draft slot. The draft seed was NaN, and the NaN began inside the MTP layer. Its projection modules had shape metadata but no loaded weight tensor.

The checkpoint stores the MTP head in BF16. Its quantization_config.ignore list excluded vision modules but omitted mtp.*, so SGLang constructed the head as compressed-tensors W4A16. The corresponding packed tensors do not exist because the weights are plain BF16. No load exception surfaced; the uninitialized path produced NaNs and argmax selected zero.

I fixed the configuration rather than changing any weight bytes. The ignore list now contains every checkpoint-facing MTP module name plus the fused qkv_proj and gate_up_proj module names SGLang constructs. Acceptance moved from almost no useful proposals to a useful range. Both Hugging Face publication paths now reject a card or config whose MTP exclusions are missing.

This is where I learned that coherent target output says very little about the draft. I now check acceptance and the modules that loaded beside the generated text.

Finding the useful MTP window at short context

I swept several draft widths on the native context profile. Narrow drafts did not amortize the speculative overhead, while wider drafts crossed into a more expensive matrix tier and committed a smaller fraction of their proposals. A width of eight gave the best balance on this setup.

Eight was the last width inside the oneDNN M=8 tier, where verify can reuse a weight read. Wider drafts accepted more tokens in absolute terms but accepted a smaller fraction and crossed into a more expensive matrix tier.

Two configured XPU paths were also declining this exact verify shape:

XPU dense int8: declined because M=8 exceeded a cap of 4
SYMM AR: declined because 81,920 bytes exceeded a cap of 65,536

I raised the decode caps to M=8 and 131,072 bytes. The short-context point improved slightly, but the long-context curve remained broken. Once again, a server accepting a flag at startup did not mean that the request used the kernel I expected.

Reducing an eight-query verify with a severe shape cliff

Before the long-context fixes, MTP helped at the shortest tested context but fell behind target-only decoding as the prompt grew. The regression became progressively worse, which pointed away from fixed Python overhead and toward work that scaled with the resident KV.

I first blamed the Python stack. Every py-spy sample sat in resolve_seq_lens_cpu. Turning overlap off left the step time effectively unchanged. Python was waiting for device work there; the host function did not own the cost.

I then removed the model and held the KV bytes constant. At the TP4 attention shape and 262,144 cached tokens, one and eight queries both read 268 MB:

The eight-query shape was dramatically slower than the one-query shape even though both read the same KV bytes. Explicit split-count changes left the result unchanged, so the obvious tuning flag was not connected to this path.

For a top-k-one chain, proposal position j attends to the committed prefix and only the earlier proposals. The same causal relation can be represented as independent one-query rows with a separate cache length for each position:

equivalent formulation	relative outcome
one call with eight queries	slowest reference shape
eight serial one-query calls	substantially faster
batch of eight one-query rows	fastest tested equivalent

Representing a linear top-k-one verify as eight independent single-query rows removes the measured shape cliff without changing causal dependencies.

The verify rewrite applies the batched single-query formulation to target verification. Accepted length and output stayed unchanged, and decode improved at both short and long context. The long-context result still lagged badly, showing that I had fixed only part of the problem. Verify attention was expensive, but the draft path was still getting slower with context.

Adding the XPU multi-step draft backend

After the verify rewrite, phase attribution showed that the seven draft forwards still cost more than target verification. The MTP draft model was still using a generic Triton attention backend.

I added native XPU metadata and attention for draft decode and draft extend. Each linear draft position receives the committed prefix plus the earlier proposed positions. Tree drafting is rejected because the metadata layout only describes a linear top-k-one chain.

Graph capture exposed another context assumption. Decode graph buckets [1,2,4,8] describe batch sizes, but every bucket still needs page-table and sequence metadata wide enough for the full 1M context. The first implementation retained a lazily allocated 262K-shaped buffer and failed on a later bucket. The repaired backend allocates one max-context buffer set with stable views for each captured batch size.

The first useful long-context probes after that change covered several prompt shapes and draft widths. They confirmed that the native backend removed the catastrophic scaling failure. Because the prompts, outputs, and widths differed, I used them only as diagnostics rather than as comparative results.

Timing the 524K step after the draft fix

I still suspected the MTP auxiliary head, so I added device phase timers. I was wrong.

I instrumented one long-context request with width three. The device timers add enough overhead that their throughput is not representative, but their ordering is still useful:

device phase, rank 0	qualitative share
target verify	dominant phase
draft extend	secondary phase
both draft replays	smallest categorized phase

After the native XPU draft backend landed, target verification became the dominant categorized phase.

The categorized phases closely accounted for the observed step. The timer-enabled request was slower than the timer-free request, a reminder that the instrument itself changes the result.

I had also suspected the Gated Delta recurrent commit. Small probes showed that the Mamba commit and hidden-state materialization were minor phases compared with target verification. Gated Delta state capacity still affects how many requests fit, but it did not explain this decode slowdown.

I tried two more target-attention ideas. Folding query positions into the head dimension was bit-exact with a contiguous page table, but it was consistently slower than independent single-query rows. Forcing a fixed KV split count also lost to automatic selection at both tested long-context shapes.

Part IV: DFlash2 needed more than an XPU flag

Pinning the DFlash2 source I used

DFlash2 arrived from an unmerged SGLang development line. At one point my generated overlay contained the newer DFlash worker and the release image’s older dflash.py, which exported classic DFlash only. The worker looked right. The model registry showed that I had mixed two source trees.

The release process now pins the public SGLang DFlash2 revision. The port copies only that revision and records where it came from before checking that DFlash2DraftModel is registered. The launch reads the checkpoint’s declared architecture, block size, and sliding window. A classic DFlash model cannot start under a DFlash2 name.

Turning a logical window into a physical KV ring

DFlash2 declares a 2,048-token sliding window. The first compact implementation only shortened the request table. Its physical draft allocator still reserved one draft KV row for every target token.

That distinction mattered immediately. A full-context BF16 draft KV allocation reduced the target pool below the configured capacity. A draft-only FP8 diagnostic recovered some capacity but still missed the goal and then exhausted workspace on its first prefill. Raising the memory fraction could improve the startup banner while removing the workspace needed by a real request.

I replaced it with a private, page-aligned ring for each live request. With a 2,048-token window, page size 64, and eight-token verify block, the aligned stride is 2,176 slots. Eight effective requests therefore need:

8 requests × 2,176 draft slots = 17,408 physical draft-KV slots

The opt-in one-million-token layout keeps the full target pool while absolute draft positions wrap through compact private rings; a radix hit refills the visible draft tail from verified target hidden rows.

Absolute token positions map into that private ring. The compact request table contains draft-local physical indices, while target verification continues to use the full target pool. On a radix prefix hit, the target holds back the visible draft tail, recomputes its hidden states, and repopulates the new request’s ring. This is why a logical sliding-window flag alone could not solve the capacity problem.

For the native-context profile, I use the upstream-style compact request table with the target allocator’s exact physical KV indices. The private ring remains only in the opt-in 1M profile, where it prevents the drafter from reserving another million-token KV pool.

Capturing both target and draft graphs initially reduced the target pool below the configured capacity and produced unstable long-tail steps. Keeping the target decode graph and running the much smaller DFlash2 draft eagerly preserved the configured target pool. I therefore kept target graphs and ran the small DFlash2 draft eagerly.

Surviving cache and graph lifecycle failures

The explicit POST /flush_cache path once produced a successful first generation, a successful flush response, then UR_RESULT_ERROR_DEVICE_LOST on the next generation. Client disconnects and a 12-turn shifting-prefix session passed; the defect belonged to the administrative allocator and graph lifecycle rather than the normal OpenAI-compatible agent path.

I tried all of the following with draft graphs still enabled:

clearing or zeroing the compact request table;
synchronizing before the scheduler cleared pools;
skipping torch.xpu.empty_cache();
synchronizing the target graph;
forcing the selector eager;
creating a private draft graph pool.

Graph-off passed but gave up too much throughput. A graph teardown and post-flush recapture hook passed small lifecycle probes, then the recaptured short-context path developed severe tail regressions. The process survived, but the serving result was worse.

At this point I disabled DFlash2 draft graphs and retained target graphs. The draft is small enough that eager execution works here. I kept the cache hooks in case someone overrides that setting. Part VII returns to the lifecycle problem and gets both graphs running together. DFlash2 also rejects logprob-returning requests before scheduling because its chain-stride logprob path is not implemented. Without that guard, the request could reach a crash-prone path.

Part V: Then I measured the whole server

Prefill was still prefill

Speculative decoding changes output-token generation. It does not make a 524K-token prompt cheap to ingest.

The matched sweep showed prefill slowing steadily as context grew. An instrumented long-context request made the shape visible across its fixed-size chunks: early chunks were much faster than the final chunks because each new full-attention chunk scans a larger committed prefix. An isolated-layer probe showed the same monotonic growth. Qwen3.8-27B has 16 full-attention layers, so that repeated scan dominates the final chunks.

I also tried several prefill changes:

chunk sizes from 1K through 16K did not change the per-token attention rate;
pack_gqa was not consumed by this paged Xe prefill path;
activation quantization plus torch._int_mm was slower than complete BF16 projections after quantize and dequantize costs;
a standalone Gated DeltaNet kernel improvement did not move serving TTFT;
a smaller KV tile improved isolated attention but made matched server TTFT slightly worse;
raising the symmetric collective bound into prefill made TTFT substantially worse.

The image keeps KV tile 64 and applies the larger symmetric bound only where the decode and verify shapes benefit.

The eager-draft comparison

For this sweep I used the same full target pool and a draft width of eight for both modes. Each row below is one exact-shape prompt after one warm-up. The table preserves the direction of the result without publishing absolute measurements.

context region	decode comparison	proposal behavior
short-long	MTP ahead	MTP committed longer useful prefixes
middle-long	MTP ahead	MTP committed longer useful prefixes
native boundary	effectively tied	proposal quality was similar
extended-long	DFlash2 ahead	DFlash2 committed a slightly longer useful prefix

With the DFlash2 draft still running eagerly, MTP leads in the earlier context regions, the modes tie near the native boundary, and DFlash2 leads at the longest tested region.

I initially expected DFlash2’s one-forward block proposal to win everywhere. The measurements did not agree. In the earlier long-context regions, MTP accepts enough more tokens to be faster. At the longest tested region, DFlash2 accepts a slightly longer block and finishes each streamed token sooner. Proposal cost is only one part of the step. Target verification, acceptance, and state updates matter too.

The two modes produced byte-identical greedy output at every tested context region. This confirms only that each matched pair generated the same completion; broader correctness needs a larger test.

Ten-prompt results at 8K and 16K

I also ran ten reported prompts after a ten-prompt warm-up at 8K and 16K. The MTP runs came from the model-scoped image; both modes used the same exact cached datasets, a cold radix cache, and the full target-pool settings. At the shorter shape, their streaming distributions nearly overlapped. At the longer shape, MTP had the tighter tail and the better median. Increasing concurrency reduced the per-user rate while increasing aggregate throughput, which is why I keep those two measures separate.

In the short-context campaign, the distributions nearly overlap at the shorter shape; at the longer shape, DFlash2 develops a wider tail while MTP remains compact.

All concurrency-one completions were byte-identical between modes. The short result agrees with the longer comparison: MTP is the stronger default before the eventual crossover, while DFlash2’s advantage appears only at the far end of the measured range.

Per-user TPOT and aggregate completion throughput answer different questions. Multiplying a per-user median by concurrency did not reproduce observed server throughput because admission, TTFT, and synchronization costs remain.

Results I kept outside the matched comparison

The non-speculative baseline used a different output length at the long-context point, so it stays outside the matched comparison. It showed that the full target-pool allocation itself did not materially slow a short request relative to the native pool within restart variation. A separate DFlash2 run used a different physical-ring layout, so I kept it out too.

Did the generated code still run?

I ran HumanEval and MBPP alongside the matched campaign. MTP introduced no regressions among cases that passed without speculation, and most completions were byte-identical. DFlash2 likewise introduced no HumanEval regressions. I did not complete its MBPP run.

This is why I avoid the word “lossless.” Verification remains target-controlled, but Qwen’s recurrent state can travel through a different batched path than strictly sequential decode. Executed behavior and task-specific quality matter more than a label.

Part VI: What I shipped and what I did not

Shipping the clean image without the diagnostics

I rebuilt the release overlay from the pinned qwen3.8-27b-20260816 base and the fixed XPU changes. The DFlash2 port is applied before the XPU cache and lifecycle work so a later source import cannot silently overwrite it.

The final image removes the temporary phase profiler, diagnostic environment switches, experimental draft-KV dtype plumbing, backup files, bytecode, and test logs. CPU-only image checks cover:

the XPU W4A16 oneDNN path and compressed-tensors capability branch;
the block-FP8-to-BF16 path used by the FP8 target option;
Qwen3.8 and the BF16 MTP head registry;
the pinned DFlash2DraftModel class and source manifest;
the compact draft-ring implementation and logprob validation;
the absence of the DFlash phase profiler and generated artifacts;
all environment defaults used for these tests.

The resulting model-scoped tags are rahulunair/sglang-xpu:qwen3.8-27b-20260819 and the moving rahulunair/sglang-xpu:qwen3.8-27b. The repository-wide latest tag remains on the base used by other published model families.

The clean image reached startup with the configured target pool, compact draft rings, target graphs captured, and DFlash2 draft graphs disabled. A subsequent one-token cold /health_generate request left ranks inside first-use compilation and draft-state materialization. The server was restarted while that work was in flight, and the host rebooted while I was switching modes. I could not tie that incident to a card defect or one runtime failure, so I left the cold generation probe out of the public instructions. I first checked the image offline against the same overlay I had already measured. I then started one MTP-only server from the pushed tag, let it warm normally through /health, and used it for the short-context checks above. It completed the campaign, stopped gracefully without a mode switch, and left all four devices in their normal state.

I am keeping that incident here because it changed how I run the server: do not restart a TP server merely because the first compiled request has not returned, and do not use a generated-token health endpoint as cold readiness.

Things I tried and rejected

experiment	observation	disposition
disable overlap scheduler	no meaningful step-time change	rejected; Python stack was a wait site
force verify split counts	no meaningful change across tested counts	rejected; flag did not change this path
BF16 Mamba state	worse decode and proposal behavior	rejected
target and draft graphs	smaller target pool and unstable tails	initially rejected; later qualified after fixing graph-memory ownership
`SGLANG_ENABLE_WAR_BARRIER=1`	substantial decode regression	rejected as over-serialized
fixed GPU clocks	no useful change	restored defaults
skip `empty_cache` during flush	small lifecycle probe passed, 8K test failed	rejected
graph teardown and recapture	process survived probes, short-context decode regressed with poor tails	not the default
draft-only FP8 KV	pool remained below the goal, then first prefill ran out of resources	rejected
fold query rows into heads	correct but substantially slower	rejected
force 20 KV splits	slower than automatic selection at 524K and near 1M	rejected
prefill KV tile 32	isolated improvement, slight end-to-end TTFT loss	retained only as a negative patch
naive INT8 prefill	slower end to end	rejected
larger prefill symmetric collective	substantially worse TTFT	rejected
custom XPU top-16	much slower than `torch.topk`	rejected; keep `torch.topk`

I keep these failures because otherwise I will eventually repeat one of them.

What I mean by 1M support

The server and target pool support 1,048,576 total resident target tokens. That capacity does not make 1M input + 1K output a valid request. Input plus requested output must remain at or below the target limit.

The Gated DeltaNet state pool sets an effective speculative concurrency of eight on this launch even though the scheduler ceiling is 64. Requests beyond active capacity queue. Eight users can run concurrently only when their combined resident target tokens fit the shared pool; eight simultaneous 512K sessions cannot.

Both MTP and DFlash2 have now completed the same exact 1,000,000-token five-needle retrieval request. This validates capacity and focused retrieval, not broad 1M model quality.

Radix prefix caching is enabled in the server. I flushed it and checked for zero cached prompt tokens before every matched row so that each one measured full prefill. I did not complete a separate agent workload with radix hits, so I do not report cache-hit throughput.

Part VII: Finishing the DFlash2 graph path

The eager-draft campaign isolated graph lifecycle as the remaining gap. I made the draft graph usable, stopped flush from tearing down live graph memory, moved native 256K to the shared-index cache layout, and limited the private draft ring to the local 1M profile.

That gave me a DFlash2 path I could compare with MTP on matched prompts. The choice still depends on proposal quality, which changes with the workload.

What is public and what is local

The XPU changes are not published as a source branch. The pinned image under Running it is the public artifact. Readers can reproduce that runtime from the container, but there is no corresponding public source branch.

The native-256K graph path and both 1M profiles in this part are local. The source revision, model identities, settings, and tests below describe them, but pulling the public tag will not reproduce their graph or cache behavior.

The native path uses the public SGLang DFlash2 revision, the main revision of ulkaa/Qwen3.8-27B-AWQ-INT4, and incoai/Qwen3.8-27B-DFlash2 at revision adde41d8fde3a75dc905a7df0bd5088d2a44b5a1. The 1M test uses the target’s public 1m revision, which applies YaRN factor four to the same weights, plus a local DFlash2 checkpoint with the matching YaRN settings. That local draft checkpoint is not published.

Source and checkpoint integrity

The full DFlash2 model is present.

A mixed source tree could combine the speculative worker with the classic DFlash model file. The worker name looked right, but the model behind it did not contain the complete DFlash2 path.

The port now comes from one pinned SGLang revision. It includes the five-layer drafter, grouped two-tap causal convolution, target feature capture at layers [5, 19, 33, 47, 61], top-16 candidate generation, rank-256 predecessor and successor codebooks, the causal transition walk, and the accept, bonus-token, and commit steps. Startup checks the checkpoint architecture and the DFlash2DraftModel class before serving a request. A classic DFlash checkpoint cannot pass under the DFlash2 name.

Missing trained tensors stop startup.

A missing checkpoint name could leave a DFlash2-only tensor at its zero initialization when a checkpoint name failed to resolve. The server would start, and coherent target output could hide the dead selector or convolution weight.

The loader now checks all 23 DFlash2 selector and convolution tensors. Each one must exist, resolve to the expected parameter, have the expected shape, and contain a nonzero payload. I also tested a deliberately incomplete load. The server refused to start instead of running with a silent zero tensor.

Selector and recurrent math

The transition selector runs on XPU.

Each draft position begins with 16 candidates. Adjacent positions therefore form a 16 x 16 transition table. The learned interaction uses rank-256 predecessor and successor rows, then a short causal walk chooses one chain for target verification.

The XPU path now computes those transition tables and the walk with Triton. Tests cover greedy selection, sampling, top-k one, production shapes, exact ties, candidate ids, and proposal distributions. Those tests compare the XPU result with a direct PyTorch implementation rather than another call through the same selector code.

Small loop references check the math.

Fast code can agree with itself while carrying the same mistake in two places, so I wrote plain loop versions of the operations that mattered. One reference computes the grouped convolution taps, including the first position in a new block. Another expands the predecessor and successor scores for every candidate pair.

The same approach checks accepted-prefix length, bonus-token placement, and the physical cache rows committed after rejection. The final cache state must match the state produced by target-controlled sequential commit. These tests caught index and block-boundary mistakes without requiring a full server run.

Gated DeltaNet keeps beta in FP32.

Packed decode and ReplaySSM were sending sigmoid beta through BF16 before the recurrent update. Target verification kept beta in FP32. That difference was small enough to escape ordinary tolerance checks and large enough to change the state carried into later tokens.

Both speculative paths now compute beta in FP32 and convert only where the destination requires it. A focused test keeps the old BF16 round trip beside the new expression. On the tested values, the round trip changed beta by up to approximately 0.00189.

Cache and graph lifecycle

Native 256K and extended 1M use different cache layouts.

The native path now follows current upstream behavior. A compact request table describes the 2,048-token draft window, while target and draft use the target allocator’s exact physical KV indices. There is no private modulo ring in this profile. It uses native RoPE, page size 64, a 4,096-token prefill chunk, and 262,144 for both context length and resident token capacity. The boundary request was 260,992 input tokens plus 1,024 output tokens.

The local 1M path has a different memory problem. Giving the five-layer drafter another million-token physical pool would take away the target capacity I was trying to preserve. That profile keeps a 2,048-token logical draft window but maps it into a private, page-aligned ring of 2,176 slots per state slot. Eight state slots use 17,408 draft slots, while the target keeps all 1,048,576 resident token slots. Target and draft use matching YaRN settings in this profile.

Flush clears logical state without destroying graph memory.

The failing flush path let the scheduler, target pool, draft pool, and graph cleanup touch overlapping state. Depending on timing, the next request could hit an index assertion or UR_RESULT_ERROR_DEVICE_LOST. Tearing down and recapturing the graph avoided one crash but produced poor tail latency.

The scheduler now clears request-visible cache state while leaving live XPU graph workspaces alone. Native 256K has no private draft allocator to clear. The 1M profile resets its private ring once, in the code that owns it. Normal flush does not tear down and recapture either graph. Generate, flush, generate passed; client cancellation followed by a new request passed; and the four cards returned to their normal state after clean container shutdown.

The five-layer draft and selector stay graphed.

The pinned public image sets SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=1. Target verification was graphed, but the five-layer drafter and transition selector ran eagerly. A generic “graphs enabled” check missed that split.

Once flush stopped clearing graph memory, the target graph and the complete DFlash2 draft graph could remain active together. I compared eager and graphed execution on two matched prompts. Proposal acceptance was identical within each pair, while decode improved on both. Since proposal quality did not move, the gain came from lower execution cost.

Draft KV materialization uses the fused XPU helper.

After the draft graph was enabled, projection, K normalization, Neox RoPE, and the five layer-specific KV writes still ran as separate work. SGLang already had a fused helper for this sequence, but XPU was excluded from its device check.

The XPU path now stacks the five projections, normalizes K, applies RoPE, and writes the five KV layers through that helper. An isolated test compared every intermediate tensor, then a live server test exercised the actual pool writes. On the same prompt, proposal acceptance and the verification count stayed unchanged while both verification time and decode improved slightly.

Whole-server validation

I checked the pinned source and checkpoint ids, all 23 trained tensors, target and draft RoPE settings, selector results, grouped convolution, fused KV writes, exact greedy output, MBPP, flush, cancellation, graph capture, allocator capacity, and clean shutdown. The DFlash2 MBPP run introduced no regressions among the target-only passing cases.

I also tested the custom XPU top-16 kernel. Its BF16 result matched torch.topk, but it was much slower, so torch.topk remains in the runtime.

The two runtime changes below held proposal quality fixed:

change	decode outcome	proposal check
draft graph, prompt 1	clear improvement	identical acceptance
draft graph, prompt 2	clear improvement	identical acceptance
fused draft KV	small improvement	identical acceptance and verify count

The graph change had the larger effect. Fused KV removed a smaller cost after the graph was already working. Neither change can compensate for a prompt on which the drafter proposes a poor continuation.

The matched prompt comparison

These rows use four Arc Pro B70 cards at TP4, greedy generation, concurrency one, a cold radix cache, one unreported warm-up prompt, and one measured prompt at each exact shape. A single prompt at each shape can show prompt and context sensitivity, but it cannot form a latency distribution.

context region	result	proposal behavior
shortest tested	tie	similar acceptance
short	MTP ahead	similar acceptance
middle	MTP ahead	DFlash2 acceptance fell
long	MTP ahead	DFlash2 acceptance fell
native boundary	DFlash2 ahead	DFlash2 acceptance rose

The faster graph could not make up for extra target verification cycles on the middle prompts. At the native boundary, stronger DFlash2 proposals reversed the result. Cycle time and accepted length decide the outcome together. Kernel work changes the first term; prompt quality changes the second.

Both modes completed the one-million-token retrieval

The 1M run used an exact 1,000,000-token natural-text archive. I placed five independent codes from near the beginning through the middle and near the end. Both servers used four B70s at TP4, greedy generation, concurrency one, no radix-cache hits, no scheduler retractions, page size 64, a 4,096-token prefill chunk, an exact 1,048,576-token resident target pool, and --max-mamba-cache-size 8. The output budget was fixed and both modes stopped naturally.

Both modes returned all five codes exactly with no scheduler retractions. In this measured pair, MTP had the stronger decode and proposal behavior. DFlash2 happened to record a lower TTFT, but prefill uses the same target path in both modes. One run per mode cannot assign that prefill difference to speculative decoding.

I used this as an engineering smoke test for capacity, cache layout, and focused retrieval. Broad 1M reasoning quality needs a suite such as RULER. After the DFlash2 run, /flush_cache returned HTTP 200. A fresh short-context five-needle request had zero cached prompt tokens and recovered every code. The captured graph and private ring were still usable after the million-token request.

What this prompt sample suggests

Each recommendation below is scoped to the measured prompt sample. A workload with different proposal acceptance can reverse the choice, so the table is a starting hypothesis for qualification rather than a universal mode ranking.

workload	default	why
8K and below	MTP or either	The matched row was effectively tied.
Around 16K	MTP	The DFlash2 distribution was slower and wider.
32K to 131K, or an unknown prompt mix	MTP	MTP led on the measured prompts and accepted longer blocks.
Work concentrated near native 256K	DFlash2 after testing the real prompts	DFlash2 led on the matched boundary prompt.
One-off 1M retrieval	Either for retrieval; MTP for decode	Both recovered every code; MTP had the stronger decode.
Repeated 1M agent turns or cached-prefix decode	MTP first	Decode matters more after the first prefill, and MTP led in the 1M run.
Mixed contexts where consistency matters	MTP	Acceptance varied less in the measurements I have.
A prompt set with measured high DFlash2 acceptance	DFlash2	DFlash2 can lead when it commits more tokens per cycle.

Use MTP as the conservative default for mixed or unknown workloads and for the current 1M deployment. Use DFlash2 near native 256K after measuring acceptance on representative prompts. DFlash2 can lead more strongly in this sample when acceptance is favorable; MTP is more consistent across the prompts I tested.

The settings differ enough that I treat them as three profiles:

profile	models and position scaling	cache and graph settings	validated capacity
Native 256K DFlash2	target `main`; public DFlash2 revision above; native RoPE	shared target physical KV indices; target and draft graphs; fused XPU KV; page 64; prefill 4,096; Mamba state 48	context and resident pool 262,144; `input + output <= 262,144`; concurrency one tested
One-million-token MTP	target `1m`; YaRN factor four	embedded MTP state; target graph; page 64; prefill 4,096; Mamba state 8	resident pool 1,048,576; concurrency one tested
One-million-token DFlash2	target `1m`; local draft with matching YaRN settings	private 2,176-slot rings; 2,048-token draft window; target and draft graphs; fused XPU KV; page 64; prefill 4,096; Mamba state 8	resident target pool 1,048,576; concurrency one tested; local only

Returned logprobs remain unsupported for DFlash2. The native-256K results need more prompts at 32K, 131K, and the boundary before I would draw a new crossover curve. The 1M result covers concurrency one and one retrieval pattern. Repeated million-token turns with radix reuse still need their own test.

For my mixed long coding sessions, MTP stays the default. I would switch a native-256K workload to DFlash2 only after its own prompts show the same acceptance advantage as the boundary row. The next useful dataset is a set of representative prompts at those three long-context regions, measured as a distribution rather than another single peak.

What I want to try next

Extended context is still nowhere near the short-context experience. Target verification is the largest MTP phase I measured there, and a fully cold long-context prefill remains expensive. My next decode experiment is to split target verification into full attention, Gated Delta recurrence, dense projections, collectives, and state commit, then work on whichever part is largest. I also want a real agent workload with radix hits and a broader long-context quality run.

The small tests saved me the most time in this project. The model registry showed that I had mixed DFlash source files. A fixed-byte attention test showed that the scheduler was not causing the shape cliff. Device timers showed that the MTP draft head was no longer the largest part at long context. Counting physical KV slots showed that a logical sliding window had not saved the memory I thought it had. Each time, one small measurement gave me a better next step than another round of tuning flags.