Routing Qwen3.8-27B onto the right Intel XPU paths

The official Qwen3.8-27B FP8 checkpoint loaded cleanly the first time I ran it on Intel Arc Pro. The trouble was what happened next: every dense projection fell through to a generic kernel written for other hardware. The model was running, but it had taken the wrong road through the stack.

The four-bit build I use now is 18.2 GiB and works well for my local coding-agent sessions on both two-card and four-card machines. Very little of that came from writing kernels. Most of it came from removing one that should never have been running, fixing a routing bug that made an existing path invisible, choosing a quantization group size, and letting oneDNN do the matrix work.

Follow-up: I used the same AWQ target to study where MTP and DFlash2 trade places as context grows. The crossover, the acceptance behavior, and the commands are in Speculative decoding at 512K on Intel Arc Pro.

I wanted Qwen3.8-27B as the local backend for coding agents: good at code, useful at long context, small enough to run on a few GPUs. Local agents make latency personal. A slow token is not a number in a benchmark, it is an interruption while you are editing a file, searching a repository, or waiting on a tool call.

Getting there took longer than expected. The model is dense, multimodal, and mostly built from Gated DeltaNet layers. It has a multi-token-prediction head, a vision tower, and enough different execution regimes that a change which looks great in a standalone kernel can vanish, or turn into a regression, once it is inside the server.

The useful discoveries were all unglamorous. The FP8 checkpoint had a missing dispatch branch and a routing bug. The four-bit path settled on a library primitive that already existed. Group size 128 became the better serving layout than group size 32. A cost I had written off as fixed host overhead turned out to be dense device work, and the 48 DeltaNet layers I expected to dominate decode were not where the time went.

This is independent work I did in my free time on hardware available to me. It is not an Intel release or an official Intel performance result. Intel XPU support has been arriving quickly in SGLang and the surrounding stack, so the job here was to optimize this particular model and add the model-specific pieces missing from the version I used. Before applying anything described below, check whether current upstream already has it.

Running it

If you only want the model serving, this is the setup I use daily: four Arc Pro cards at tensor parallelism 4, the model’s full 256K context, and room for several requests in flight at once. I use graph capture for the batch sizes that occur in ordinary agent sessions and leave prefill eager so I can inspect it.

The weights are at ulkaa/Qwen3.8-27B-AWQ-INT4 and the pinned serving image is on Docker Hub as rahulunair/sglang-xpu, tag qwen3.8-27b-20260816.

docker run --rm --device=/dev/dri -v /dev/dri:/dev/dri \
  --group-add video --group-add "$(getent group render | cut -d: -f3)" \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --ipc=host --shm-size=64g --ulimit memlock=-1 \
  -p 30000:30000 -v /path/to/Qwen3.8-27B-AWQ-INT4:/model:ro \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  rahulunair/sglang-xpu:qwen3.8-27b-20260816 \
  python -m sglang.launch_server --model-path /model --device xpu \
    --tp-size 4 --attention-backend intel_xpu --page-size 64 \
    --context-length 262144 \
    --chunked-prefill-size 4096 --mem-fraction-static 0.85 \
    --cuda-graph-config '{"decode":{"backend":"full","bs":[1,2,4,8]},"prefill":{"backend":"disabled"}}'

Use --tp-size 2 on a two-card machine. Leaving --max-total-tokens unset lets the server size the KV pool from the memory it finds; pin it lower if you are sharing the cards with something else. The SYS_PTRACE and unconfined seccomp flags are there for one optional fast collective I describe later; drop both and the model still runs normally.

Do check current upstream SGLang and Intel images before reaching for a pinned image. A pinned release is useful for reproducing this article; it is not a reason to keep that patch set around forever.

The rest of this post is how I got there.

What are we actually running?

Qwen3.8-27B has 64 decoder layers. Forty-eight use Gated DeltaNet, a linear attention recurrence, and 16 use full attention. It is dense rather than mixture-of-experts, so almost every large text-model weight participates in every decode step. The checkpoint also carries a 27-layer vision tower and a multi-token-prediction head.

That shape splits the problem into two regimes:

• Decode at batch one has very little arithmetic reuse. It is mostly a question of how many bytes must cross memory for every new token, plus small kernels, layer boundaries, and collectives.
• Prefill turns the same projections into larger matrix operations and adds context-growing attention work. Compute throughput and tile efficiency matter much more here.

I kept two checkpoints working throughout.

1. The official Qwen3.8-27B-FP8 checkpoint, my reference while getting everything working.
2. My Qwen3.8-27B AWQ W4A16 checkpoint: 18.2 GiB, asymmetric group-128 quantization, BF16 vision tower and MTP tensors preserved.

“FP8 model” and “AWQ model” describe how a checkpoint stores its weights, not necessarily the instruction that runs every projection. The runtime may dequantize into BF16, keep an extra signed-INT8 copy for decode, or dispatch an actual four-bit primitive. Keeping storage precision, activation precision, and selected kernel separate in my head avoided a lot of confusion.

These are the configurations I kept for the final checks. Output was coherent, graph replay and the expected paths were active, and I used the second run of each shape. Earlier runs helped me debug the model but are poor baselines.

checkpoint and path	device / TP	input / output	graph state	why I kept the run
official FP8, dense library path	B65 / TP4	8K / 1K	decode captured	reference checkpoint after dispatch repair
AWQ group 128, oneDNN W4A16	B65 / TP2	8K / 1K	decode captured	two-card serving check
AWQ group 128, oneDNN W4A16	B65 / TP4	8K / 1K	decode captured	same-device sharding check
AWQ group 128, oneDNN W4A16	B70 / TP2	8K / 1K	decode captured	device-change check at fixed TP
AWQ group 128, oneDNN W4A16	B70 / TP4	8K / 1K	decode captured	daily serving topology

These are complete setups rather than one-change-at-a-time comparisons: concurrency 1, second run, and roughly 9K context at the end of decode.

The first two rows use different checkpoint formats, card counts, and code paths, so an efficiency ratio between them would mean very little. What matters for this investigation is that the two-card AWQ setup became practical for my interactive work and that every claimed path was proved active.

The official FP8 checkpoint stays useful. It is the standard model, it works in the release image, and it helped me separate real model behaviour from bugs in my own quantization pipeline.

Making it fast

Two pieces of method first, then the changes that actually moved the number.

Before writing a kernel, do the boring math

The first question about any operator is whether it is limited by arithmetic or by memory traffic:

time >= max(useful_FLOPs / achievable_FLOPs_per_second,
            bytes_moved / achievable_bytes_per_second)

For batch-one dense decode, arithmetic intensity is close to one operation per weight byte. The memory term wins quickly, so the first useful model is just counting bytes:

bytes_per_token = sum(weights and metadata read by one decode step)
seconds_per_token = bytes_per_token / aggregate_achievable_bandwidth
tokens_per_second = 1 / seconds_per_token

Count from the safetensors headers, not from a number in config.json. An embedding table is resident but only one row gets looked up. The vision tower is resident but idle during a text-only turn. The MTP head is only read when speculation is active. On a multimodal model, counting every resident tensor can add several GiB the step never actually reads.

Quantization metadata counts too. A nominal four-bit weight with group scales and zero points is not exactly 0.5 bytes per parameter. In this format it is about 0.52 bytes at group 128 and about 0.59 at group 32, and that metadata is read alongside the payload on every token.

Prefill is a different calculation. A reasonable first approximation for the dense projections is 2 × parameters × prompt_tokens floating-point operations, with the attention term added separately. This is why one number called “model throughput” should never be used for both phases.

Both the B65 and B70 have 32 GiB of memory and sit in the same advertised memory-bandwidth class. B70 has more Xe2 execution resources. That difference means decode and prefill can respond differently to a device change even before software enters the picture.

A roofline gives you a boundary. Mostly I use it as a warning system: it tells me when a proposed optimization cannot possibly repay its complexity, when a measurement looks suspiciously good, and when writing another kernel is unlikely to be worth the time.

Check upstream first, then add the missing bits

Pinned containers are good for reproducibility, and they also freeze the stack at one moment. The XPU stack kept moving while I worked, so my loop for every missing path became:

1. Check current SGLang, torch-xpu, oneDNN, Intel’s images, and the XPU kernel packages.
2. Confirm whether the failure is still present in the pinned release.
3. Separate a missing platform-specific branch from a missing implementation.
4. Add the smallest overlay that makes the model correct.
5. Delete the overlay when upstream covers the same case.

Step three is the one that saves weeks, and it came up immediately.

SGLang reads AWQ checkpoints through a component called compressed-tensors, which decides how packed four-bit weights get unpacked and which matrix kernel serves them. In the version I was using, two small things assumed an NVIDIA GPU. A helper that rearranges packed weights into the layout the kernel expects was imported only when CUDA was available, then called later regardless. Separately, the code that picks a quantization scheme asked CUDA for the device’s compute capability before it had chosen a scheme at all, which fails on a machine with no CUDA device.

These were dispatch failures. The four-bit kernel already existed, but the checkpoint could not reach it through the NVIDIA-only route. Adding the missing branch was enough.

The MTP path had the same shape. I registered the XPU attention backend in the speculative draft maps, routed the token-tree convolution to a Triton implementation that accepts tree arguments, and relaxed two helpers that rejected non-CUDA tensors even though their implementation was already Triton. Small integration fixes, not a replacement serving stack.

One more lesson: “the model loaded” is only the beginning of correctness. My first text-only quantization build silently omitted the vision tower and the MTP head. The library never instantiated an MTP module, so save_pretrained could not save tensors it did not know existed. I now build with the multimodal class, copy the MTP tensors explicitly, and verify the tensor list before any performance run.

The FP8 checkpoint was running the wrong kernel

The official FP8 checkpoint is where this started. Its first clean run was slow enough that the model was awkward to use, but the problem was upstream of quantization quality. It was one missing branch, one fact about the hardware, and one routing bug.

Figure 1. The first three states use the same weights. The change is dispatch and routing, followed later by a new checkpoint layout.

state	path the server took	evidence	decision
FP8 block fallback	generic Triton block-FP8 matmul	dispatch trace	remove this fallback on Xe2
scales folded at load	BF16 linear through oneDNN	implementation string and output check	use as the correctness baseline
dense INT8 enabled	unquantized XPU method owns eligible layers	route counters and controlled removal	keep the route visible to XPU dispatch

The missing branch. _dispatch_auto_backend() in fp8_utils.py has no XPU arm. Block-FP8 falls past DeepGEMM, FlashInfer, CUTLASS and AITER and lands on the generic Triton w8a8_block_fp8_matmul. Every dense projection in a dense 27B model was running an untuned kernel written for other hardware.

The hardware fact. Intel’s Arc B-Series XMX data-type list includes BF16, FP16, INT8 and INT4, but not FP8. FP8 reduces stored bytes here and adds conversion work, so I stopped tuning the fallback. I folded the block scale in at load time and served the result as a plain BF16 linear through oneDNN.

The routing bug, which is the part worth telling. The BF16 arm returned torch.nn.functional.linear straight out of Fp8LinearMethod.apply, while every XPU dense fast path binds UnquantizedLinearMethod.apply. The dequantized layers were invisible to all of them, so dense INT8, which the image enables by default, had been declining every single tensor:

before   intended dense tensors served: none
after    eligible dense tensors served; incompatible tensors declined

Had I A/B tested dense INT8 in that state I would have measured nothing and concluded the INT8 lever does not transfer to this model. The fix is also the truer description of the arm: once the block scale is folded in, the weight is an unquantized BF16 linear, so the unquantized method should be the one serving it. The dispatch repair moved the model onto an existing device path without writing a kernel.

One trap on the way out. The logprob gate reported the BF16 and INT8 arms identical to four decimal places, which reads like INT8 being lossless. It is not. Input logprobs are computed during prefill, where the INT8 path declines because M is the prompt length and its max_m is 4. The metric is structurally blind to a decode-only change. Comparing generated text instead, which can see it: three of four prompts identical, the fourth diverging at character 97 into a different but equally coherent continuation, which is a near-tie logit flipping under greedy decode.

The existing library primitive became the baseline

Once the model was correct enough to benchmark, the obvious question was which four-bit matrix path should serve the dense projections.

The answer was not my hand-driven kernel. On the same community AWQ checkpoint, barrydeen/Qwen3.8-27B-AWQ-4bit, the same two B65 cards, and the same captured 128-input/512-output shape:

Figure 2. The existing oneDNN primitive matched the checkpoint layout and became the serving baseline.

candidate path	layout work	what the experiment established	disposition
awq_dequantize followed by torch.matmul	materialize a full activation-precision weight	useful correctness oracle, poor serving structure	keep for checking only
moe_grouped_mm_nt_xe20_w4a16, driven as a dense GEMM	adapt grouped-MoE ownership to a dense shape	reachable and correct	keep as a comparison path
the same kernel with split-K	add a reduction across K partitions	configuration mattered but integration stayed custom	stop once the library baseline held
aten::_weight_int4pack_mm_with_scales_and_zeros / oneDNN	use the checkpoint’s packed layout	existing XPU dispatch, no model-specific tuning file	serving baseline

That ATen primitive dispatches to oneDNN weight decompression. It needed no model-specific tuning file and, for this checkpoint layout, no second weight repack. Reading its source also explained a BF16/F16 difference I had measured. In the pinned oneDNN source, the specialized four-bit dequantization path accepts F16 and F32 destination types. BF16 therefore falls through to a more generic tile-conversion branch.

I spent a day pushing the hand-driven path before accepting the stop condition: once the library primitive held up in the real server and needed less custom machinery, it was time to keep it as the baseline and move up a level.

That moved the whole-server result, and it exposed the next problem. The community AWQ checkpoint left the three large Gated DeltaNet projections in BF16, and those represented nearly half the bytes read during a decode step. The kernel was no longer the only question. The checkpoint itself had become part of the latency path.

Four bits is not really four bits

My checkpoint inventory measured the three large projections in every DeltaNet layer (input QKV, input Z, and output) at roughly 10.36 GiB in BF16. They were among the largest weights read on every decode step. I nearly left them alone. Then I looked at the official FP8 release and found scale tensors for those same projections, with only the small surrounding tensors excluded. Useful prior: the model authors quantize these in their own low-precision release.

My AWQ build quantizes 24.33 billion parameters with asymmetric group-128 W4A16. Per-group scales and zero points bring the stored cost of those weights to 4.16 bits per parameter. Another 3.45 billion parameters stay BF16: embeddings, output head, norms, small DeltaNet gates, vision tower, and MTP head. Across the full checkpoint that averages 5.63 bits per parameter, or 18.2 GiB.

I built it with llm-compressor, using its AWQ modifier with an MSE observer and duo_scaling=False. Calibration ran sequentially by Qwen3_5DecoderLayer over 512 chat-templated sequences of 1,024 tokens: three code-instruction sequences from codeparrot/self-instruct-starcoder for every two general-instruction sequences from HuggingFaceH4/ultrachat_200k. W4A16 keeps the activations in BF16; only the selected weights are packed to asymmetric INT4.

I did not arrive at group 128 immediately. The first all-four-bit build used group 32. It had finer quantization groups and fewer total model bytes than the community checkpoint, and it ran slower. Two measurements explained the sign:

• Group 32 reads four times as many scale and zero-point groups.
• On the same down_proj tensor at M=1, the group-128 layout held up better in the oneDNN path. I checked the matching BF16 GEMV as a control.

Group size turned out to be a serving-layout decision as much as a quality setting, and re-quantizing at 128 was worth more than another day of kernel changes.

Here the comparison is like for like: same cards, same TP2 setup, same 128-input/512-output shape, same server, and two four-bit checkpoints of the same model.

checkpoint layout	DeltaNet input and output projections	weight stream	observed step-level direction
community AWQ checkpoint	BF16	larger	baseline
my group-128 AWQ checkpoint	packed INT4	smaller	improved, but by much less than the byte model predicted

The byte model found the right opportunity and badly mispriced it. I spent a long time assuming I had lost a saving somewhere inside the merged DeltaNet projection. The last section settles that question, and not in the direction I expected.

Four bits buys decode and sells prefill

Quantizing to four bits is a trade, and it is worth stating in both directions.

Figure 3. The packed path favored decode, while the FP8 path favored prefill in the tested setup. The drawing is schematic.

phase and workload	path favored in this setup	physical reason to check
decode, concurrency 1	packed four-bit	each token rereads the weights with little arithmetic reuse
prefill, concurrency 1	FP8 checkpoint	larger matrices give the compute path more work per weight read
decode, concurrency 8	packed four-bit	the expected crossover did not appear

Coding agents send large prompts and then spend a long time decoding, so the choice belongs to the workload rather than to a blanket precision rule.

I expected the four-bit advantage to shrink at concurrency 8. Dequantization work grows with M while the weight-side saving does not, and the usual expectation is a batch size where the FP8 path catches up. The run went the other way. That fits Xe2 having no FP8 matrix path: conversion work scales with activations, activations grow with batch, and the weight-side saving remains.

The missing return from the byte model was therefore a concurrency-one effect, not a fixed property of four-bit decode. At one stream the step has slack the byte model cannot see. At eight requests, the ordering moved closer to what the weight traffic predicted. I kept queued TTFT out of this comparison because it mixes request contention with prefill work.

Where the time actually goes

Once it was quick enough to use every day, the question became where the rest of the step was going, and whether the tools telling me were trustworthy.

Graph capture changes both performance and observability

Decode is a long chain of small operations. Graph replay removes much of the per-operation submission cost, so graph capture is part of the setup I actually use, not an optional benchmark trick. Earlier work on two other models in this series measured it as a first-order improvement. For Qwen3.8 I do not have a clean graph-on/graph-off comparison, so I attach no new percentage to it.

Capture also changes what instrumentation means. A Python-side counter that gets incremented while the graph is being captured can replay the device work forever without ever incrementing the Python value again. I once had a fast collective serving happily inside the graph while its own log insisted served=0.

The only trustworthy proof was controlled removal:

• build the same graph with the path enabled;
• build it again with exactly that path disabled;
• run the same shape after compilation;
• require the timing to move by more than the quiet-box noise floor.

One of those A/B tests was for a small collective operation. When the model is split across four cards, every layer has to sum partial results from all of them, an operation called an all-reduce. During decode that message is tiny: one token at hidden size 5,120 in BF16 is about 10 KiB. At that size almost none of the time goes into moving bytes. It goes into the fixed cost of setting up the exchange and getting the cards to agree they are ready.

So a leaner path for small messages is worth having. The one I use is built on shared memory that the rank processes map directly, capped at 64 KiB, handing anything larger (prefill, mostly) back to the standard collective. Disabling it in the saved TP4 A/B caused a repeatable server-level regression beyond the quiet-box noise floor.

Because it maps shared-memory handles across those processes, it needs SYS_PTRACE and seccomp=unconfined in my container. Those permissions widen what the container can do. They are required for this optional collective, not for running Qwen3.8 in general, so remove them and disable the path if that tradeoff does not suit you.

Hybrid models have another memory pool to watch. The recurrent state pool competes with the KV cache and can bound concurrency before the ordinary request limit does. I once configured it below the number of slots a single request needs and got a server reporting zero runnable requests, with an error that mostly talked about memory. Raising the request limit was never going to help. There was nowhere to put the recurrent state.

The trace blamed the wrong component

At one point I fit TP2 and TP4 measurements to a simple model with a sharded weight term and a constant term. The fitted remainder did not shrink with more cards, so host overhead or graph replay looked guilty.

Rather than apportion a busy trace, I priced components by removal on the real captured graph. Replace one component with shape-matched zeros, keep the rest of the graph and the collectives intact, and measure against an unchanged repeat. This does not produce a perfect additive profile, since collectives overlap some module rows, but it answers the causal question: what disappears if this component is removed?

The removal campaign produced this ordinal budget:

removed component	causal rank in this run	overlap caveat
dense MLP bodies across 64 layers	dominant	includes dense projection work that scaled poorly across cards
attention sublayers across 64 layers	substantial	includes projections, gates, RoPE, recurrence, and collectives
attention and GDN kernels alone	secondary	only the recurrent and attention kernels, not their surrounding linears
layer-stack all-reduces	secondary	overlaps rows above, so ranks are not additive
sampling path	small	measured after the layer stack

The supposedly fixed host term was mostly dense MLP device work, whose poor multi-card scaling made a two-point fit look constant. The DeltaNet recurrence itself was small. Had I trusted the fit, I would probably have spent another week optimizing the wrong layer family.

That changed the order of future work: merged four-bit projection shapes first, then dense scaling, then whatever a fresh removal budget turns up. The recurrence is not free, but it is not the first problem.

The latency tail stayed close to the body

A mean TPOT can hide an ugly tail, so I measured the inter-token latency distribution for the final AWQ release. Each point below summarizes 1,023 inter-token intervals from the second run of a 1,024-token output, concurrency one, graph capture on.

Figure 4. The tail stayed close to the body in every tested cell. Marker spacing is schematic and carries no magnitude.

TP	prompt	B65 distribution	B70 distribution
TP2	256	p50, p90, and p99 stayed close	p50, p90, and p99 stayed close
TP2	8K	p50, p90, and p99 stayed close	p50, p90, and p99 stayed close
TP4	256	p50, p90, and p99 stayed close	p50, p90, and p99 stayed close
TP4	8K	p50, p90, and p99 stayed close	p50, p90, and p99 stayed close

This is the kind of result I like: not dramatic, just steady. At these contexts and this concurrency, the ordinary token-to-token experience hides no large tail. That held across both devices and both TP layouts.

What B70 actually did

Before I had B70 cards I wrote down two competing predictions. If decode were purely a weight-bandwidth problem, the device change should do little. If the non-weight work mattered, B70’s additional execution resources should show up. Then the cards arrived and I measured the real server.

Figure 5. The device swap rejected the pure-bandwidth hypothesis at both TP sizes. The diagram shows logic, not magnitude.

question	prediction	observation	reading
does equal advertised bandwidth fix decode ordering?	little device separation	B70 led at TP2 and TP4	decode contains material execution work
does more sharding remove that work?	TP4 should approach the byte model	the remainder persisted	collectives, recurrence, and dense kernels still set the step
does prefill follow decode?	no, larger matrices should react differently	its ordering differed from decode	phase-specific measurements remain necessary

The useful result from the swap is the rejected hypothesis. Batch-one decode on this model is not purely bandwidth bound.

The same point shows up in how badly TP4 scales over TP2:

Figure 6. More sharding reduces the per-rank weight read while the rest of the step persists. Geometry is schematic.

B70, 8K / 1K, c=1	TP2 to TP4 behavior	interpretation
per-rank weight read	shrinks with sharding	expected bandwidth term
collectives, recurrence, scheduling, and other kernels	does not shrink with the weights	limits multi-card scaling
complete decode step	improves sublinearly	another card changes more than the byte term

The weight term follows the sharding model. The remainder does not. That remainder is the next useful target, not another calculation that assumes the whole step is a weight read.

A kernel-local win the server rejected

Since the thesis here is about method, a clean negative result belongs in it.

I built a successive-halving tuner for the Gated DeltaNet chunk kernel over the Xe2 configuration space, with a measured noise floor and a refusal to promote any winner inside it. It found a clear local winner at the exact per-rank shape the server launches. (Hg=4, H=12 at TP4, not the full-model geometry. I tuned the wrong shape first and had to redo it.)

Figure 7. The kernel cleared its local gate and failed the server-level gate.

End to end it was worth nothing I could separate from noise across four server starts. The kernel is confirmed reachable, selected, and executed 96 times per 8K prefill. The isolated saving predicted a visible server change. None appeared, and I cannot yet explain where it went.

What I can explain is why the lever was small to start with. On the two earlier hybrid models the tuned kernel was the MoE grouped GEMM: dominant, on the decode path, with a real configuration space and no tuned file for those shapes. All three of those conditions fail here. Qwen3.8 is dense, so there is no MoE. The kernel I tuned sits on the prefill path, and prefill was already past target. And this model’s hot decode kernel is a oneDNN library call with no num_warps to sweep. The one Triton decode kernel that did have a configuration space had already been retiled from 32 to 16 on BV, and that change was already part of the serving path rather than another lever waiting to be applied.

A tuning lever earns its place only when the complete step moves.

What the method was worth

What carried over from earlier models, and what the coding agents were and were not good for.

A quick detour through the other models

Qwen3.8 was the fourth model in this line of work. The earlier three are not apples-to-apples competitors, since they use different architectures, card counts, TP sizes, and workloads. Their value here is showing which methods survived contact with another model.

model	failure or observation that transferred	what I brought forward
Qwen3-Coder-Next 80B-A3B	graph replay changed the useful baseline, and small collectives survived inside it	TP-specific tuning, collective latency, and counting only the weights used per token
Ornith 1.0 35B	a Qwen attention override regressed a superficially similar hybrid model	check packed quantization against real tensors and transfer methods rather than settings
DeepSeek-V4-Flash	cold prefill and long-context capacity needed separate campaigns	start with upstream and remove components to find where time actually goes

The most useful warning came from Ornith. It looked similar to Qwen: hybrid attention, Gated DeltaNet, packed four-bit experts. Yet its asymmetric zero points changed the correctness contract, its router already used a good XPU top-k path, and the Qwen attention override made it slower. The method transferred; the configuration did not.

Those rows are reference points, not a competition between models. Each one deserves its own post, and I plan to write them. This one is about Qwen3.8 and the method I used to work out what it needed.

That is also why I do not present a bag of “Xe2 optimizations” to enable all at once. Graph capture, rooflines, proof that a fast path actually ran, and controlled A/Bs are methods. A tile size, context cutoff, or packed layout belongs to the exact model, card, and TP combination that earned it.

What were Codex and Claude actually useful for?

I used coding agents extensively during this work, mainly Codex and Claude. They were useful in a narrower and more practical way than “the agents wrote the optimization”.

• Reading an unreasonable amount of source. One investigation followed the SYCL grouped GEMM through its tile policy and reorder atoms, then compared it against oneDNN. That is how I learned the clever four-bit conversion I was about to implement already existed.
• Surveying an API surface. An agent enumerated quantized primitives with real XPU dispatch, their schemas, and their layout contracts, which quickly separated usable operators from names that merely sounded relevant.
• Building harnesses from a strict specification. The speculative-decoding harness refuses failed requests and keeps cache-sensitive natural-text tests separate from random-token kernel isolation.
• Running serialized campaigns. Tuning many shapes is boring but valuable, and it is a good background job when only one process may own the GPUs.

They were also entirely capable of producing a confident conclusion from an empty log. Twice a container had stopped and the missing output was treated as a finding. A pkill -f pattern matched the replacement command itself four times. One benchmark reported a rejected over-context request as zeros, which looked like an amazing performance result for a few seconds.

The fix belonged in the harness rather than the prompting: fail loudly, record the exact setup and run number, keep retractions beside the results, and refuse to print metrics for a rejected request. My own job stayed the same: one benchmark on the GPUs at a time, one changed variable, and no number without a saved log.

Was any of it correct?

Speed is easy to measure and easy to fake. These are the checks, including the one where I had the wrong model in my head.

Fast gibberish is still gibberish

Two quantization mistakes produced clean server starts and output consisting almost entirely of exclamation marks.

First, the text-only model class changed module prefixes. Weight loading had a name-translation layer, but the quantization ignore list did not, so exclusions silently missed their targets.

Then the quantizer added the container module layers.N.linear_attn to the ignore list, because the container itself was not a Linear. The runtime matched ignore entries by substring, so that one entry hid every projection beneath it. Packed tensors existed in the checkpoint and were never loaded into those linears.

Both cases now fail at build time, and I also compare packed dequantization against an independent float64 reference on real tensors. Shape checks alone are not enough. A wrong packed orientation can be shape-valid and still produce fluent-looking, numerically wrong output.

The final group-128 AWQ checkpoint passed health, determinism, and coherence checks on code, reasoning, factual, and summarization prompts. Against the BF16 reference:

quality check	result
mean rank of the BF16 reference token on aligned steps	1.0000
BF16 reference token outside AWQ top 8	0
mean KL on aligned steps	0.0201
prefill perplexity	8.87 → 9.33, +5.2%
prompts whose greedy text diverged	8 of 8
aligned steps after divergence	71 of 957

Exact argmax agreement applies only where the sequences are still aligned. Every greedy sequence eventually diverged, leaving 7.4% of steps directly comparable. So the claim stays modest: the quantized model tracks BF16 closely before divergence, at a measured +0.46 prefill-perplexity cost. It is still a four-bit model, not an exact BF16 drop-in.

I keep IFEval out of the comparison because I do not have a BF16 run through the same harness and scored subset. Without that matched reference, the number would answer a different question.

Multimodal inference works. The checkpoint keeps the vision tower in BF16, images load, and the model answers questions about them. What I have not done is score it against a broad image benchmark suite, so I can say the capability is present and working without yet quantifying what the four-bit text weights cost it. The MTP path is enabled as well, though acceptance rates and end-to-end benefit on natural coding prompts still need a proper campaign. Random token IDs are useful for defeating the prefix cache, but they pin speculative yield near its minimum and cannot answer whether MTP helps a real session.

The gap I went looking for was not there

Back to the smaller packed weight stream that produced a much smaller step-level change than the byte model predicted. I spent a long time treating that gap as a saving I had somehow dropped between the checkpoint and the step. It turns out there was nothing to find. The saving was never there to collect.

Three measurements closed it.

The shape is not unusually slow. Timing the weight-decompression matmul at every real decode shape, per rank, at M=1:

projection	K	N	packed-path reading
merged linear-attention input	5120	8192	in the same operating band as the other packed shapes
feed forward gate_up	5120	8704	healthy control shape
feed forward down	8704	5120	healthy control shape
attention qkv	5120	7168	somewhat less favorable, but not an outlier
attention output	2560	5120	somewhat less favorable, but not an outlier

The suspect merged shape was not far enough from the feed-forward controls to explain the missing step-level return. Redoing the byte arithmetic with its own measured rate still over-predicted what the complete server could recover.

The path is taken. Logging every distinct shape the first time it reaches the quantized kernel showed both linear-attention projections present on both ranks. Reachable, selected and executed all hold.

The ablation was sitting in my own notes. I already had a matched build with those projections left at 16 bit. Moving them to four bits produced a repeatable step-level change, but far less than their byte count implied.

So the model was wrong, not the build. A bandwidth model prices a weight at bytes over bandwidth and assumes that time comes back when you shrink it. That only holds while the weight read is the critical path for that stretch of the step. In a hybrid recurrent architecture it frequently is not: the recurrence kernel, its state traffic and the pointwise work around it are serialised with the projection in every layer, and making the projection smaller leaves all of that untouched.

One reader worked backwards from the step-level gap to what the new projections must be achieving and called the low-utilization explanation a falsifiable prediction. It was, and the direct kernel measurement falsified it. The shape ran in the same band as the other packed projections. The reasoning was sound; the premise it rested on, that the removed BF16 traffic had been costing what the byte model assigned it, was not.

The byte model belongs in the investigation because it found both the group-size problem and the BF16 DeltaNet projections. It also over-promised here. Both halves are worth knowing, and a model that is useful and wrong in known places beats one that has been quietly tidied.

Some of the things I want to do next

1. Diff the ONEDNN_VERBOSE implementation strings between a merged DeltaNet projection and a dense feed-forward projection at M=1. If the two shape families land on different impls, that is a configuration fix rather than a mystery.
2. Count reorders and copies around the QKVZ output, where a short convolution follows. A layout the convolution does not want would cost a copy proportional to the bytes I thought I had saved.
3. Measure energy efficiency under a matched load. The choice between two and four cards is partly a power question, and I have not measured it.
4. Work out how many concurrent sessions actually fit at 256K on two cards. With only 16 full-attention layers, that is the structural selling point of this architecture and I have not measured it.
5. Get the two compressed-tensors platform assumptions fixed upstream, so the next person hitting that wall does not have to rediscover them.
6. See how often MTP’s speculated tokens are accepted in real coding-agent sessions instead of synthetic prompts.
7. Run a full multimodal suite so the vision path has numbers too.

What I would carry to the next model is the method rather than this configuration: check upstream first, count bytes from the checkpoint, keep the vendor library as the baseline, prove every fast path actually served, measure with graph capture on because that is how the model will be used, and remember what a byte model cannot see.

That is slower than collecting one exciting number, but it makes the next model faster to understand. And Qwen3.8 is now a local coding-agent backend I can use without thinking about every token it generates, which is what I wanted from it.