With Qwen3.5-9B it looked like quantization broke the MTP head. The BF16 model accepted more, the gap looked bigger on long prompts, and the explanation was easy: the head was trained with BF16 hidden states, and NVFP4 changes them.
At first the numbers looked bad. The old battery showed an apparent 16–43% quant loss on the longer prompts. That was exactly the result I expected. Quantize the base, move the hidden states, and the co-trained head should stop agreeing with it. Maybe it gets worse with context.
The problem was that the comparison was not clean.
The first result was a trap
The BF16 arm was generating repetition loops on some of the long prompts. One p3 run repeated “right before it” and reached 96.9% acceptance. The model was not suddenly very good at speculation. It had made the same easy tokens over and over.
That inflated the BF16 self-agreement.
The other early comparison used a GGUF arm with a different quantization pipeline: Q4/Q5 components, an 8-bit projection, a Q6_K output head and a different converter path. That was not “BF16 base versus NVFP4 base”. It was several changes at once.
So I threw those numbers away as a base-quantization result.
The clean comparison
Both arms used the same BF16 mtp.* head weights. The only difference was the base:
- Qwen3.5-9B BF16 safetensors
- Qwen3.5-9B with MLP-only NVFP4
I used teacher-forced replay over fixed 64k-token corpora: agentic text, code and technical documentation. Same contexts, K=4, every 16th position, with depth buckets out to 64k.
The first draft token looked like this:
- Agentic: BF16 59.24%, NVFP4 59.12%.
- Code: BF16 62.61%, NVFP4 61.32%.
- Techdoc: BF16 51.48%, NVFP4 51.60%.
The code corpus is the largest difference, 1.3 points. That is a real row, but it is not the 16–43% collapse from the first battery.
The depth buckets bounce around instead of growing with context. One shallow bucket moves 5.5 points in favor of NVFP4. It does not repeat as context grows. I am not going to write zero difference or ±3 points everywhere. The useful result is that there is no length trend.
The two bases agree on 87–92% of their greedy picks.
So on the clean metric, NVFP4 did not break the MTP head.
I checked its own text too
The forced corpus could still be hiding something, so I generated text with the quantized model and replayed that exact text through both arms.
BF16 got 56.2%. NVFP4 got 55.4%.
That is a 0.8-point difference, inside the measured chunk-FP noise floor. Even on the quant model’s own generation distribution, I could not find a replay-visible gap large enough to heal.
The live gap is a different thing
This is not the same as live speculative decoding. One live K=3 smoke was 50.0% for NVFP4 versus 55.6% for BF16.
I ran the sampled battery too, but it was not clean enough to use as a final number. Several BF16 cells in one seed ran out of memory, and the remaining prompt-level results moved around in both directions. Single sampled prompts are not stable measurements here.
The live path is different from replay. The model generates its own text, builds draft KV from its generated hidden states, and keeps accumulating session state. A difference there can come from generation dynamics, not from the static MTP head weights disagreeing with the quantized base.
That is where the real question moved.
What I think now
The original premise was too simple. NVFP4 weight compression does not show a context-growing MTP compatibility failure on Qwen3.5-9B. The large loss came from greedy loops, mixed quantization pipelines and single-seed comparisons.
But I also can’t say that NVFP4 has zero cost in every live sampled setup. The clean replay says the head is compatible. The live smoke says there may still be a generation-regime or draft-KV issue.
That changes the fix. There is no forced-replay gap to heal, so retraining the head against the quantized base is not justified by this evidence. The next useful test is cross-text replay or a proper balanced live battery.
I can’t claim more without measuring.
The receipts are in hqmtp: replay v2, generation-distribution replay, and the NVFP4 verdict.