A beta gate changes what length-penalized RL teaches Qwen3.5
Tiyuvta Research
Read the code and receipts on GitHubAlso published on Hugging Face
On this page
Reasoning models think too long. A large part of those thousands of tokens buys no accuracy. The usual fix is in the reward: penalize length, and RL learns to stop sooner.
We kept the reward fixed and changed the network instead.
The change
Qwen3.5 is a hybrid. Most of its layers are Gated DeltaNet, a recurrent layer that writes into a state matrix at every token. How strongly it writes is one number per token and head, called beta. The same beta sets how much old content is erased and how much new content is written. In the shipped model beta stays between 0 and 1.
We added one small matrix per layer. It reads the hidden state and pushes beta up or down for each token. It starts at zero, so the model starts exactly where it was. On Qwen3.5-4B that is about 2 million new parameters. LoRA trains next to it. We call it the beta gate.
We built two versions. One lets beta go up to 2, which gives the recurrence negative eigenvalues. The other caps beta at 1. Same parameters, no new range.
The result
Every arm used the same GRPO recipe and the same length penalty: learning rate 2e-5, 150 steps, a 512-token training cap, GSM8K training problems, 3 seeds per arm. Hypotheses were written down before the runs that test them, and the analysis code was frozen before each test read.
On Qwen3.5-4B, at an 8,192-token thinking budget:
| arm | MATH-500 anytime accuracy | MATH-500 reasoning tokens | GSM8K reasoning tokens |
|---|---|---|---|
| base model | 0.699 | 5,282 | 3,271 |
| LoRA | 0.786 | 4,223 | 1,666 |
| LoRA, rank 20 (1.9% more parameters than the gate arm) | 0.792 | 4,003 | 1,296 |
| beta gate, beta up to 1 | 0.852 | 2,202 | 297 |
| beta gate, beta up to 2 | 0.849 | 2,153 | 261 |
Anytime accuracy is the area under the accuracy-budget curve: accuracy averaged over thinking budgets from 256 to 8,192 tokens on a log scale. 1 means correct at every budget.
The gate with beta up to 1 reasons 48% shorter than LoRA on MATH-500 and 82% shorter on GSM8K. It is more accurate on MATH-500 at every budget up to 8,192.
The rank-20 LoRA barely moves. The gate beats it by 0.060 anytime accuracy on MATH-500 on all 3 seeds. On GSM8K length it wins by 1,000 tokens under our rule, but with 3 seeds a seed-level t-interval for that contrast includes 0.
It is not only traces that used to run out and now finish. On GSM8K only 1.6% of LoRA’s traces were still unfinished at 8,192, and correct answers alone shrink from 1,535 tokens to 271.
At 2B the gains are smaller. The gate with beta up to 2 adds 0.027 MATH-500 anytime accuracy and held on 3 new seeds. The gate with beta up to 1 passed our rule only on the second set of seeds.
A fresh test set, read once
MATH-500 and GSM8K had been read more than once. So we read a new set once: 130 problems from AIME 2024, 2025, 2026 and AMC 2023. We wrote the two hypotheses down before the read.
On Qwen3.5-4B, the gate beat LoRA by 0.110 anytime accuracy (0.285 to 0.395) and reasoned 1,329 tokens less. Both held on 3 of 3 seeds, and both stay clear of zero when seed variance is counted. The rank-20 LoRA did not close the gap.
These problems are hard. Most traces were still unfinished at 8,192 tokens: 79% for LoRA, 52% for the gate. So a large part of the gain is finishing inside the budget. We did not test past 8,192.
At 2B the gain did not pass our rule. Two of six seeds went the wrong way.
Update, 16 October 2026: the same recipe at 9B
We ran the same recipe on Qwen3.5-9B: LoRA against LoRA plus the beta gate with beta up to 1, 3 seeds each. The only change is a memory setting. Both arms of a seed start from one shared 40-step warm-up. Training ran on two DGX Spark GB10 units.
On a GSM8K validation slice, 500 problems x 2 samples, at an 8,192-token thinking budget:
| arm | anytime accuracy | accuracy | reasoning tokens |
|---|---|---|---|
| untouched Qwen3.5-9B | 0.956 | 96.8% | 2,870 |
| LoRA, 3 seeds | 0.968 | 97.6% | 1,825 |
| beta gate, beta up to 1, 3 seeds | 0.971 | 97.6% | 502 |
The gate reasons 73% shorter than LoRA, 1,324 tokens less, and shorter on every seed (63% to 79%). Accuracy does not move. GSM8K is near the ceiling for a 9B model, so it cannot show an accuracy gain.
So we read harder problems: AIME 2025, both papers, 30 problems x 8 samples, read once. No 9B model had seen them. We wrote two hypotheses down before the read and froze the analysis code by checksum.
| arm | anytime accuracy | accuracy at 4,096 | accuracy at 8,192 | reasoning tokens at 8,192 |
|---|---|---|---|---|
| untouched Qwen3.5-9B | 0.122 | 21.3% | 22.5% | 8,032 |
| LoRA, 3 seeds | 0.169 | 24.4% | 27.5% | 7,869 |
| beta gate, beta up to 1, 3 seeds | 0.272 | 38.1% | 48.2% | 7,210 |
Both hypotheses held on 3 of 3 seeds. The gate beat LoRA by 0.103 anytime accuracy (95% interval 0.061 to 0.147) and reasoned 659 tokens less at 8,192 (323 to 1,036). The gate reaches the untouched model’s best accuracy, 22.5%, with about 1,200 reasoning tokens. LoRA needs about 3,100.
Limits: 30 problems give a wide interval. One benchmark year, one model family, GSM8K-only training, budgets up to 8,192. Most AIME traces still hit the cap: 87% for LoRA, 71% for the gate. So part of the gain is finishing inside the budget. At 9B there is no rank-20 LoRA control and no beta-up-to-2 arm yet.
Where the shortening lives
We took the trained 4B models apart on a validation slice.
The gate arm reasons 1,293 tokens less than the LoRA arm there. Remove the gate and 72% of that gap stays: the LoRA that trained next to the gate learned to be short. The other 28% goes away with the gate, so the gate also acts at inference. Scale the trained gate from 0 to 2 and reasoning gets shorter at every step.
During training, the gated runs started closing their own reasoning inside the cap much sooner. The LoRA arms were still hitting the cap on 80% to 94% of rollouts at the end. So this is a result about speed under a fixed step budget. A higher learning rate or a longer run might close part of the gap. We have not tested that yet.
Negative eigenvalues are not needed
The gate was built to open negative eigenvalues. That part is not needed for the effect. Beta up to 2 and beta up to 1 never separate on MATH-500 or GSM8K under our rule. On the fresh set the range added 0.012 anytime accuracy, a tenth of the gate’s gain. The comparison is not clean either: at the same weights, the version with beta up to 2 moves beta about twice as far.
Olmo-Hybrid-7B was pretrained with negative eigenvalues. Clamping them at inference raised held-out loss by 0.020 nats per token and cost common-word counting 3.8 to 6.3 points. Other tasks stayed within 4 points, but 100 items per task cannot see differences below 3 to 10 points.
What we have not tested
One learning rate, 150 steps, one training cap. No gate-only arm. No LoRA on all linear layers. No gated arm without the length penalty. No evaluation at the vendor’s thinking budget. One model family at three sizes (2B, 4B, 9B), 3 seeds at 4B and 9B. The next version runs these controls.
Why it matters for teams running open models
If you fine-tune a hybrid model for shorter reasoning, where you add trainable parameters changes what RL learns in a fixed budget of steps. A gate on the write strength did more than the same budget of LoRA rank. It is a small change to try.
Code and adapters
The paper is “Beta Gate: Shorter RL Reasoning in Qwen3.5 Without Negative Eigenvalues” by Avi Fenesh, Tiyuvta.ai. The code, the pre-registration records, every analysis receipt and a checker that ties each number in the paper to a result file are at github.com/avifenesh/gdn-beta-gate (tag v1.2.0 adds the 9B record). The trained adapters are in a Hugging Face collection, the 9B ones in Avifenesh/gdn-beta-gate-qwen3.5-9b-grpo, and the generation traces of every readout are on Zenodo. There is also a write-up on the Hugging Face blog with the gate’s code.