Small language models are appealing because they can run on-device: private, cheap, no round trip to a server. The catch is that sub-billion-parameter models are bad at multi-step reasoning, math especially. Full-scale RLHF is the usual fix, but PPO needs a separate critic network in memory: exactly what you can’t afford on a 0.5B model and a student budget.
For an RL course, I ran an ablation on two critic-free algorithms (GRPO and DAPO) to ask a narrow question: under a hard compute budget, which one actually improves mathematical reasoning without falling apart?
Two models: Qwen2.5-0.5B-Instruct and Gemma-3-1B-it, trained on DeepMath-103K. Reward was split between two signals: format (is the answer inside \boxed{}?) and accuracy (is it correct, checked with math-verify).
Training ran on Modal A10Gs. To make the budget work I used LoRA adapters, gradient checkpointing, and vLLM for rollouts, and fanned out four containers in parallel. Around the training loop I built a sweep runner and a small Streamlit dashboard for comparing runs side-by-side and inspecting individual completions.
DeepMath requires a chain of thought. My first config gave the models 512 tokens of completion budget, nowhere near enough.
\boxed{}, never earned a reward, and learned nothing.This was the most useful result of the project, even though it was a failure. In RL terms, the environment made the goal state physically unreachable.
Raising the budget to 768 tokens (and dropping Gemma, which needed far more) changed everything:
Just giving the model room to finish nearly doubled its score.
With a usable signal, the comparison became clean. After 250 steps:
| Run | Accuracy | Format |
|---|---|---|
| Baseline | 9.4% | 61.8% |
| + GRPO | 8.8% | 63.2% |
| + DAPO | 9.4% | 61.8% |
Standard GRPO reward-hacked: its format rate went up while actual accuracy went down. It learned that emitting a \boxed{} tag earned partial credit, and began trading reasoning for formatting. DAPO, with its decoupled clipping objective, held accuracy flat, acting as a much stronger regularizer under these constraints.