A new controlled study challenges a widely held assumption in the machine learning community about how large language models are most effectively distilled into smaller versions. Researchers found that the direction of KL divergence optimization produces more significant improvements than the sampling strategy researchers have long credited as a core advantage of reinforcement learning approaches.

According to AI Weekly, researchers Julianna Piskorz, Antonin Berthon, and Mihaela van der Schaar conducted experiments comparing different distillation techniques using Llama-3.1-8B as a teacher model. The team systematically varied three independent factors: the rollout policy, the direction of token-level KL divergence, and the learning rate. Their findings, posted to arXiv on September 28, suggest that on-policy rollouts, long considered the structural advantage of reinforcement learning fine-tuning over supervised fine-tuning, do not consistently produce measurable gains when other variables remain controlled.

What This Means for Model Compression

The research has immediate practical implications for organizations working to deploy smaller, faster language models without proportional sacrifices in quality. Model distillation, the process of compressing a large model's knowledge into a smaller one, has become increasingly important as enterprises seek cost-effective inference solutions.

Traditional wisdom suggested that reinforcement learning approaches, which use real rollouts from a model's behavior, offered inherent advantages over purely supervised methods. This assumption shaped years of research direction and engineering choices. The new findings suggest that optimization direction matters more than this structural difference.

Key Variables in Focus

  • Rollout policy: The strategy for generating training samples from the model
  • KL divergence direction: Whether optimization minimizes divergence forward (teacher to student) or reverse (student to teacher)
  • Learning rate: The step size controlling how quickly the model adapts during training

The distinction between KL divergence directions carries particular weight. Forward KL divergence optimization emphasizes matching the teacher's exact probability distributions, while reverse KL divergence allows the student model more flexibility in its outputs. The researchers discovered that this choice produces more substantial performance differences than previously attributed to the rollout mechanism itself.

Implications for Future Research

This work suggests that future distillation research should prioritize investigating optimization objectives and hyperparameter interactions rather than assuming that sampling methodology provides inherent superiority. The controlled experimental design allows other researchers to build upon these findings with confidence in their reproducibility.

The implications extend beyond academic curiosity. Teams developing production language models face real trade-offs between model size, inference speed, and output quality. Identifying which levers actually produce meaningful improvements helps allocate research effort toward high-impact modifications rather than pursuing structural changes that may not justify their complexity.

As language model deployment becomes increasingly cost-sensitive, and as applications demand real-time inference, the demand for effective distillation techniques continues growing. Research clarifying which optimization approaches actually deliver measurable gains becomes correspondingly valuable for practitioners implementing these systems at scale.