Current developments in LLMs comparable to OpenAI-o1, DeepSeek-R1, and Kimi-1.5 have considerably improved their efficiency on advanced mathematical reasoning duties. Reinforcement Studying with Verifiable Reward (RLVR) is a key contributor to those enhancements, which makes use of rule-based rewards, sometimes a binary sign indicating whether or not a mannequin’s answer to an issue is right. Past enhancing remaining output accuracy, RLVR has additionally been noticed to foster useful cognitive behaviors like self-reflection and enhance generalization throughout duties. Whereas a lot analysis has centered on optimizing reinforcement studying algorithms like PPO and GRPO for better stability and efficiency, the affect of coaching information—its amount and high quality—stays much less understood. Questions round how a lot and how much information is actually efficient for RLVR are nonetheless open, regardless of some work like LIMR introducing metrics to establish impactful examples and cut back dataset dimension whereas sustaining efficiency.
In distinction to the in depth analysis on information choice in supervised fine-tuning and human feedback-based reinforcement studying, the position of information in RLVR has seen restricted exploration. Whereas LIMR demonstrated that utilizing a small subset of information (1.4k out of 8.5k examples) may keep efficiency, it didn’t look at the intense case of minimal information use. One other concurrent research discovered that even coaching with simply 4 PPO examples led to notable enhancements, however this discovering wasn’t deeply investigated or benchmarked towards full-dataset efficiency. Though RLVR exhibits nice promise for enhancing reasoning in LLMs, a deeper, systematic research of information effectivity and choice on this context continues to be missing.
Researchers from the College of Washington, College of Southern California, Microsoft, College of California, Santa Cruz, and Georgia Institute of Expertise present that RLVR can considerably improve giant language fashions’ mathematical reasoning utilizing a single coaching instance, 1-shot RLVR. Making use of it to Qwen2.5-Math-1.5B improves its MATH500 accuracy from 36.0% to 73.6%, matching the efficiency of a lot bigger datasets. The enhancements generalize throughout fashions, duties, and algorithms. The research additionally reveals results like cross-domain generalization, elevated self-reflection, and post-saturation generalization, and highlights the roles of coverage gradient loss and entropy-driven exploration.
The research investigates how a lot the RLVR coaching dataset might be decreased whereas retaining comparable efficiency to the total dataset. Remarkably, the authors discover {that a} single coaching instance—1-shot RLVR—can considerably increase mathematical reasoning in LLMs. The research exhibits that this impact generalizes throughout duties, fashions, and domains. Apparently, coaching on one instance typically enhances efficiency on unrelated domains. A easy information choice technique primarily based on coaching accuracy variance is proposed, however outcomes present that even randomly chosen examples can yield main features.
The research evaluates their technique utilizing Qwen2.5-Math-1.5B as the first mannequin and different fashions like Qwen2.5-Math-7B, Llama-3.2-3 B-Instructt, and DeepSeek-R1-DistillQwen-1.5 BB. They use a 1,209-example subset of the DeepScaleR dataset for information choice, and the MATH dataset for comparability. Coaching entails the Verl pipeline, with fastidiously chosen hyperparameters and batch configurations. Surprisingly, coaching with only one or two examples—particularly π1 and π13—results in sturdy generalization, even past math duties. This “post-saturation generalization” persists regardless of overfitting indicators. The research additionally finds elevated mannequin self-reflection and exhibits that even easy examples can considerably improve efficiency throughout domains.
In conclusion, the research explores the mechanisms behind the success of 1-shot RLVR, demonstrating that base fashions already possess sturdy reasoning skills. Experiments present that even a single instance can considerably enhance efficiency on reasoning duties, suggesting the mannequin’s inherent capability for reasoning. The research highlights that coverage gradient loss is essential to 1-shot RLVR’s effectiveness, with entropy loss additional enhancing efficiency. Moreover, encouraging exploration by strategies like entropy regularization can enhance post-saturation generalization. The findings additionally emphasize the necessity for cautious information choice to optimize the mannequin’s efficiency, notably in data-constrained situations.
Try the Paper and GitHub Web page. Additionally, don’t neglect to observe us on Twitter and be part of our Telegram Channel and LinkedIn Group. Don’t Overlook to affix our 90k+ ML SubReddit. For Promotion and Partnerships, please discuss us.
Sana Hassan, a consulting intern at Marktechpost and dual-degree scholar at IIT Madras, is obsessed with making use of know-how and AI to handle real-world challenges. With a eager curiosity in fixing sensible issues, he brings a contemporary perspective to the intersection of AI and real-life options.
