Training-Free GRPO: Policy Optimization via Context Space

The October 9, 2025 paper from **Tencent Youtu Lab** introduces **Training-Free Group Relative Policy Optimization (Training-Free GRPO)**, a novel method designed to enhance the performance of Large Language Model (LLM) agents without requiring expensive parameter updates or fine-tuning. This approach, rooted in reinforcement learning principles, shifts policy optimization from the **parameter space to the context space** by iteratively distilling high-quality **experiential knowledge** into a token prior. Experiments in mathematical reasoning and web searching demonstrate that Training-Free GRPO significantly boosts the performance of large, frozen LLMs like DeepSeek-V3.1-Terminus, achieving superior results compared to traditionally fine-tuned smaller models while requiring **substantially less data and computational cost**. The method replaces the numerical advantage used in vanilla GRPO with a **semantic group advantage** to guide model behavior, confirming the effectiveness and efficiency of context-based alignment, which also preserves **superior cross-domain generalization**.

Source:

https://arxiv.org/pdf/2510.08191