The researchers Meta fair And National University of Singapore have developed a novel reinforcement learning framework for self-improving AI systems.
It is called Self-Play (Spice) in a Corpus Environmentthe framework pits two AI agents against each other, creating its own challenges and gradually improving without human oversight.
Although currently a proof of concept, this self-play mechanism could provide a foundation for future AI systems that can dynamically adapt to their environment, making them more robust against the unpredictability of real-world applications.
The challenge of self-improvement
The goal of self-improvement is to create systems that can enhance their capabilities by interacting with their environment.
A common approach is reinforcement learning with measurable reinforcement rewards (RLVR), where models are rewarded for providing correct answers to problems. It is often limited by reliance on human-generated problem sets and domain-specific engineering, making it difficult to scale.
Self-play, where a model improves by competing with itself, is another ingenious paradigm. But current approaches to self-play for language models are often limited by two main factors.
fThe questions and answers created compound actual errors, leading to a deceptive feedback loop.
When problem generators and solvers lack information asymmetry (ie, share the same knowledge base), they fail to generate genuinely new challenges and fall into repetitive patterns.
As the researchers note in their paper, “These systematic experimental failures demonstrate that self-improvement requires interaction with an external source that provides diverse, corroborating feedback, rather than closed-loop pure chaos.”
How Masala Works
Spice is a self-play framework where the same model acts in two separate roles.
a "Challenger" Curriculum develops challenging problems from a large corpus of documents.
a "Reasoning" It then tries to resolve these issues without accessing the source documentation.
This setup breaks the information asymmetry that limits other methods of self-play, because the arguer does not have access to the documents and knowledge that the challenger uses to formulate the problems.
Grounding tasks in a wide and diverse corpus of documents prevents delusions by anchoring questions and answers in real-world content. This is important because for AI systems to reliably improve themselves, they need an external grounding source. Therefore, LLM agents should learn from interactions with humans and the real world to avoid compounding errors.
The antagonistic dynamic between the two characters creates an automatic course.
The challenger is rewarded for creating problems that are diverse and at the border of reasoning ability (not too easy and not impossible).
The reasoner is rewarded for giving a reasonable answer. This symbiotic interaction forces both agents to constantly explore and overcome new challenges.
Because the system uses raw documents rather than predefined question-answer pairs, it can generate diverse task formats, such as multiple-choice and free-form questions.
This flexibility allows Spice to be applied to any domain, breaking the barrier that has limited previous approaches to narrow fields such as mathematics and code. This reduces reliance on expensive human-induced datasets for specialized domains such as legal or clinical analysis.
Spice in action
The researchers evaluated Masala on several bass models, including QWEN3-4B-BASS and Octothinker-3B-Hybrid base.
They compared its performance against a base model such as a base model with no training, a reasonable model trained with a fixed "Strong challenger" . The assessment covers various standards of mathematics and general reasoning.
In all models, spices consistently improved baselines, leading to significant improvements in both arithmetic and general reasoning tasks.
The results show that the reasoning abilities developed by corpus-grounded self-play transfer across different models, thanks to the diverse external knowledge corpus they used.
A key finding is that adversarial dynamics produce an effective automatic course. As training progresses, the challenger learns to create increasingly difficult problems.
In one experiment, Reasoner’s pass rate on a fixed set of problems went from 55% to 85% over time, demonstrating his improved abilities.
Meanwhile, later versions of the challenger were able to produce questions that dropped the pass rate of the initial phase of reasoning from 55% to 35%, confirming that the two roles were successfully matched.
The researchers conclude that this approach offers a paradigm shift from “closed-loop self-play” in self-improving reasoning practices that are often stagnant due to illusory flows, to open-ended improvement through interaction with the vast, verifiable knowledge embedded in Web document corpora. “
Currently, the corpus used for spice represents the human experience captured in text. The ultimate goal is to generate questions based on interaction with reality for self-improving systems, including human interaction in the physical world, the Internet, and multiple modalities such as video, audio, and sensor data.