MethodThe KAI research team has released a new major language model (LLM) for coding, which has not only learned the understanding of the code and learned how the code looks like, but also what it does when it is implemented. Model, which name is Code World Model .
In addition to learning the dynamics of its environment, CWM shows strong performance on standard coding and mathematics standards, and opens a potential new direction for AI agents training that can handle more complex, dynamic software development works in the enterprise environment. CWML LMS is a wide range of efforts to push into developing global models ahead of the next token forecast.
Standard code generation limits
Despite the recent developments in the AI ​​Code Generation, producing high quality, reliable code, even the latest LLM, remains a challenge. Meta researchers suggest that this is insufficient to specialize in the complexities of general training sample programming.
In general, a model learns the code by predicting the next directive in a program, as it will predict the next word in a sentence. However, researchers argue that in order to truly master in coding, a model should understand that “not only does a code look like a code, but what happens when it is implemented.” This skill is fundamental to software engineers, whose general understanding is how the changes in the code will affect local variables or how to affect the general behavior of their application. The programmer code does not think of the token as a series of token but as a series of relevant components (variables, items, functions, modules, etc.), after which they translate into a series of instructions. In other words, they prepare a “world model” of their application when they prepare it or make changes to it.
These "Global modeling" Until the central training is completed, the capacity is often ignored in LLM, which is a process that the Meta team challenges.
How does the Code World Model work
CWM is a new LLM that is designed to deal with these challenges through extensive training "Code world modeling data." Instead of waiting for the last fine toning phase, CWM is taught how the code behaves during "Middle training" Step The stages of learning reinforcement.
Researchers focused on two important types of data. The first is the implementation of the code, which is a step -by -step record of how the internal state of the program, like its variables, is operated every line of the code (this is contrary to the classic training scheme that trains models on the code and final results). Through the training of these observation process, CWM gains a keen sense of how the instructions affect the overall behavior of the program. Researchers write, “Our basis here is to educate CWMs and not only the syntax of the programs as well as the written code, as well as in arguments such as verification, testing and debugging.”
The second data type contains agent interactions in the Dokar environment. The team created an artificial data generator, known as Forgiving, which imitates the software engineering agent, such as fixing insects or imposing new features. At the beginning of this training, by observing these multilateral interactions on a large scale, the CWM learns the dynamics of these environments before it is always fine for specific tasks in the same environment.
In practice, it allows CWM to make the code about the code that is imitated by a human developer. For example, when the problem of competitive programming is entrusted, CWM can create an initial solution, then the accuracy check can devise its input output test, and finally compare its predicted output against the actual results. This self -verification loop is a direct result of the training of its global model.
CWM in action
Meta researchers used data and training prescriptions for training 32 billion parameter models, with up to 131,000 tokens. The model shows promising results on key industry standards. SWE Bench certified, a benchmark that includes solving real -world problems with gut hub reservoirs, CWM achieved a 65.8 % pass rate, improving other open -weight models. He also scored a lot on Livecodebinch (a benchmark for competitive programming), mathematics -500 and Aime 2024 (mathematical reasoning), and cruisol (predicted by -code output).
Based on these results, researchers believe that global models can “benefit agent coding, enable the implementation of the codes, enable the implementation of the codes, and show its preliminary results on how the latter can benefit from reasoning.” However, they also emphasize the limits of the model. The CWM has been released as a research model under a non -commercial license and is not intended to be a general purpose assistant or chatboat. Although he received some instructional following data, he did not need a widespread improvement of the use of discussion.
Despite being hopeful about the future of this approach, the Meta team notes that these are “just our first steps in this direction.” They see an important opportunity for future work, which states that “the strong ways to take advantage of the global model’s knowledge to improve the performance in numerous tasks through indicators or fine toning is a solid area for research.”
Global model is the key to intelligence
CWML LLMS comes against the backdrop of growing interest, which is much more than the ability to predict the next token. China’s thinking (COT) arguments, such as the most famous techniques, compels models to write their “ideas” before preparing the final answer. Reasonable models such as DiPsic-R 1 LLM to Long Long Lonely Lonely Lonely Lonely Lonely Lonely Long Lonely Lonely Lonely Lonely Lonely Lonely Lonely Lonely Lonely Lonely Lonely Lonely Long tall long tall long tall tall long tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall tall talle tall tall tall tall tall tall tall tall tall tall tall tall tall tah lol tall tall But COT is still the process of token breed, and there is evidence and research that shows that COT represents only The illusion of thinking And cannot be relied on as a real proof of reasoning.
The global model is a recent and modern war on this issue. Instead of developing this problem as aim for the next token forecast, they try to force the LLM to create a model in the world in their own place, which is not necessary in the output token. One more The recent paper LlMs connect the powers ZipurA deep learning architecture that is specially designed for global modeling. Preliminary results suggest that LLM Jepa is stronger against changes in its environment and can learn new tasks more effectively than trained models on pure next token forecast.
It remains to be seen how well researchers can reconcile these different AI architecture. But what is definitely known is that having a strong global model makes the AI ​​system stronger and reliable in a permanent changing environment of real -world applications.