Open Eye There are researchers Experimenting with a new perspective To design neural networks, with the goal of making AI models easier to understand, debug and govern. Sparse models can provide businesses with a better understanding of how these models make decisions.
Understanding how models choose to respond, a larger view of reasoning models for businesses, can provide a level of confidence for organizations when they turn to AI models for insights.
This approach calls on OpenAI scientists and researchers to look at and evaluate models not by analyzing post-training performance, but by adding interpretation or understanding through sparse circuits.
OpenAI notes that much of the obscurity of AI models stems from how most models are designed, so to get a better understanding of the model’s behavior, they must generate work.
“Neural networks power today’s most capable AI systems, but they are difficult to understand,” Openai wrote in a blog post. “We don’t write these models with clear step-by-step instructions. Instead, they learn by adjusting billions of internal connections or weights until they master a task. We design the training rules, but the specific behaviors that emerge, and the result is a dense web of connections that a human can’t easily judge.”
To enhance the interpretation of mixes, Openai tested an architecture that trains intractable neural networks, making them easier to understand. The team trained a language model with a similar architecture to existing models such as GPT2 using a similar training schema.
The result: better interpretation.
The path to interpretation
Understanding how the models work, giving us insight into how they’re making their decisions, is important because they have real-world implications, says Oppani.
The company defines annotation as “methods that help us understand why a model produced a given output.” There are many ways to achieve interpretation: chain-of-mind interpretation, which inference models often take advantage of, and mechanistic interpretation, which involves reverse-engineering the model’s mathematical structure.
Oppani focused on improving the mechanistic interpretation, which he said “has so far been less immediately useful, but could, in principle, offer a more complete explanation of the model’s behavior.”
According to Openi, “By trying to explain the model’s behavior at a very granular level, a mechanistic interpretation can make fewer assumptions and give us more confidence. But going from low-level details to explaining complex behavior is harder and harder.”
Improved interpretation allows better monitoring and provides early warning signs if model behavior does not conform to policy.
Optimizing the mechanistic interpretation is “a very ambitious bet,” Oppney notes, but research on sparse networks has improved.
How to disable a model
To eliminate the clutter of a model’s connections, OpenEye first cut off most of those connections. Because transformer models like the GPT2 have thousands of contacts, the team had to “zero out” these circuits. Each will talk to only one selected number, so contacts become more organized.
Next, the team performed “circuit tracing” on the tasks to group the interpretive circuits. The final task involved pruning the model “to obtain the smallest circuit that captures the target loss over the target distribution,” According to Openai. He targeted a loss of 0.15 to isolate the exact nodes and weights responsible for the behavior.
“We show that pruning our weighted sparse models yields about 16 times smaller circuits on our tasks than pruning dense models of comparable pretraining loss. We also show that we are able to construct arbitrarily correct circuits at the cost of maximum edges.”
Smaller models are easier to train
Although Openai succeeded in creating sparse models that are easy to understand, they remain significantly smaller than most foundation models used by enterprises. Enterprises increasingly use smaller models, but frontier models, like its flagship GPT-5.1, will still benefit from better rendering down the line.
Other model developers also aim to understand how their AI models think. Anthropicwho has been researching the interpretation for some time, recently revealed that he had “hacked” Claude’s mind – and Claude saw. Meta It is also working to understand how reasoning models make their decisions.
As more enterprises turn to AI models to help them make informed decisions for their business, and eventually users can be researched to understand how the models are perceived, many organizations will need to rely more on models.