To get there, a few things need to happen. First, LLMs should be more efficient and cheaper to run. Some of the greatest progress has been made on this front. One approach, called the mix of specialists, divides the LLM into smaller sections and assigns each one expertise in a different type of work. This means that only certain parts of the model need to be turned on at a given time.
Another way to make LLMs more efficient might be to ditch transformers—the type of neural network that dominates almost all today—in favor of diffusion models, an alternative type of neural network commonly used for image and video generation. There are also more empirical methods. Last year, Chinese AI firm DeepSeek demonstrated a way to encode text in images, reducing computational costs.
Another important area of ​​development has to do with the context window of the LLM. This is the amount of text (or video) that a model can take in at once, equal to its working memory. A few years ago, LLMs could process several thousand tokens (words or parts of words) or a few dozen pages of text at once. The latest models now have context windows of up to a million tokens – a whole stack of books. But the larger the context window and the longer the task, the more likely the models will go off the rails or forget what they were doing. There are also successes. A recent paper by MIT CSAIL researchers introduced what they call Iterative LLMs. Instead of taking in a wide context window at once, recurrent LLMs break their input into chunks and send each to one of their own copies, which in turn can re-break those chunks and send the results to more copies. Multiple LLMs that process small pieces of information seem to be far more reliable for long, difficult tasks. The result is LL.M., but as we know.