Currently, most major language models break text into thousands of smaller units called tokens. This converts the text into representations that the model can understand. However, these tokens quickly become expensive to store and compute as interactions with end-users take longer. When a user chats with an AI for long periods of time, this challenge can cause the AI to forget things the user has already told it and the information becomes garbled, a problem some call “context rot.”
New methods developed by (and published in) Deepasik. Latest paper) can help overcome this problem. Instead of storing words as tokens, his system packages written information into images, like it’s photographing pages from a book. The researchers found that this allowed the model to retain nearly the same information while using far fewer tokens.
Essentially, the OCR model is a testbed for new methods that allow more information to be tied into AI models more efficiently.
In addition to using visual tokens instead of mere text, the model is built on a kind of tiered compression not unlike how human memories are lost: older or less critical content is stored in a slightly more fuzzy form to save space. Nevertheless, the authors of the paper argue that this compressed content can remain accessible in the background, while maintaining a high level of system performance.
Text tokens have long been the default building block in AI systems. The use of visual tokens instead is unconventional, and as a result, Dipsec’s model is increasingly capturing the attention of researchers. Andrej Karpathi, former chief of Tesla and a founding member of OpenAI, praised the paper xstating that images may ultimately be better than text as inputs to LLM. He wrote that text tokens “can be useless and just plain awful at input.”
This paper presents a new framework for addressing current challenges in AI memory, says Menling Li, assistant professor of computer science at Northwestern University. “While the idea of using image-based tokens to store context isn’t exactly new, this is the first study I’ve seen that takes it that far and shows that it can actually work,” Lee says.