Transformers have powered nearly every major large language model since Google researchers introduced the architecture in 2017. But as AI systems handle longer inputs and more demanding reasoning tasks, the technology is becoming a bottleneck. Dense attention, a core transformer mechanism, compares every token with every other token, causing computing requirements and energy use to rise rapidly as context grows, tells Forbes.
Several startups are now exploring alternatives. Miami-based Subquadratic is developing sparse attention, which calculates relationships among only selected words rather than every possible pairing. Its SubQ model identifies which words matter as it processes text. Manifest AI takes another route with power retention, a mechanism that maintains a rolling summary of relevant information while discarding less useful data. The company believes this approach could support tasks involving enormous amounts of context, including long videos and AI agents operating for extended periods.
Liquid AI combines transformers with liquid neural networks, which can adapt their behavior as they receive new information. Its hybrid liquid foundation models are smaller and more energy efficient than conventional LLMs, with some capable of running on a Raspberry Pi. The company says its models can match competitors several times larger.
Inception is pursuing diffusion-based LLMs that generate multiple tokens simultaneously instead of producing text one word at a time. Its latest model, Mercury 2, reportedly matches some GPT-4 models while operating 10 times faster.
Pathway is taking an even broader approach. Its Dragon Hatchling model replaces attention with state-space mathematics, allowing information to be represented more abstractly rather than strictly through language. The model solved more than 97% of a benchmark containing over 250,000 difficult sudoku puzzles.
Together, these approaches suggest that the next generation of LLMs may rely less on scaling transformers and more on rethinking the underlying architecture of AI itself.
