Large language models have transformed artificial intelligence, but they come with a significant drawback. As the amount of text they process grows, the computing power, cost, and energy required increase dramatically. Miami-based startup Subquadratic claims it has found a solution to this long-standing challenge with a new model architecture called SubQ, tells MIT Technology Review.
Unlike conventional large language models built on transformer architecture, SubQ replaces dense attention with a dynamic sparse attention mechanism. Traditional transformers compare every token in a document with every other token, resulting in an enormous number of calculations that grow quadratically with document length. SubQ instead identifies only the most relevant relationships between tokens, dramatically reducing the computational workload while maintaining high accuracy.
Initially, the company’s announcement was met with skepticism because it provided little evidence beyond internal benchmarks. Since then, independent testing conducted by Appen has lent credibility to several of Subquadratic’s claims. According to the evaluation, SubQ achieved frontier-level coding performance while delivering substantially higher processing speed than previous sparse attention techniques. The model also demonstrated the ability to retrieve information accurately from extremely large datasets using context windows of up to 12 million tokens, roughly 12 times larger than those of today’s leading models.
The potential advantages extend beyond speed. Subquadratic says its architecture significantly reduces inference costs and energy consumption, making it well-suited for applications such as analyzing hundreds of documents, searching massive code repositories, and handling enterprise-scale knowledge bases. If these claims hold up under broader real-world testing, the technology could lower the cost of deploying advanced AI while making long-context reasoning far more practical.
Despite the promising benchmark results, experts remain cautious. The model has not yet been widely released, limiting independent verification. Researchers also note that SubQ was bootstrapped using the weights of an existing open-source model rather than being trained entirely from scratch, raising questions about the extent of its architectural breakthrough. Even so, Subquadratic’s work suggests that improving AI efficiency through new mathematical approaches, rather than simply scaling hardware and data, may define the next stage of large language model development.