Startups explore alternatives to transformer-based language models
Transformers have powered major large language models since Google researchers introduced the architecture in 2017. Their dense-attention mechanism compares tokens across a passage, helping models capture meaning but requiring rapidly increasing computation as inputs grow. The resulting costs and limits on context make it harder to process extensive document collections, code bases or the notes generated by reasoning systems.
Several startups are testing different approaches. Miami-based Subquadratic says its sparse-attention model, SubQ, selects the words most relevant to each input. San Francisco’s Manifest AI is developing power retention, which keeps a rolling summary of context, and has released PowerCoder and Brumby. Liquid AI combines transformers with liquid neural networks in smaller models designed for devices such as cars and Raspberry Pi computers; it says its latest systems match models four times larger. Palo Alto-based Inception uses diffusion to generate multiple tokens at once and claims Mercury 2 is comparable to some GPT-4 models while being ten times faster. Pathway is also testing Dragon Hatchling, which it says solved more than 97% of a demanding Sudoku benchmark.
Written by Sakhanda Wire AI from reporting by MIT Tech Review. Facts, quotes and full context are in the original article below.