Sakhanda Wire
NVDA $152.18 -2.54% MSFT $505.66 -2.38% GOOGL $279.68 -1.43% META $617.65 -1.40% AMZN $256.02 -0.67%
← Back to the news

Startups explore alternatives to transformer-based language models

Transformers have powered major large language models since Google researchers introduced the architecture in 2017. Their dense-attention mechanism compares tokens across a passage, helping models capture meaning but requiring rapidly increasing computation as inputs grow. The resulting costs and limits on context make it harder to process extensive document collections, code bases or the notes generated by reasoning systems.

Several startups are testing different approaches. Miami-based Subquadratic says its sparse-attention model, SubQ, selects the words most relevant to each input. San Francisco’s Manifest AI is developing power retention, which keeps a rolling summary of context, and has released PowerCoder and Brumby. Liquid AI combines transformers with liquid neural networks in smaller models designed for devices such as cars and Raspberry Pi computers; it says its latest systems match models four times larger. Palo Alto-based Inception uses diffusion to generate multiple tokens at once and claims Mercury 2 is comparable to some GPT-4 models while being ten times faster. Pathway is also testing Dragon Hatchling, which it says solved more than 97% of a demanding Sudoku benchmark.

Written by Sakhanda Wire AI from reporting by MIT Tech Review. Facts, quotes and full context are in the original article below.

Originally published by MIT Tech Review on

Read the original on MIT Tech Review ↗

The reporting and the image belong to MIT Tech Review and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news