Kog seeks faster AI inference from conventional GPUs
French startup Kog is developing software intended to increase the speed of AI inference on standard data-center GPUs, including AMD MI300X and NVIDIA H200 systems. A technology preview using its open-source Laneformer 2B model reached about 3,000 tokens per second per request. The model has roughly 2 billion parameters, so Kog still needs to demonstrate that its method works with larger language models.
CEO Gaël Delalleau said the preview generated 200 tangible business leads. The company expects software engineering to be an initial use case, particularly for professional AI workflows where users can wait hours for results. Kog also has design partners building games and applications from prompts, for whom faster output could support higher revenue.
Kog says prospective customers are not currently prepared to fine-tune small models, prompting the startup to focus on accelerating larger ones. Delalleau argues that newer GPUs’ memory bandwidth can be better utilized and that their suitability for decoding is being underestimated. The 11-person team may spend weeks or months optimizing for each new GPU. Kog aims to reach 10-times faster inference on a major model in September, a step it considers important for customer traction and a Series A fundraising round.
Written by Sakhanda Wire AI from reporting by TechCrunch. Facts, quotes and full context are in the original article below.