Together AI launches compiler for continuously optimized model inference

Together AI has introduced Together Compiler (TKCC), an agent-driven system that repeatedly optimizes a model’s inference forward pass as production traffic changes. The system accepts a server launch command, traffic profile and hardware configuration. Coding agents propose alternatives, while a persistent serving engine measures them and can deploy the strongest result. Together AI says the compiler already operates on several of its serverless endpoints.

The engine starts once rather than restarting vLLM, SGLang or TensorRT-LLM for each trial. Candidates are hot-reloaded through RPC, followed by CUDA graph recapture. A 12-layer fast tier screens ideas and a 60-layer tier validates them against real distributions. Together AI reports 252 times greater agent experiment throughput than an engine-restart baseline, with 1,354 candidates tested versus 168 using only the slower tier.

In the company’s comparisons, TKCC reached up to 2.6 times the transactions per second (TPS) of the best published InferenceX configurations in under 24 hours per setting, using no more than two nodes per model. The highest listed result was 1,205 TPS for DeepSeek v4.1 Flash on GB300; MiniMax M3.1 Flash reached 1,111 TPS on B200, versus a 633 TPS baseline. Results varied by model and hardware: GLM 5.3 on MI355X improved from 463 to 480 TPS.