Chutes Pushes Single-GPU Inference to Nearly 18,000 Tokens per Second

Parallax reached nearly 18,000 tokens per second on a single Nvidia GeForce RTX 5090 while handling 256 concurrent requests. The figure reflects the combined output across all requests. Chutes (SN64) recorded it in an early performance test using a custom version of vLLM that the team is still tuning.

Chutes (SN64) provides on-demand access to AI models through a distributed network of GPUs. Its Parallax model has 8 billion parameters but activates only about 1.25 billion to process each token. Its expert layers use 4-bit arithmetic, and most of its attention layers use a fixed-size cache. These choices reduce computation and memory demands, giving the GPU room to handle more requests at once.

For inference providers, generating more tokens per second on the same hardware can bring down the cost of serving each response. The preview measures how much output the card can produce under heavy load. Metrics such as time to first token and generation speed per request would show how quickly individual users receive their answers.

Chutes also designed Parallax’s training around consumer hardware. The team pre-trained the model across hundreds of GPUs in multiple countries, including fleets of RTX 5090s. Each group of GPUs trained a subset of the model’s experts and used lightweight approximations for the others. This reduced the amount of data the groups needed to exchange and made it possible to train over ordinary network connections.

Chutes expects further tuning of its vLLM fork to increase throughput. The early result shows how far the team has already taken a model designed for widely available hardware, from training it across hundreds of consumer GPUs to serving 256 simultaneous requests on a single card.

➛ Access Parallax Here.