Chutes’ (SN64) Parallax Built a Global AI Supercomputer From 240 Consumer GPUs

Training a frontier-class AI model currently costs hundreds of millions of dollars and requires permission from a handful of cloud providers.

Chutes (SN64), with Parallax, proved that a similar-class model can be trained on ordinary consumer GPUs sitting in homes around the world.

Chutes (SN64) Releases Parallax, a New Blueprint for Decentralized MoE Training

Its 8B pre-training production run went live on September 10, coordinating 240 RTX 5090 cards across 30 hosts in 13 countries through a single dashboard.

Live metrics showed throughput above 3 million tokens per second at roughly $11 per billion tokens processed across the fleet.

What Parallax Is

Parallax is a decentralized training system for sparse Mixture-of-Experts models, built by Chutes (SN64), that lets consumer GPUs train frontier-scale models together.

1. Each participant holds real experts locally while small surrogates approximate the rest, keeping synchronous training free from network coordination costs.

2. Shared components sync asynchronously on tiered schedules, so adding more machines lowers individual load instead of raising fleet-wide coordination overhead.

3. Consumer cards known for instability like RTX 5090s and 4090s perform competitively against expensive professional GPUs, streaming data peer-to-peer without full dataset downloads.

Together, these design choices make permissionless training of open models possible using idle consumer hardware from anywhere in the world.

Touring the Parallax’s 8B Live Register

The public dashboard exposes every meaningful metric from the production run in real time through six connected views.

1. The Overview page gives you the big picture at a glance, with metrics like total tokens trained, live hosts, mean loss, peak utilization, Syncer health, and a country map, all refreshing every few seconds.

2. The Hosts table lets you check in on every GPU in the fleet, seeing where it’s located, how fast it’s running, and when it last checked in.

3. The Syncers view tracks the four regional hubs, Tokyo, Finland, Oregon, and Hong Kong, keeping tabs on message traffic and how well-connected each one is to its peers.

4. The Experts panel peeks under the hood at the 4,096 expert keys spread across 32 layers, showing how the routing stays consistent as training moves forward.

5. The Timeline view rewinds the last 48 hours into one scrollable set of charts, so you can watch throughput, loss, and bandwidth shift over time.

5. The Benchmarks section grades the model itself, running it through ARC-Challenge, HellaSwag, PIQA, SciQ, and more every hour to track real progress.

The whole dashboard operates in read-only mode without any login required, giving anyone full visibility into decentralized training economics.

Cheap Hardware, Limitless Tokens

The bigger opportunity behind Parallax extends beyond training, with the 8B sparse MoE architecture also designed for efficient inference.

Sparse FP4 compute, a fixed KV cache, and compact model weights can make these models cheaper to serve on the same consumer hardware used for training.

More importantly, the $11-per-billion-token cost shows that distributed consumer GPUs can support serious model training without depending entirely on massive centralized clusters.

If that efficiency scales, high-quality open models could increasingly be trained and served across permissionless networks of widely available hardware.

➛ Access Parallax Live Records Here.