Parallax Made It Possible to Train Giant AI Models on Scattered, Ordinary GPUs

Training a large AI model across the open internet faces one brutal problem: every step requires machines to exchange hundreds of gigabytes of data before moving forward.

That delay is why serious AI training remains confined to expensive, tightly connected data centers instead of cheaper hardware around the world.

Parallax, a new training module developed by Chutes (SN64), removes that bottleneck by changing what has to travel between machines.

Below is a summary note on the full 36-page technical report, distilling how the method works and why it matters.

The Wall Everyone Hits

Modern AI models are often built as a Mixture of Experts (MoE), meaning the model is split into many specialized sub-networks and only a few activate for any given piece of input.

On paper that looks efficient, but the way these models train assumes every machine sits inside one fast data center.

1. The traffic is constant: At each training step, data gets routed to whichever machine holds the right expert, then routed back before the step can finish.

2. The volume is enormous: For the sizes tested, that amounts to hundreds of gigabytes crossing the network every single step.

3. The delay is fatal: The next step cannot begin until the slowest transfer lands, so the whole system moves at the speed of its weakest connection.

Over the open internet, where links are slower and less predictable, this pattern simply breaks down, which is why decentralized training has stayed largely out of reach.

What Parallax Changes

Parallax rethinks who is responsible for what. Instead of every machine reaching across the network to run remote experts mid-step, each machine keeps a small, lightweight stand-in for the experts it does not own.

1. Each machine owns a slice: A participant holds a fraction of the experts at full quality and stores compact approximations of the rest.

2. The heavy traffic disappears: Those approximations let a machine complete a full step locally, so the giant per-step data exchange leaves the critical path entirely.

3. What replaces it is tiny: Rather than hundreds of gigabytes per step, the machines exchange roughly 100 MB of background updates that can lag, batch, or retry without ever stalling training.

By refreshing approximations in the background instead of fetching them every step, Parallax eliminates the need for training to pause. 

Proving It Works on Cheap Hardware

Parallax was not just introduced as an idea, it was tested with real models: A 20-billion-parameter model trained across four machines over a wide-area network closely matched a conventional data-center training run using the same amount of training.

1. It matched the baseline, landing within a fraction of a percent of a model trained the traditional way on premium, tightly linked hardware.

2. It ran on modest GPUs, including a harder test that used cheaper consumer-grade cards as helpers to handle expert updates remotely.

3. It scaled on paper, with the design showing larger savings in compute and memory as more machines joined, though the team is careful to call the biggest configurations projections rather than finished results.

They were also honest about the limits, noting that the largest setups remain untested, the cheaper-hardware version is not yet speed-optimized, and a 176-billion-parameter run was only a brief feasibility check rather than a full training run.

Why This Matters Beyond the Lab

Parallax points toward a future where frontier AI no longer requires massive, tightly connected data centers. It turns the biggest communication bottleneck into background traffic that can tolerate slow and unreliable networks.

That makes it possible to train serious models on fragmented, mismatched, and affordable hardware spread across the internet.

For decentralized networks like Chutes, the demonstration shows that large models can train coherently across scattered machines without the network becoming the limiting factor.

➛ Read the Full Technical Report on Parallax Here.