Why More GPUs Can Still Leave AI Models Out of Memory

Buying more GPUs sounds like the obvious answer when an AI model runs out of memory.

Pareton (SN10) recently showed why it sometimes is not.

In a thread posted via the project’s X page, the team broke down an inference workload that was running out of GPU memory. The natural fix would have been to spread the workload across more GPUs. But even doubling the GPU count would not have solved the problem.

The issue was not how much memory the entire cluster had. It was where that memory was available and what was trying to use it.

That distinction matters as AI companies spend heavily on increasingly large GPU clusters.

More GPUs do not create one giant memory pool

Imagine eight GPUs with 32GB of memory each. On paper, that is 256GB.

But an AI workload cannot necessarily use those 256GB as one large block.

During inference, GPU memory is being used for several things at once. The model itself takes space. The KV cache stores information the model needs while processing longer conversations. Temporary calculations also need their own room.

Some of those workloads can be divided across GPUs. Others cannot.

Pareton’s findings came down to a temporary tensor that needed about 1.9GB of memory on an individual GPU.

With the serving configuration set to reserve most of each GPU’s memory for the model and KV cache, only about 1.7GB remained free on the device when that tensor appeared.

The result caused the system to crash with an out-of-memory error.

Adding more GPUs could increase the amount of distributed memory available for things such as the KV cache, but it would not change the fact that this particular operation still needed 1.9GB of free space on one GPU.

There could be plenty of unused memory elsewhere in the cluster and the request would still fail.

A small configuration change made the difference

The example involved SGLang, an open-source system for serving large language models.

One of its settings, mem-fraction-static, controls how much GPU memory is reserved for model weights and the KV cache. SGLang documentation describes the setting as the fraction allocated to those relatively static parts of the workload, with the remaining memory available for runtime needs such as activations and temporary buffers.

At a setting of 0.85, Pareton’s workload had about 1.7GB free when the 1.9GB tensor arrived.

After lowering the setting to 0.4, roughly 19GB was available, and the operation could run.

That does not mean 0.4 is a magic number or that operators should simply lower the setting whenever something fails. Reserve too little memory for the KV cache and the server can run into a different problem.

The broader lesson is that an out-of-memory error does not automatically mean the machine needs more hardware.

Sometimes the memory is simply being allocated in the wrong place.

This becomes even more important as context windows grow

Modern AI systems are being asked to process increasingly long conversations, documents and agent histories. That makes the KV cache one of the biggest consumers of GPU memory.

Here, adding GPUs can genuinely help because that cache can be distributed across devices.

Recent work from the SGLang, Qwen and NVIDIA teams shows just how deliberate this balancing act has become. In a test involving a one-million-token context, researchers reduced the static-memory allocation from 0.90 to 0.75 specifically to leave more space for temporary work during the initial processing of the prompt.

The point to make from this is that different parts of an AI workload scale differently.

More hardware can solve a model-weight problem or provide more room for a growing KV cache. It may do very little for a temporary operation that still has to fit on an individual GPU.

Why this matters for Pareton

This is the kind of problem Pareton is building Bittensor Subnet 10 to solve.

Instead of asking developers to manually try endless combinations of serving settings, Pareton has miners search for ways to make AI inference faster and cheaper.

Contributors can submit changes involving areas such as kernels, caching, batching and quantization, which are then benchmarked against a fixed workload. Pareton’s documentation says a new configuration only moves forward when it beats the existing baseline while still meeting the required performance limits.

That makes this memory problem a useful example of why inference optimization is becoming its own field.

The expensive answer would have been more GPUs, but the better answer was understanding what the GPUs were doing.

As AI infrastructure grows larger and more costly, improvements will not come only from adding more compute. They will also come from squeezing more useful work out of the hardware already sitting in the rack.

For Pareton, that is the opportunity behind SN10.

Read more about Pareton below:

Pareton (SN10) to Run Models Cheaper Without Compromising Quality