TPN (SN65) Is Shrinking AI Models for Browsers and Phones

An AI model can fit on a phone and still lose too much of its ability to be useful. TPN(SN65) is using Bittensor to find out how much of that capability miners can preserve as they shrink open models such as Mistral, Llama and Gemma.

The aim is to make those models practical to run on devices with limited memory.

On Hash Rate Pod, TPN’s co-founder, Mentor Palokaj, called the competition “model squashing.” Miners can use quantization to store model weights at lower precision or distillation to train a smaller model to imitate a larger one.

Half the score comes from Fluid Bench, which refreshes its test questions to limit overfitting. The other half measures how closely the compressed model matches the original’s predictions across all possible next tokens.

Gratis AI, a product of TPN, gives users a way to try the results. The demo checks the visitor’s hardware and loads a model that runs locally in the browser through WebGPU, with no app to install.

Users can choose a faster model for simple tasks or wait a little longer for a more capable one. Mentor compared the current demos’ capabilities to those of early ChatGPT.

At the time of the interview, TPN was running a competition to see how far miners could shrink a larger Gemma model before its performance dropped sharply.

The team also plans to publish models compressed using its own workbench and put them into more public demos. Mentor said its versions outperformed the original developers’ quantized releases in TPN’s side-by-side tests.

TPN began as a decentralised VPN before the team pivoted to model compression. Mentor said their global miner base lets the team tap computing capacity and test more approaches than it could easily manage in-house.

The immediate priority is refining the scoring system so that models that win the competition also hold up in everyday use.

Longer term, TPN wants to sell compression as a service to customers who already have a model and need it to run on specific hardware.

Within Bittensor, the same service could connect subnets that train models with providers that serve them. Compressing models between those two stages could reduce the memory needed to run them and lower the cost of inference.