Synthdata Open-Sources AutoQuant to Test Frontier AI Agents Against Bittensor Quant Miners

Synth (SN50) has open-sourced AutoQuant, a benchmark that puts leading AI models to the test against specialized quantitative forecasting systems.

Synth (SN50) allows for miners to compete in producing accurate price forecasts across cryptocurrencies, commodities, and equities. Validators evaluate their predictions and distribute rewards based on performance.

The network also provides forecasting data through a commercial API for traders and institutions.

AutoQuant gives AI coding agents 18 hours and a $100 budget to develop a Monte Carlo forecasting model. Each agent receives a minimal starter model, a frozen snapshot of one minute price data, and Synth’s official backtesting tools.

Participants must follow strict rules prohibiting access to future information, with a predefined scoring formula governing the evaluation.

Each agent selects its strongest model, known as its champion, which is then evaluated against historical records of approximately 250 live miners.

The assessment uses market data from the following 30 days, applying the same forecasting prompts and aggregation methods used by Synth’s validators.

Scoring heavily favors the highest performers, leaving average competitors with little reward. Rankings are published across three competitions covering cryptocurrency forecasts over one hour and 24 hours, alongside 24 hour forecasts for commodities and equities.

No tested AI agent has consistently outperformed Synth’s leading miners. Claude Opus 5 achieved the strongest overall result, scoring 33.5 in the second campaign, equivalent to approximately one third of the top competing miner’s average emissions.

Some agents briefly ranked first in individual competitions but could not sustain their performance. Other tested models included variants of GPT-6 Astra and Gemini, with stronger contenders generally averaging between 90th and 170th place among fields of up to 257 miners.

Unlike benchmarks that rely on static datasets, AutoQuant measures performance against real miners competing for financial rewards and continuously improving their strategies.

Its use of future market data prevents agents from relying on answers known during model development, while reproducible evaluations allow results to be independently verified. Success therefore requires consistently accurate probability-based forecasts rather than simply performing well on historical tests.

AutoQuant was described as “the final boss of quant benchmarks” and has made its methodology, leaderboards, source code, agent development logs, and evaluation reports available through its GitHub repository.

Developers can reproduce experiments, test their own agents, or deploy successful models as live miners on SN50. The benchmark now provides a public measure of how quickly AI agents can close the gap with specialized quantitative forecasting systems.

➛ Access AutoQuant Here.

How to Use Synth LLM, the New AI Interface for Monte Carlo Trading Forecasts