ORO Handed the World’s Best AI Models a Small Business to Run. Most of Them Lost Money

Imagine running a business where a supplier regularly delivers late, customers receive incomplete orders, and some gradually stop buying from you.

Could an AI recognize that its purchasing decisions are costing the business money and change its approach before the losses grow?

That’s the question Bittensor’s ORO (SN15) set out to answer with SMB Bench, a new benchmark released on November 6, 2026.

ORO tested nine leading AI models by putting them in charge of purchasing decisions for a simulated wholesale business over 90 days.

The goal was to see whether they could make profitable decisions, learn from mistakes, and respond to problems whose consequences only become visible weeks later.

The results revealed a major limitation. Only one of the nine models earned more than a simple, automated buying rule on average, while several ended the simulated quarter with substantial losses.

How ORO’s New Benchmark Works

ORO is a Bittensor subnet focused on developing and evaluating AI agents for commerce.

On SN15, miners build shopping agents that compete to complete purchasing tasks, with validators testing their performance and rewarding the strongest agents.

SMB Bench is a separate research benchmark developed by the ORO team to investigate a more demanding type of decision-making.

Instead of asking an agent to find a product or complete a single purchase, SMB Bench places it in charge of buying fresh produce for a wholesaler that supplies restaurants, catering businesses, and walk-in customers.

Each morning, the agent reviews inventory records, sales figures, supplier prices, and delivery schedules. It must decide what to buy, how much to order, and which suppliers to use. It can also purchase emergency stock to cover shortages and keep notes to guide its decisions on subsequent days.

The challenge is that the agent doesn’t know everything happening behind the scenes. If shelves are empty, lost sales aren’t recorded. If restaurants receive incomplete orders, they may quietly reduce future purchases. Suppliers can also begin delivering late without warning. The agent must recognize these patterns from the available records and adjust its decisions.

ORO tested three conditions. In a normal quarter, there were no unusual consequences. In a second scenario, customers reduced their orders after receiving incomplete deliveries. In the third, customers behaved the same way while the cheapest supplier also became unreliable.

Across 485 runs covering nine models and 18 tasks, ORO measured how these decisions affected the business’s profitability. Each model was also compared against a fixed buying rule that simply looked at recent sales, considered delivery times, maintained extra stock, and purchased from the cheapest supplier.

What the Nine AI Models Achieved

GPT-6 Astra emerged as the strongest model, generating an average simulated quarterly profit of $7,749. The fixed buying rule earned $6,412, outperforming all eight other models. Claude Fable 5.1 and GPT-6.1 Sol followed with $5,427 and $5,404, respectively.

At the other end, four models lost money on average, with Muse Wonder 1.0 recording the largest loss at $8,291.

Importantly, Astra’s advantage did not come primarily from learning to adapt. In nearly every run, it began by instructing the business to cover customer shortages through emergency purchases, even when those purchases were expensive. That decision helped preserve customer orders and protect future revenue.

The findings also showed that all nine models consistently bought less produce than the fixed rule, purchasing between 5% and 42% less over the quarter. They frequently recognized shortages but failed to increase their orders enough to prevent them from happening again.

AI Could Recognize Problems Without Fixing Them

One of SMB Bench’s most important findings was the gap between recognizing a problem and actually doing something about it.

In 323 runs where customers reduced purchases after receiving incomplete orders, the models explicitly connected declining demand to their own delivery failures in only 69 runs. Even among those 69 cases, they responded by protecting every affected customer within three days just 12 times.

ORO also compared runs where customer behavior changed with runs where it didn’t. Agents changed their approach in 20% of the runs with customer consequences, compared with 11% without them. This suggested that some models were responding to delayed feedback, but meaningful adaptation remained uncommon.

There were encouraging exceptions. In one Gemini 4 Argon run, the model noticed that four customers receiving incomplete deliveries had reduced their orders by roughly 40–50%, while two customers receiving full deliveries maintained their demand. It identified the relationship and changed its purchasing approach to protect the remaining customers.

That is precisely the behavior SMB Bench is designed to identify. An AI agent must do more than recognize a business problem. It needs to understand how its earlier actions contributed to it and make better decisions while there is still time.

How far are we from a real expert?

To show how much money was left on the table, ORO added two more buyers. One is a “perfect” buyer that can see the future and re-plans every day, which sets the ceiling. The other is a scripted expert, built from rules ORO worked out with GetASAP and real wholesalers.

The expert’s edge comes from habits any seasoned buyer would recognise:

1) It never lets a restaurant down. It switches on cover for every account from day one. Astra did this in nearly every run; Sol and Gemini in about half; the rest rarely.
2) It buys from one supplier to save on delivery costs, while the AI models split orders across several.
3) It trusts the order book. It stocks for delivery time plus a buffer, using restaurants’ booked orders as its forecast. The AI models made forecasts, then bought well below them.

The expert strategy performed better by combining orders to reduce freight costs, maintaining sufficient inventory, and protecting restaurant customers from shortages. These were relatively straightforward business practices that even advanced models struggled to apply consistently.

The results suggest there is considerable room for improvement, not only in how AI models reason but also in how they turn their reasoning into decisions that produce better economic outcomes.

Why This Matters for Bittensor

The significance of SMB Bench extends beyond identifying which AI model performed best. ORO is developing ways to evaluate whether AI agents can make decisions that hold up over time, especially when mistakes have consequences that aren’t immediately obvious.

This builds on the subnet’s existing work in evaluating shopping agents through competitive environments. SMB Bench expands that research into longer-term business decisions, where success depends on profitability, customer retention, inventory management, and learning from experience.

The benchmark is still a simulation, not evidence that these models are ready to operate real businesses independently. However, ORO has made its results and execution records available through Hugging Face, allowing researchers to inspect the decisions behind the scores.

As AI agents move toward handling more business responsibilities, their value will increasingly depend on what their decisions achieve over time.

With SMB Bench, ORO is helping establish a way to measure that ability and identify what needs to improve before businesses can confidently delegate more work to AI.

➛ More on ORO (SN15):

ORO Becomes First Live-Token Crypto Startup Accepted Into Y Combinator