Vocence (SN78) Ends Guesswork in Prompt-Driven TTS Evaluation with VocenceBench

Prompt-driven text-to-speech (TTS) has been impossible to evaluate honestly until now. When a model receives an instruction like “a 70-year-old Polish man speaking in a quiet, low-pitched rumble,” verifying whether it actually delivered that voice has been a matter of subjective judgment or narrow probes that miss half the picture.

Vocence (SN78) just released VocenceBench, described as the world’s first prompt-driven TTS evaluation benchmark, alongside VocenceArena, a leaderboard that compares seven voice models against 200 instruction-controlled prompts using the framework. The release lands alongside a whitepaper and the confirmed rollout of King of the Hill architecture on the subnet, previously discussed with Const.

What Vocence (SN78) Just Made Measurable

Every prompt-driven voice generation contains two separate questions that most benchmarks try to answer with one score, and the answer has been sloppy for years:

The Question TTS Benchmarks Ask

1. Trait Adherence: Did the model actually produce the voice that was requested. Age, gender, emotion, accent, tone, pitch, loudness, and pace, each checked as an absolute yes-or-no against the instruction.

2. Naturalness: Does the output sound like a real person speaking, especially on hard text like numbers, dates, URLs, and nested clauses. Judged head-to-head against another clip because how real something sounds is inherently comparative.

VocenceBench measures each question separately and only combines them at the final verdict. That single design choice is what makes the results reproducible bit-for-bit across any evaluator’s machine.

Why Naïve Scoring Falls Apart

Existing evaluation methods for TTS fail in three specific ways, and Vocence engineered against each of them explicitly.

1. Binary metrics drown out graded ones in weighted sums. A yes-or-no gender check ends up dominating a nuanced emotion score.

2. Arithmetic means letting excellence in one dimension buy back total failure in another. A model with perfect pitch and no emotional range still averages out to a middle score.

3. Point estimates crown winners on random noise. A single-sample comparison declares a champion when the difference could not be heard by a human.

VocenceBench‘s answer is a geometric mean across sub-scores, so any near-zero dimension collapses the whole sample. Every objective trait is measured from the waveform by deterministic probes, and the audio-LLM judge only handles what genuinely needs an ear.

Fixed equal weights and a fixed random seed produce identical numbers on any machine, and a challenger only wins when its advantage clears a margin grounded in human just-noticeable difference, verified through paired bootstrap testing.

VocenceArena: The Seven Models That Just Got Compared

VocenceArena applies the benchmark to seven text-to-speech systems, generating 1,400 audio clips across 200 instruction-controlled prompts. The models evaluated include Gemini, VoxCPM, Qwen3, Maya1, OpenAI, Parler-TTS, and ElevenLabs.

Every clip runs through three judges:

1. An intelligibility gate uses speech-to-text to verify the target text is actually being read, and a clip that cannot be understood gets hard-zeroed regardless of anything else.

2. Seven independent probes measure adherence across pace, pitch, loudness, gender, age, emotion, and accent, with a holistic tone check on top.

3. A pairwise naturalness judgment compares clips in both orders to cancel positional bias. Every model then duels every other model across all 200 prompts, and a Bradley-Terry fit turns the pairwise results into a final per-model rating.

Why This Matters for the Subnet

The benchmark unlocks what Vocence has been building toward: a measurement layer capable of ranking miner submissions honestly.

1. Reproducibility for Miners: Any miner can run VocenceBench locally and know exactly how their model would score before submitting on-chain.

2. Gaming Resistance: The geometric mean structure means a miner cannot juke the benchmark by optimizing for one dimension while dropping others.

3. King of the Hill Implementation: The confirmed KOTH architecture on the subnet turns the benchmark into an ongoing incentive. Each new submission has to beat the current champion to earn.

The whitepaper published alongside the release documents the full methodology and the incentive design.

Reproducible All the Way Down

The contract VocenceBench makes is that identical inputs produce identical verdicts on any evaluator’s machine. No hidden weights, no per-run tuning, no human labels influencing the ranking. That contract is what turns voice model evaluation from a subjective conversation into something anyone can audit and reproduce.

For a subnet whose entire product depends on knowing which model is objectively better than the last one, this is the piece that turns miner competition into a compounding flywheel. VocenceArena’s current leaderboard is one snapshot. The point is that any future snapshot can be produced by anyone with the same inputs, and the winner will be the same every time.

➛ Explore VocenceBench Repository (Scoring Library), VocenceArena Repository (Seven-Model Evaluation Pipeline), Vocence Corpus Dataset, Vocence Evaluation Corpus, VocenceBench PyPI package, and Vocence Subnet Whitepaper.