Templar, Bittensor Subnet 3, has completed pre-training for Covenant72B, a 72-billion-parameter language model. This is the largest frontier-scale model ever trained in a fully permissionless and decentralized setting. The run was coordinated across a global network of independent GPUs with no central datacenter, no single owner, and no gatekeeping.
Pre-training processed roughly 1.2 trillion tokens, using Templar’s production stack. Gauntlet handled on-chain incentives, rewarding miners for real loss reduction and reliable participation. SparseLoCo, Templar’s communication-efficient optimizer, kept network overhead near 6 percent, even across unreliable internet connections. Training aggregated heterogeneous hardware from dozens of contributors worldwide, with all updates and rewards verifiable on the Bittensor chain.
Earlier checkpoints already showed competitive performance. Covenant72B outperformed prior decentralized runs and tracked closely with centralized open models like LLM360’s K2 at comparable compute stages. This held despite real-world constraints such as network jitter, node churn, and bandwidth limits that centralized labs typically avoid with expensive infrastructure.
Covenant72B proves frontier-scale training does not require billion-dollar datacenters. The model was built through open, permissionless participation, market-driven incentives, and internet-native coordination. Anyone with capable hardware could contribute and earn TAO based on performance.
In capability terms, the run targets roughly Llama-2-70B-era performance, with a more permissive licensing intent. The broader significance is structural. It demonstrates that distributed GPU coordination can rival centralized clusters when incentives and verification are designed correctly.
Covenant72B is part of a larger decentralized stack. Templar (SN3) handles pre-training. Basilica (SN39) provides a trustless GPU marketplace. Grail (SN81) focuses on post-training, reinforcement learning, and alignment. Together, they form an end-to-end decentralized frontier lab.
With pre-training complete, the project now moves into supervised fine-tuning, alignment, and the Crusades phase, where miners will compete to produce high-quality post-training improvements. Evaluation results and leaderboards are expected next.
For Bittensor, this is a proof point. A subnet with meaningful emissions has delivered the largest permissionless training run in history, fully verifiable on-chain. Decentralized AI at frontier scale is no longer theoretical. It is already happening.
Be the first to comment