HRM-Text is a 1B-parameter text generation model and full pretraining framework from Sapient Intelligence, built on the Hierarchical Reasoning Model (HRM) architecture. Its central claim is efficiency: the project positions foundation-model pretraining as accessible for roughly $1,000, reporting 130-600x less compute and 150-900x less data than conventional scaling-based training. Rather than shipping only weights, HRM-Text provides the end-to-end recipe — data pipeline, training code, evaluation, and checkpoint tooling — so researchers can pretrain a capable model from scratch.

Hierarchical Recurrent Architecture

HRM-Text is built on a hierarchical recurrent architecture strengthened by task completion and latent-space reasoning, a departure from the standard decoder-only transformer scaling recipe. The design emphasizes reasoning in latent space, and the framework integrates PrefixLM sequence packing, FlashAttention 3 kernels, and PyTorch FSDP2 for distributed training. Because the attention path depends on FlashAttention 3, Hopper-class GPUs (H100) are the expected training target.

Two Reference Sizes

The project publishes two reference configurations with reproducible cost and time estimates. The L size (0.6B parameters) trains on 8 H100s in a single node in about 50 hours ($800), while the XL size (1B parameters) uses 16 H100s across two nodes in about 46 hours ($1,472). These concrete recipes make the efficiency claims verifiable rather than aspirational, and they lower the barrier for academic labs and independent researchers who lack hyperscaler budgets.

Benchmark Results

The reference runs report strong results for models of this scale. The XL (1B) model reaches 84.7% on GSM8k, 56.5% on MATH, 82.3% on DROP, 60.7% on MMLU, 81.9% on ARC-C, 63.4% on HellaSwag, 72.4% on Winogrande, and 86.2% on BoolQ. The L (0.6B) model trails but remains competitive, for example 77.6% on GSM8k and 56.6% on MMLU. The emphasis on math and reasoning benchmarks aligns with the architecture's latent-reasoning focus.

Complete Pretraining Toolchain

HRM-Text is designed to be run, not just read. It pairs with a companion data_io pipeline for cleaning, tokenizing, and stratified-sampling the pretraining corpus, and it supports single-node and multi-node layouts with shared storage. A published Docker image bundles the full environment including PyTorch, CUDA, and FlashAttention 3, and training integrates Weights & Biases for metric tracking. The workflow covers NCCL communication checks, torchrun launch commands, and checkpoint conversion.

Assessment

HRM-Text is a notable entry in efficient pretraining, offering a reproducible path to a reasoning-capable 1B model at a fraction of typical cost. Its main constraints are practical: it targets Hopper-class hardware due to FlashAttention 3, and reproducing the headline results still requires access to 8-16 H100s. The strong self-reported reasoning benchmarks warrant independent verification, but the open framework, transparent cost estimates, and complete tooling make it a valuable resource for researchers exploring alternatives to brute-force scaling.