MIRA / Results

Measured language modeling. Dedicated GPU execution.

Benchmark results with their evaluation conditions and source evidence.

Language-model evaluation

Validation results reproduced from the manuscript at a context window of 512 tokens. FineWeb-Edu (FWE) uses the held-out last 5% of the corpus; WikiText-103 (WT) and HellaSwag (HS) use their test splits. Accuracy is next-token prediction accuracy, including on HellaSwag; it is not multiple-choice task accuracy.

ModelParameters (M)State / params (‰)FWE lossFWE PPLFWE acc (%)WT lossWT PPLWT acc (%)HS lossHS PPLHS acc (%)
Transformer (MHA)142.520.0003.39829.9035.383.47532.3035.863.44431.3138.29
RWKV-6144.644.4183.47132.1634.373.53234.2134.963.52834.0635.41
RWKV-7142.663.6343.49332.8734.143.59836.5134.313.65938.8234.78
Mamba-1148.732.1073.55535.0033.853.55134.8534.763.65338.6034.65
Mamba-2148.4116.8903.47832.3834.583.50633.3235.283.45031.4938.51
MIRA-S144.724.0763.32727.8636.983.44131.2236.663.49132.8237.07

Lower loss and perplexity (PPL) are better; higher accuracy is better. State / params is the total recurrent state across all layers divided by model parameters, in parts per thousand. MIRA-S carries 589,824 recurrent state values across 12 layers.

MIRA-S reports the lowest loss and perplexity and highest next-token accuracy on FineWeb-Edu and WikiText-103 in this comparison. MHA reports the lowest HellaSwag loss and perplexity; Mamba-2 reports the highest HellaSwag next-token accuracy.

View the manuscript training-curve figure (PDF) ↗

CUDA performance

The manuscript compares parameter-matched modules on an NVIDIA RTX A6000 at 32,768 tokens. The reported MIRA forward-pass time is 11.37 ms; reported maximum-batch throughput is 8.36 million tokens/s. These measure the module, not full-model generation throughput. Latency and maximum-batch throughput use different measurement conditions and should not be converted into each other.

BaselineForward-pass speedupMaximum-batch throughput ratio
MHA with FlashAttention4.13×6.41×
Mamba (S6)1.32×2.20×
RWKV-62.15×1.02×
RWKV-72.85×1.28×

Ratios are MIRA speedups over each baseline, as reported in the manuscript. Hardware, batch policy, context length, precision and implementation affect comparisons.

View the CUDA performance figure (PDF) ↗