Language-model evaluation
Validation results reproduced from the manuscript at a context window of 512 tokens. FineWeb-Edu (FWE) uses the held-out last 5% of the corpus; WikiText-103 (WT) and HellaSwag (HS) use their test splits. Accuracy is next-token prediction accuracy, including on HellaSwag; it is not multiple-choice task accuracy.
| Model | Parameters (M) | State / params (‰) | FWE loss | FWE PPL | FWE acc (%) | WT loss | WT PPL | WT acc (%) | HS loss | HS PPL | HS acc (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Transformer (MHA) | 142.52 | 0.000 | 3.398 | 29.90 | 35.38 | 3.475 | 32.30 | 35.86 | 3.444 | 31.31 | 38.29 |
| RWKV-6 | 144.64 | 4.418 | 3.471 | 32.16 | 34.37 | 3.532 | 34.21 | 34.96 | 3.528 | 34.06 | 35.41 |
| RWKV-7 | 142.66 | 3.634 | 3.493 | 32.87 | 34.14 | 3.598 | 36.51 | 34.31 | 3.659 | 38.82 | 34.78 |
| Mamba-1 | 148.73 | 2.107 | 3.555 | 35.00 | 33.85 | 3.551 | 34.85 | 34.76 | 3.653 | 38.60 | 34.65 |
| Mamba-2 | 148.41 | 16.890 | 3.478 | 32.38 | 34.58 | 3.506 | 33.32 | 35.28 | 3.450 | 31.49 | 38.51 |
| MIRA-S | 144.72 | 4.076 | 3.327 | 27.86 | 36.98 | 3.441 | 31.22 | 36.66 | 3.491 | 32.82 | 37.07 |
Lower loss and perplexity (PPL) are better; higher accuracy is better. State / params is the total recurrent state across all layers divided by model parameters, in parts per thousand. MIRA-S carries 589,824 recurrent state values across 12 layers.
MIRA-S reports the lowest loss and perplexity and highest next-token accuracy on FineWeb-Edu and WikiText-103 in this comparison. MHA reports the lowest HellaSwag loss and perplexity; Mamba-2 reports the highest HellaSwag next-token accuracy.
View the manuscript training-curve figure (PDF) ↗CUDA performance
The manuscript compares parameter-matched modules on an NVIDIA RTX A6000 at 32,768 tokens. The reported MIRA forward-pass time is 11.37 ms; reported maximum-batch throughput is 8.36 million tokens/s. These measure the module, not full-model generation throughput. Latency and maximum-batch throughput use different measurement conditions and should not be converted into each other.
| Baseline | Forward-pass speedup | Maximum-batch throughput ratio |
|---|---|---|
| MHA with FlashAttention | 4.13× | 6.41× |
| Mamba (S6) | 1.32× | 2.20× |
| RWKV-6 | 2.15× | 1.02× |
| RWKV-7 | 2.85× | 1.28× |
Ratios are MIRA speedups over each baseline, as reported in the manuscript. Hardware, batch policy, context length, precision and implementation affect comparisons.
View the CUDA performance figure (PDF) ↗