MIRA / Introduction

Signal-processing-inspired large language models.

Signal-processing-inspired large language models

MIRA is a signal-processing-inspired large language model architecture. Its lattice–ladder dynamics provide expressive recurrent memory without pole-geometry bias. Input-conditioned rotations and radii govern the dynamics; a selective readout combines information across the lattice stages.

This introduction presents the dynamics and readout. Read Motivation for the signal-processing perspective behind the architecture.

Stable recurrent lattice dynamics

Normalize the token representation and project it into the expanded channel space:

zt = RMSNorm(xt),   x̃t = Win zt
st = SiLU(Ws zt + bs),   Ut(K) = x̃t st⊤

For token positions t ∈ {1, …, T}, each lattice stage couples the forward token signal U with the delayed backward state V. The stage recursion is evaluated from k = K down to 1:

Ut(k−1) = rt(k) ⊙ (cos θt(k) ⊙ Ut(k) − sin θt(k) ⊙ Vt−1(k−1))
Vt(k) = rt(k) ⊙ (sin θt(k) ⊙ Ut(k) + cos θt(k) ⊙ Vt−1(k−1))

The forward and backward signals have shape de × dm. Channel-wise radii and rotations broadcast over the memory dimension. The boundary conditions are Vt(0) = Ut(0) and V0(k) = 0 for an independent sequence.

Input-conditioned control signals

ct = SiLU(Wc zt + bc)
αt = Wα ct + bα,   βt = Wβ ct + bβ
cos θt = tanh(αt),   sin θt = sech(αt),   rt = 1 / (1 + βt2)

These maps transform the control signals into the lattice rotation and radius. The write bottleneck has dimension dr, and the expanded dynamics have de channels. Rotation preserves norm; the radius is bounded by one. Memory carried between tokens has K × de × dm values per layer and sequence, independent of context length.

The ladder readout

yt = Wout ((∑k=0K ρt(k) ⊙ (Vt(k) νt(k))) ⊙ γt)

The input-conditioned control signals ρ weight each stage, ν selects a direction in its memory dimension, and γ = SiLU(Wγzt + bγ) gates the result before output projection. The readout includes stages 0 through K.

A complete MIRA layer adds residual connections around the normalized mixer and the normalized feed-forward network. Follow the implementation →