Signal-processing-inspired large language models
MIRA is a signal-processing-inspired large language model architecture. Its lattice–ladder dynamics provide expressive recurrent memory without pole-geometry bias. Input-conditioned rotations and radii govern the dynamics; a selective readout combines information across the lattice stages.
This introduction presents the dynamics and readout. Read Motivation for the signal-processing perspective behind the architecture.
Stable recurrent lattice dynamics
Normalize the token representation and project it into the expanded channel space:
st = SiLU(Ws zt + bs), Ut(K) = x̃t st⊤
For token positions t ∈ {1, …, T}, each lattice stage couples the forward token signal U with the delayed backward state V. The stage recursion is evaluated from k = K down to 1:
Vt(k) = rt(k) ⊙ (sin θt(k) ⊙ Ut(k) + cos θt(k) ⊙ Vt−1(k−1))
The forward and backward signals have shape de × dm. Channel-wise radii and rotations broadcast over the memory dimension. The boundary conditions are Vt(0) = Ut(0) and V0(k) = 0 for an independent sequence.
Input-conditioned control signals
αt = Wα ct + bα, βt = Wβ ct + bβ
cos θt = tanh(αt), sin θt = sech(αt), rt = 1 / (1 + βt2)
These maps transform the control signals into the lattice rotation and radius. The write bottleneck has dimension dr, and the expanded dynamics have de channels. Rotation preserves norm; the radius is bounded by one. Memory carried between tokens has K × de × dm values per layer and sequence, independent of context length.
The ladder readout
The input-conditioned control signals ρ weight each stage, ν selects a direction in its memory dimension, and γ = SiLU(Wγzt + bγ) gates the result before output projection. The readout includes stages 0 through K.
A complete MIRA layer adds residual connections around the normalized mixer and the normalized feed-forward network. Follow the implementation →