1. Install
python -m pip install .Run inside your local clone to install MIRA into your Python environment.
2. Import
from mira_llm import MiraMixer, MiraBlock, MiraLLMMiraMixer provides the recurrent mixing operation. MiraBlock provides a complete residual layer with normalization and a feed-forward network. MiraLLM provides the project’s complete language model.
3. Compose your architecture
Use ordinary PyTorch modules around the imported blocks. This example defines a model directly, with its vocabulary, depth and lattice geometry supplied by the caller.
import torch
from torch import nn
from mira_llm import MiraBlock
class MyLLM(nn.Module):
def __init__(self, vocab_size, width, depth, order, d_e, d_r, d_m):
super().__init__()
self.embedding = nn.Embedding(vocab_size, width)
self.layers = nn.ModuleList([
MiraBlock(
width=width,
lattice_order=order,
d_e=d_e,
coefficient_bottleneck=d_r,
memory_dimension=d_m,
ffn_hidden=4 * width,
).post_init(depth)
for _ in range(depth)
])
self.norm = nn.RMSNorm(width, eps=1e-8)
self.head = nn.Linear(width, vocab_size, bias=False)
def forward(self, input_ids):
x = self.embedding(input_ids)
for layer in self.layers:
x = layer(x)
return self.head(self.norm(x))Pass token IDs with shape [batch, sequence length]; the model returns logits with shape [batch, sequence length, vocabulary size]. This example uses independent, unpadded sequences. It contains no dataset or training framework: connect it to your own application.
For a complete reference with a dataset reader, training loop and CUDA loader, see Code. To train the supplied model using the project’s facilities, see Pretraining MIRA.