MIRA / Customise LLM

Build your own LLM with MIRA.

Install the package, import the building blocks and compose your architecture.

1. Install

shell
python -m pip install .

Run inside your local clone to install MIRA into your Python environment.

2. Import

python
from mira_llm import MiraMixer, MiraBlock, MiraLLM

MiraMixer provides the recurrent mixing operation. MiraBlock provides a complete residual layer with normalization and a feed-forward network. MiraLLM provides the project’s complete language model.

3. Compose your architecture

Use ordinary PyTorch modules around the imported blocks. This example defines a model directly, with its vocabulary, depth and lattice geometry supplied by the caller.

python
import torch
from torch import nn
from mira_llm import MiraBlock


class MyLLM(nn.Module):
    def __init__(self, vocab_size, width, depth, order, d_e, d_r, d_m):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, width)
        self.layers = nn.ModuleList([
            MiraBlock(
                width=width,
                lattice_order=order,
                d_e=d_e,
                coefficient_bottleneck=d_r,
                memory_dimension=d_m,
                ffn_hidden=4 * width,
            ).post_init(depth)
            for _ in range(depth)
        ])
        self.norm = nn.RMSNorm(width, eps=1e-8)
        self.head = nn.Linear(width, vocab_size, bias=False)

    def forward(self, input_ids):
        x = self.embedding(input_ids)
        for layer in self.layers:
            x = layer(x)
        return self.head(self.norm(x))

Pass token IDs with shape [batch, sequence length]; the model returns logits with shape [batch, sequence length, vocabulary size]. This example uses independent, unpadded sequences. It contains no dataset or training framework: connect it to your own application.

For a complete reference with a dataset reader, training loop and CUDA loader, see Code. To train the supplied model using the project’s facilities, see Pretraining MIRA.