Official implementation
PyTorch and CUDA implementations
This page explains the computation behind MIRA: a readable PyTorch reference, an explicit language-model implementation, and CUDA execution of the recurrent dynamics. For the existing pretrainer, use Pretraining MIRA. For concise library usage and architecture composition, use Customise LLM.
From lattice dynamics to an affine transition
The starting point is the lattice scattering update. The control signals at token t determine the rotation and radius before the temporal recurrence is evaluated. Define the channel-diagonal maps:
Each map has shape de × de and acts on the channel axis of a signal with shape de × dm. The token update is:
Vt(k) = Et(k) Ut(k) + Dt(k) Vt−1(k−1)
Ut(K) = x̃t st⊤, Vt(0) = Ut(0)
The stage index belongs to {1, …, K}; the within-token cascade is evaluated from K down to 1. Only the backward states at stages 0 through K−1 must be carried to the next token:
Eliminate the forward-stage dependencies
Substitute the expression for U at the next higher stage repeatedly. The first expansion introduces one delayed state; the next introduces the next delayed state multiplied by the intervening D maps. Continuing to the injected signal gives:
− ∑j=k+1K (∏i=k+1j−1 Dt(i)) Et(j) Vt−1(j−1)
An empty product is the identity map and an empty sum is zero. At k=K this reduces to the injected signal. At k=0 it gives Vt(0) through the boundary condition. Substituting the expansion into the backward-state update gives, for k ∈ {1, …, K}:
+ Dt(k) Vt−1(k−1)
− Et(k) ∑j=k+1K (∏i=k+1j−1 Dt(i)) Et(j) Vt−1(j−1)
Collect the transition and the input contribution
Every term is either linear in the preceding carried state or independent of it. The D and E maps, the projected input x̃t, and the write vector st depend on the current token representation, not on the preceding carried state. Therefore the entire cascade is an affine transformation:
Vt = ΦtVt−1 + St
Φt has shape Kde × Kde; St has shape Kde × dm. Its block row k and block column j refer to carried stages k,j ∈ {0, …, K−1}. The first row follows from the closure Vt(0) = Ut(0):
(St)0 = (∏i=1K Dt(i)) x̃t st⊤
For each remaining carried row k ∈ {1, …, K−1}, collecting the delayed-state terms gives:
(Φt)k,k−1 = Dt(k)
(Φt)k,j = −Et(k) (∏i=k+1j Dt(i)) Et(j+1), j ≥ k
(St)k = Et(k) (∏i=k+1K Dt(i)) x̃t st⊤
Stage K is included in the current token’s ladder readout, but is not an additional carried state. Its value follows directly from Ut(K) and Vt−1(K−1). Because every D and E block is channel-diagonal, the transition separates into independent K × K systems across channels; the large block matrix is a mathematical representation, not a requirement to allocate a dense Kde × Kde tensor.
Why affine transitions compose associatively
Take two transitions in chronological order, t1 < t2. Apply the earlier transition first and substitute its result into the later one:
= Φt₂(Φt₁V + St₁) + St₂
= (Φt₂Φt₁)V + Φt₂St₁ + St₂
The pair composition is consequently:
= (Φt₂Φt₁, Φt₂St₁ + St₂)
Here ⊙ denotes composition of affine transitions; in the lattice equations it denotes a broadcast elementwise product. For three chronologically ordered transitions, either grouping expands to the same pair:
= (Φt₃Φt₂Φt₁, Φt₃Φt₂St₁ + Φt₃St₂ + St₃)
= 𝒯t₃ ⊙ (𝒯t₂ ⊙ 𝒯t₁)
This follows from matrix-multiplication associativity and distributivity. The identity transition is (I, 0). Composition is generally not commutative: regrouping is valid, but swapping tokens changes the result. A parallel scan can therefore combine neighboring pairs in a tree while preserving their temporal order. The equality is exact over real arithmetic; different floating-point evaluation orders can introduce rounding differences.
Separating preceding memory from segment dynamics
Let s be the position immediately before a segment. Compose the transitions of tokens s+1 through t. The segment remains an affine pair:
Ss→t = ∑j=s+1t (ΦtΦt−1⋯Φj+1)Sj
Vt = Φs→tVs + Ss→t
The trailing product is the identity when j=t. The first term transports the memory accumulated before the segment. The second is the contribution generated inside the segment from a zero entry state; it does not depend on earlier tokens. Setting the entry state to zero isolates Ss→t. Propagating the K basis states with the additive injection suppressed, while retaining the same control signals, yields the channel-local transition Φs→t.
This separation is the key to concurrent chunk processing: a chunk can compute its transition and zero-entry contribution before its actual entry state is known. Once the entry state is recovered, the same summary describes the chunk’s correct exit state.
From the algebra to parallel CUDA execution
Partition the sequence into C = ⌈T/L⌉ chunks, each with at most L tokens. The affine derivation leads directly to three CUDA phases:
- Summarize chunks independently. Each chunk computes its affine pair from local token control signals. Its zero-entry state trajectory gives the input contribution; homogeneous propagation gives the transition. Parallel work spans chunks, channels and memory components.
- Scan the chunk boundaries. Compose chunk pairs in chronological order using the associative pair rule. An exclusive prefix gives each chunk the state accumulated before it; the first chunk receives the reset state. Only summaries and boundary states are exchanged.
- Replay chunks concurrently. Restart each chunk from its recovered entry state, restore its interior states and compute the gated ladder readout for every token. Zero-entry interior states from the first phase are not the final token states.
What the GPU needs to exchange
For one sequence and channel, a chunk summary contains a K × K transition and a K × dm input contribution. The same transition acts on every memory component. Boundary composition therefore depends on the number of chunks and state dimensions, rather than on all interior token states. The kernels reuse channel-local transitions and keep carried state in registers where the selected path permits.
Critical path and implementation choices
With sequential composition of C chunk boundaries, the simple three-phase critical path is approximately 2L + C ≈ 2L + T/L, treating K as fixed. Its minimum occurs near L ≈ √(T/2), giving O(√T) depth for that schedule. Associativity also permits a parallel boundary scan: with sufficient workers, the ideal boundary depth is O(log C), giving O(L + log C) depth for the three phases. Actual depth depends on available workers, the chunk assignment and synchronization; these expressions are algorithmic bounds, not latency predictions.
The tensor reference below removes the Python loop over K by solving all forward-stage equations together. That solves the within-token stage dependencies. The affine composition derived above addresses the different problem of across-token parallelism and motivates the CUDA scan implementation described later on this page.
Interactive CUDA execution diagram
Explore the chunk-scan schedule for one sequence, one channel and a slice of memory. Each worker owns two consecutive chunks. The diagram reads from top to bottom: chunk summaries, boundary composition, entry states, and replay. Thread colors identify ownership; dashed arrows show exchanges between workers.
PyTorch-style dynamics
This tensor reference computes all lattice stages together in one batched triangular solve per token, without a Python loop over K. It uses the cauchy_x_hyperbolic control signal maps. It receives projected control signals and write/read vectors; γ is already the SiLU gate. It omits padding masks, output projection, mixed-precision handling and the parallel kernel dispatch.
def lattice_readout(x_tilde, alpha, beta, rho, gamma, s, nu, V):
B, T, d_e = x_tilde.shape
K = V.shape[1]
identity = torch.eye(K + 1, dtype=x_tilde.dtype, device=x_tilde.device)
outputs = []
for t in range(T):
a = alpha[:, t]
b = beta[:, t]
decay = torch.exp(-a.abs())
cosine = 2 * decay / (1 + decay.square())
sine = torch.tanh(a)
radius = 1 / (1 + b.square())
diagonal = radius * cosine
coupling = radius * sine
delayed = V.transpose(1, 2)
injection = x_tilde[:, t, :, None] * s[:, t, None, :]
system = identity - torch.diag_embed(diagonal, offset=1)
rhs = torch.cat((
-coupling[..., None] * delayed,
injection.unsqueeze(2),
), dim=2)
U = torch.linalg.solve_triangular(
system, rhs, upper=True, unitriangular=True
)
levels = torch.cat((
U[:, :, :1],
coupling[..., None] * U[:, :, 1:] + diagonal[..., None] * delayed,
), dim=2)
V = levels[:, :, :K].transpose(1, 2)
read = torch.einsum("bckm,bkm->bkc", levels, nu[:, t])
y = (rho[:, t].transpose(1, 2) * read).sum(dim=1)
outputs.append(y * gamma[:, t])
return torch.stack(outputs, dim=1), VFor independent sequences, initialize V with zeros of shape [B, K, de, dm]. The control signal arrays α and β have shape [B, T, de, K], ρ has [B, T, de, K+1], and ν has [B, T, K+1, dm]. The remaining Python loop advances tokens in causal order. Within each token, the upper-triangular system solves all forward-stage signals together; tensor operations then update every backward state and evaluate the ladder readout. The dedicated CUDA implementation uses its own parallel scan kernels. This reference expects FP32 or FP64 inputs. The current code uses cosine = sech(α) and sine = tanh(α).
Residual layer structure
def forward(x):
x = x + mixer(rmsnorm_1(x))
x = x + feed_forward(rmsnorm_2(x))
return xThe dense feed-forward network uses a SiLU activation between its up and down projections. MiraBlock.post_init(n_layer) initializes lattice parameters and scales the residual output projections for the chosen depth.
PyTorch language-model implementation
The following PyTorch reference defines embeddings, normalization, residual layers, feed-forward networks and the language-model head explicitly. The recurrent computation uses the public MIRA mixer. Together with the CUDA-loading, data and training sections below, these five Python blocks form one complete custom_train.py example.
import argparse
import importlib
import math
import os
from pathlib import Path
import torch
from torch import nn
from torch.nn import functional as F
from tqdm import trange
from mira_llm import MiraMixer
from helpers.token_store import TokenStream
from pretrain.epoch import EpochCorpus
from tools import config_file
class RMSNorm(nn.Module):
def __init__(self, width, eps=1e-8):
super().__init__()
self.weight = nn.Parameter(torch.ones(width))
self.eps = eps
def forward(self, x):
normalized = x.float() * torch.rsqrt(
x.float().square().mean(dim=-1, keepdim=True) + self.eps
)
return (normalized * self.weight.float()).to(x.dtype)
class MiraLayer(nn.Module):
def __init__(self, width, depth, order, d_e, d_r, d_m, ffn_width):
super().__init__()
self.norm_mixer = RMSNorm(width)
self.mixer = MiraMixer(
width=width,
lattice_order=order,
d_e=d_e,
coefficient_bottleneck=d_r,
memory_dimension=d_m,
)
self.mixer.initialize_lattice_parameters()
self.norm_ffn = RMSNorm(width)
self.ffn = nn.Sequential(
nn.Linear(width, ffn_width),
nn.SiLU(),
nn.Linear(ffn_width, width),
)
std = 0.02 / math.sqrt(2 * depth)
nn.init.normal_(self.mixer.out_proj.weight, std=std)
nn.init.normal_(self.ffn[-1].weight, std=std)
def forward(self, x):
x = x + self.mixer(self.norm_mixer(x))
return x + self.ffn(self.norm_ffn(x))
class CustomMiraLLM(nn.Module):
def __init__(self, vocab_size, width, depth, order, d_e, d_r, d_m, ffn_width):
super().__init__()
self.embedding = nn.Embedding(vocab_size, width)
self.layers = nn.ModuleList([
MiraLayer(width, depth, order, d_e, d_r, d_m, ffn_width)
for _ in range(depth)
])
self.norm = RMSNorm(width)
self.head = nn.Linear(width, vocab_size, bias=False)
def forward(self, input_ids, targets=None, target_mask=None):
x = self.embedding(input_ids)
for layer in self.layers:
x = layer(x)
logits = self.head(self.norm(x))
loss = None
if targets is not None:
token_loss = F.cross_entropy(
logits.reshape(-1, logits.size(-1)), targets.reshape(-1), reduction="none"
).view_as(targets)
if target_mask is None:
loss = token_loss.mean()
else:
per_sequence = (token_loss * target_mask).sum(dim=-1)
loss = (per_sequence / target_mask.sum(dim=-1).clamp_min(1)).mean()
return logits, lossCUDA: parallel execution
Each token defines an affine state transition. Adjacent transitions compose associatively: a chunk can summarize how it transforms an incoming state without knowing that state in advance. The manuscript partitions T tokens into chunks of length L and uses three phases:
- Summarize. Process chunks concurrently from zero carried state and compute their transition summaries.
- Compose. Combine summaries at chunk boundaries to recover each chunk’s entry state.
- Replay. Process chunk interiors concurrently from the recovered states and produce the gated ladder readout.
Work is distributed across token chunks, expanded channels and memory components. Carried state is held in registers where the selected kernel permits. See the reported kernel measurements for the hardware and context length used in the manuscript.
CUDA C++ forward pseudocode
The following pseudocode maps the affine analysis to device computation. Each chunk kernel distributes sequence, chunk, channel and memory work across blocks and lanes. K is a compile-time lattice order, so the stage loops are unrolled. The boundary scan preserves chronological order.
Args, Plan, Mat and Vec denote tensor accessors, a launch plan and small register fragments. Indexing, load/store and cooperative reduction helpers abstract the memory layout. The pseudocode describes the forward schedule illustrated above.
Device primitives
struct Stage { float D, E; };
__device__ Stage stage(float alpha, float beta) {
float r = 1.0f / (1.0f + beta * beta);
float decay = expf(-fabsf(alpha));
return {r * 2.0f * decay / (1.0f + decay * decay), r * tanhf(alpha)};
}
template <int K>
__device__ void lattice_step(const Stage* control, float* V,
float U, float* levels) {
#pragma unroll
for (int k = K; k >= 1; --k) {
levels[k] = control[k].E * U + control[k].D * V[k - 1];
U = control[k].D * U - control[k].E * V[k - 1];
}
levels[0] = U;
#pragma unroll
for (int k = 0; k < K; ++k) V[k] = levels[k];
}
struct Map { Mat Phi; Vec S; };
__device__ Map compose(const Map& later, const Map& earlier) {
return {later.Phi * earlier.Phi,
later.Phi * earlier.S + later.S};
}The stage map uses the current hyperbolic rotation. A token step reads all delayed values before replacing the carried state. Affine composition always applies earlier first.
(i) Summarize each chunk
template <int K>
__global__ void summarize_chunks(Args a) {
auto [b, chunk, channel, memory_lane] = coordinates(a);
float V[K] = {}, basis[K] = {}, levels[K + 1];
bool memory_active = memory_lane < a.memory;
bool basis_active = memory_lane < K;
if (basis_active) basis[memory_lane] = 1.0f;
for (int t = first(chunk); t < last(chunk); ++t) {
if (!a.valid(b, t)) continue;
Stage control[K + 1];
load_control_signals(control, a, b, t, channel);
if (memory_active) {
float U = a.x_tilde(b, t, channel) * a.s(b, t, memory_lane);
lattice_step<K>(control, V, U, levels);
}
if (basis_active)
lattice_step<K>(control, basis, 0.0f, levels);
}
if (memory_active) store_column(a.S(b, chunk, channel), memory_lane, V);
if (basis_active)
store_column(a.Phi(b, chunk, channel), memory_lane, basis);
}Zero carried state produces S. Basis propagation uses the same control signals with zero additive injection and produces columns of Φ. The logical lane range covers both K basis columns and dm memory columns. Separate active tests keep basis propagation valid even when K exceeds dm; the layout helpers tile these logical lanes over actual threads.
(ii) Recover chunk entry states
__global__ void compose_boundaries(Args a, int chunks,
int workers, int chunks_per_worker) {
int unit = global_thread_index() / workers;
int worker = threadIdx.x % workers;
int lo = worker * chunks_per_worker;
int hi = min(chunks, lo + chunks_per_worker);
Map own = identity_map();
for (int chunk = lo; chunk < hi; ++chunk)
own = compose(a.chunk_map(unit, chunk), own);
Map inclusive = own;
for (int distance = 1; distance < workers; distance *= 2) {
Map earlier = group_shift_up_sync(inclusive, distance, workers);
if (worker >= distance)
inclusive = compose(inclusive, earlier);
}
Map preceding = group_shift_up_sync(inclusive, 1, workers);
if (worker == 0) preceding = identity_map();
Vec V = preceding.Phi * a.initial_state(unit) + preceding.S;
for (int chunk = lo; chunk < hi; ++chunk) {
a.entry_state(unit, chunk) = V;
Map current = a.chunk_map(unit, chunk);
V = current.Phi * V + current.S;
}
}Workers first fold their local chunks, then scan their pairs, then write the state entering each chunk. group_shift_up_sync reads the preceding scan level before any update is consumed: warp shuffles handle communication within a warp, and synchronized shared memory handles exchanges across warps. Worker groups are power-of-two sized and remain within a block; out-of-range workers contribute identity maps.
(iii) Replay and compute the readout
template <int K>
__global__ void replay_chunks(Args a) {
auto [b, chunk, channel, memory_lane] = coordinates(a);
bool memory_active = memory_lane < a.memory;
float V[K] = {}, levels[K + 1];
if (memory_active)
load_column(V, a.entry_state(b, chunk, channel), memory_lane);
for (int t = first(chunk); t < last(chunk); ++t) {
if (!a.valid(b, t)) {
if (memory_lane == 0) a.y(b, t, channel) = 0.0f;
continue;
}
if (memory_active && a.training && opens_fine_chunk(t))
store_column(a.saved_state(b, fine_index(t), channel), memory_lane, V);
Stage control[K + 1];
load_control_signals(control, a, b, t, channel);
float U = memory_active
? a.x_tilde(b, t, channel) * a.s(b, t, memory_lane) : 0.0f;
lattice_step<K>(control, V, U, levels);
float readout = 0.0f;
#pragma unroll
for (int k = 0; k <= K; ++k) {
float projected = memory_group_sum(
memory_active ? levels[k] * a.nu(b, t, k, memory_lane) : 0.0f);
readout += a.rho(b, t, channel, k) * projected;
}
if (memory_lane == 0)
a.y(b, t, channel) = silu(a.raw_gate(b, t, channel)) * readout;
}
if (memory_active && last(chunk) == a.tokens)
store_column(a.final_state(b, channel), memory_lane, V);
}
template <int K>
void forward(Args a, cudaStream_t stream) {
Plan plan = partition(a);
if (plan.chunks > 1 || a.training) {
summarize_chunks<K><<<plan.chunk_grid, plan.chunk_block, 0, stream>>>(a);
compose_boundaries<<<plan.scan_grid, plan.scan_block, 0, stream>>>(
a, plan.chunks, plan.workers, plan.chunks_per_worker);
}
replay_chunks<K><<<plan.chunk_grid, plan.chunk_block, 0, stream>>>(a);
}The memory-group reduction contracts each stage’s state with its read direction ν. The ladder weights ρ combine stages and the SiLU gate γ selects the output. The following output projection is supplied by the model layer. Saved fine-chunk entry states support recomputation during backward. Launching the phases on one CUDA stream orders their global-memory dependencies. For a single inference chunk, its entry-state accessor supplies the initial state and the host can use replay directly.
CUDA: fused kernels
The current pretraining YAML selects mira_dspv2. This self-contained package builds its own CUDA source into its own cubin. It includes chunked forward and backward paths, a tree scan, and fused scan paths.
Its specialized four-token path forms control signals and memory-read projections within the kernel using tensor cores, while the recurrence accumulates in FP32. Two groups of batch rows run offset token tiles to overlap control signal formation with recurrence work. Weight tiles are shared.
The four-token specialization targets K=2, dm=16 and dr in {16, 32, 48}, with additional dispatch conditions on precision, hardware and batch shape. Other supported cases use the package’s two-token or chunked paths. These dispatch choices preserve the model computation.
Build and load the CUDA kernel
Build the kernel from your clone before running the script:
make -f kernel/cuda/mira_dspv2/Makefile build
make -f kernel/cuda/mira_dspv2/Makefile checkThe package and compiled-kernel directory come from your stage YAML:
mira_cuda_bin: mira_dspv2
mira_cuda_kernel: build/kernel/cudadef load_cuda_from_yaml(config_path, device):
settings = config_file.read(config_path)
package = settings["mira_cuda_bin"]
kernel_root = settings.get("mira_cuda_kernel")
if kernel_root:
os.environ["MIRA_CUDA_KERNEL"] = str(Path(kernel_root).resolve())
torch.cuda.set_device(device)
mixer_module = importlib.import_module(MiraMixer.__module__)
backend = mixer_module.use_mira_cuda(package)
if backend is None:
raise RuntimeError(f"Could not import CUDA package: {package}")
ready, detail = backend.status(torch.cuda.current_device())
if not ready:
raise RuntimeError(f"CUDA kernel could not load: {detail}")
print(f"Loaded CUDA package {package} from {detail}", flush=True)The loader checks that the compiled module can be loaded on the selected GPU. The mixer then dispatches to the package’s supported CUDA paths. The selected geometry and dtype determine which path is used; loading a module alone does not guarantee that every operation uses CUDA kernels.
Load the training dataset
Prepare a token stream with the project’s tokenization facility, as described in Pretraining MIRA. Pass its stream stem, the path before _index.json, to this script. The reader verifies recorded content signatures before opening read-only memory maps.
def load_dataset(token_stem, context_length, vocab_size):
stream = TokenStream(token_stem)
if stream.n_tokens < context_length + 1:
raise ValueError("The token stream is shorter than one training window")
if stream.head["vocab_size"] != vocab_size:
raise ValueError("Token stream and model vocabulary sizes must match")
print(f"Loaded dataset {token_stem}: {stream.n_tokens:,} tokens", flush=True)
return streamPyTorch training loop
This single-device example samples packed windows, shifts inputs and targets for next-token prediction, computes cross-entropy, and updates the model. It reads the learning rate from the YAML and uses a constant rate for this example. It derives one complete corpus pass at the configured context length, masks padding in the final batch, and resumes from an atomic checkpoint containing the architecture, optimizer and random states. The standard trainer provides the full configured schedule, context handling and distributed execution.
def train(model, stream, settings, device, checkpoint_path, seed):
batch_size = int(settings["batch"])
context_length = int(settings["context_window"].split(",")[0])
data = EpochCorpus(stream, [context_length], batch_size)
total_steps = data.steps
checkpoint_every = int(settings["ckpt_every"])
if checkpoint_every < 1:
raise ValueError("ckpt_every must be positive")
optimizer = torch.optim.AdamW(
model.parameters(), lr=float(settings["peak_lr"]),
weight_decay=float(settings["weight_decay"]),
)
generator = torch.Generator().manual_seed(seed)
checkpoint_path = Path(checkpoint_path)
checkpoint_path.parent.mkdir(parents=True, exist_ok=True)
first_step = 0
if checkpoint_path.exists() and settings.get("no_resume", "0") == "0":
saved = torch.load(checkpoint_path, map_location="cpu", weights_only=True)
if saved["settings"] != settings or saved["architecture"] != model.architecture:
raise ValueError("Checkpoint configuration does not match this run")
if saved["dataset_sig"] != stream.sig:
raise ValueError("Checkpoint dataset does not match this token stream")
model.load_state_dict(saved["model"])
optimizer.load_state_dict(saved["optimizer"])
generator.set_state(saved["data_rng"])
torch.set_rng_state(saved["torch_rng"])
if device.type == "cuda":
torch.cuda.set_rng_state_all(saved["cuda_rng"])
first_step = saved["step"]
print(f"Loaded checkpoint {checkpoint_path} at step {first_step}", flush=True)
data.step = first_step
model.train()
for step in trange(first_step, total_steps, desc="Training"):
ids, _, mask = data.batch(batch_size, context_length - 1, device=device)
optimizer.zero_grad(set_to_none=True)
_, loss = model(ids[:, :-1], ids[:, 1:], mask[:, 1:])
loss.backward()
grad_clip = float(settings["grad_clip"])
if grad_clip > 0:
nn.utils.clip_grad_norm_(model.parameters(), grad_clip)
optimizer.step()
print(f"step={step + 1} loss={loss.item():.6f}", flush=True)
if (step + 1) % checkpoint_every == 0 or step + 1 == total_steps:
temporary = checkpoint_path.with_suffix(".tmp")
payload = {
"step": step + 1,
"architecture": model.architecture,
"settings": settings,
"dataset_sig": stream.sig,
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"data_rng": generator.get_state(),
"torch_rng": torch.get_rng_state(),
"cuda_rng": torch.cuda.get_rng_state_all() if device.type == "cuda" else [],
}
with temporary.open("wb") as handle:
torch.save(payload, handle)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, checkpoint_path)Run the implementation
The entry point creates CustomMiraLLM directly, opens the dataset and enters the training loop. Geometry and vocabulary size are explicit arguments; the kernel package is selected only in YAML.
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--config", required=True)
parser.add_argument("--tokens", required=True)
parser.add_argument("--vocab-size", type=int, required=True)
for name in ("width", "depth", "order", "d-e", "d-r", "d-m", "ffn-width"):
parser.add_argument("--" + name, type=int, required=True)
parser.add_argument("--device", default="cuda:0")
parser.add_argument("--checkpoint", default="checkpoints/custom_mira/model.pt")
args = parser.parse_args()
settings = config_file.read(args.config)
seed = int(settings.get("seed", "0"))
torch.manual_seed(seed)
device = torch.device(args.device)
if device.type == "cuda":
load_cuda_from_yaml(args.config, device)
architecture = {
"vocab_size": args.vocab_size, "width": args.width, "depth": args.depth,
"order": args.order, "d_e": args.d_e, "d_r": args.d_r,
"d_m": args.d_m, "ffn_width": args.ffn_width,
}
if any(value < 1 for value in architecture.values()):
raise ValueError("All architecture dimensions must be positive")
model = CustomMiraLLM(**architecture).to(device)
model.architecture = architecture
print("Created CustomMiraLLM with randomly initialized weights", flush=True)
context_length = int(settings["context_window"].split(",")[0])
stream = load_dataset(args.tokens, context_length, args.vocab_size)
train(model, stream, settings, device, args.checkpoint, seed)
if __name__ == "__main__":
main()python custom_train.py --config config/pretrain/config_pretrain.yaml --tokens "$TOKEN_STREAM_STEM" --vocab-size "$VOCAB_SIZE" --width "$MODEL_WIDTH" --depth "$MODEL_DEPTH" --order "$LATTICE_ORDER" --d-e "$EXPANDED_WIDTH" --d-r "$WRITE_BOTTLENECK" --d-m "$MEMORY_DIMENSION" --ffn-width "$FFN_WIDTH"Set the shell variables to your tokenizer, token-stream stem and chosen geometry. Run from the clone root so relative YAML paths resolve correctly. This example handles packed, unpadded windows; use the standard trainer for the project’s full pretraining workflow.