MIRA / Pretraining MIRA

Pretraining MIRA

Pretraining from scratch: download data, train BPE, tokenize data and pretrain MIRA.

Pretraining from scratch

Pretraining MIRA from scratch has four stages, in this order:

  1. Download data. Obtain the raw training corpus.
  2. Train BPE. Learn the tokenizer vocabulary and merge rules from the corpus.
  3. Tokenize data. Encode the corpus into verified, reusable token streams.
  4. Pretrain. Initialize the language model and train for exactly one corpus pass.

Installation, YAML configuration and building the CUDA package are setup tasks. Complete them before running the four stages below.

Install MIRA

shell
python -m pip install .

Run from your clone root with Python 3.10 or newer and the dependencies required by your CUDA environment. Use the existing project facilities below; no custom model or training script is needed.

Configure mira-m

YAML location: config/default/config_mira.yaml

The mira-m section defines the model architecture. Edit this named section to configure the number of layers, model width, feed-forward width, lattice geometry and control signal maps. The full active section is shown below.

yaml
# --------------------------------------------------------------------------------------------------
# mira-m  (K = 2, d = 1024, d_e = 2048, d_r = 64, d_m = 64, 24 layers)
#   parameters per MIRA layer            15,764,736
#     mixer                               7,368,960
#     FFN                                 8,393,728
#     two RMSNorm                             2,048
#   embedding (bpe V = 50,259, tied)     51,465,216
#   mask embedding + final RMSNorm            3,072
#   total parameters                    429,821,952   = 24 x 15,764,736 + 51,465,216 + 3,072
#   state per layer                         262,144   = K x d_e x d_m = 2 x 2048 x 64
#   state per sequence                    6,291,456   = n_layers x K x d_e x d_m
#   state-to-parameter ratio               0.014637
# --------------------------------------------------------------------------------------------------
mira-m:
  n_layers:                    24
  d_model:                     1024
  ffn_width:                   4096
  lattice_d_e:                 2048
  lattice_d_r:                 64
  lattice_d_m:                 64
  lattice_K:                   2
  lattice_modulus_map:         cauchy
  lattice_argument_map:        hyperbolic
  lattice_initializer:         W_uniform_b_jeffreys
  lattice_init_horizon:
    default:  [1,2000]
    L0:     [1,128]
    #L1:    [2,256]
    #L2:    [4,512]
    #L3:    [8,1024]
  #moe_total_experts:          1
  #moe_layers:
  #moe_active_experts:
  #moe_expert_width:
  #moe_shared_experts:         0
  #lattice_init_freq:
  #   default:

n_layers sets the number of layers, d_model the representation width, and ffn_width the feed-forward hidden width. lattice_K sets the lattice order; lattice_d_e, lattice_d_r and lattice_d_m set the expanded channels, write bottleneck and per-channel memory. The modulus and argument maps specify how control signals determine lattice radii and rotations. lattice_init_horizon specifies the initializer’s horizon range, with L0 overriding the first layer.

The pretraining YAML selects this section with mira_config: mira-m. Architecture settings belong in this model file; corpus, batch, epoch and reporting settings belong in the pretraining file below.

Configure the pretraining YAML

YAML location: config/pretrain/config_pretrain.yaml

The existing pretrainer reads config/pretrain/config_pretrain.yaml. Edit that file directly; no configuration wizard or custom training script is required. The complete configuration below includes the pretraining, tokenizer and token-stream settings used by the four-stage workflow. Checkpointing and plotting are set to 2,000 steps. Comments describe supported choices and the generated file locations. Copy it into the YAML file and adjust the model, corpus and runtime settings for your run.

yaml
# Runtime
py: python                 # Python executable
gpu: cuda                  # auto | cuda | npu | mps | cpu
seed: 0                    # Integer random seed

# Data and corpus parsing
pretrain_dir: data/download/llm-pretrain-fineweb-edu-100BT
corpus_text_column: text    # Field containing text in structured corpus records
corpus_max_words: 344       # Positive document-packing word budget
corpus_exclude: test,valid,validation # Comma-separated directory names excluded from scanning

# BPE tokenizer training
bpe_merges: 50000           # Number of BPE merges to learn
bpe_min_freq: 2             # Minimum frequency for a candidate merge
libbpe: build/tokenizer/cpp/libbpe.so # Optional compiled tokenizer library
bpe_flags:                 # Optional additional BPE-stage command-line arguments

# Token-stream preparation
tokens_shard_mb: 100        # Target token-shard size in MB
tokens_verify: md5         # Content signatures are required before a consumer maps tokens
tokenize_flags:            # Optional additional tokenization-stage arguments

# Model selection and training context
mira_config: mira-m         # mira-xs | mira-s | mira-m | mira-l
context_window: 4096        # One fixed training context; 4,096 tokens

# CUDA kernels
mira_cuda_bin: mira_dspv2    # mira_dspv1 | mira_dspv2
mira_cuda_kernel: build/kernel/cuda # Compiled-kernel directory

# Training duration: exactly one epoch
pretrain_epochs: 1          # Exactly one complete corpus pass; only 1 is supported
pretrain_steps:             # Leave blank in epoch mode; the trainer derives the step count

# Full-batch execution and memory
batch: 16                  # Logical batch shared by four channel-sharded H100s
mixed_precision: 1         # 1 = supported bfloat16 autocast; 0 = FP32
grad_checkpoint: 1         # 1 = recompute block activations in backward; 0 = retain them
loss_chunk: 0              # 0 = no loss chunking; positive values select a loss-slice token count

# Distributed execution
distributed_parallelism: fsdp-channel # ddp | fsdp-layer | fsdp-channel; explained below

# Optimization
peak_lr: 5e-4              # Peak learning rate
final_lr: 1e-6             # Final learning rate for cosine decay
lr_schedule: cosine       # cosine | constant
warmup_steps: 1000         # Number of linear-warmup steps; 0 disables warmup
adam_beta1: 0.9            # First-moment decay, in [0, 1)
adam_beta2: 0.99           # Second-moment decay, in [0, 1)
weight_decay: 1e-2         # Decoupled weight decay
grad_clip: 1.0             # Global gradient-norm limit; 0 disables clipping
adam_eps: 1e-8             # Positive numerical-stability constant

# Checkpoints, history, figures and resume
ckpt_every: 2000           # Save checkpoints and durable training history every 2,000 steps
plot_every: 2000           # Update training figures every 2,000 steps; 0 disables periodic plots
profile_every_steps: 2000  # Profile dynamics every 2,000 steps
no_resume: 0               # 0 = resume a compatible checkpoint; 1 = disable resume
# Checkpoint: checkpoints/pretrain/<run>/checkpoint.pt
# Training history: checkpoints/pretrain/<run>/history.json
# Training figures: outputs/plots/pretrain/
# BPE tokenizer: checkpoints/bpe/bpe.json
# Token streams: cache/tokens/

# Inference and evaluation cache settings
prefix_cache: 1            # 1 = enable the prefix cache; 0 = disable
prefix_cache_size: 2       # Prefix-cache memory ceiling in GiB
hf_max_context: 262144     # Context ceiling reported by Hugging Face export
kv_cache: 1                # 1 = incremental generation cache; 0 = recompute the prefix
kv_cache_size: 2.0          # Generation cache ceiling in GiB

# Advanced options
extra_set:                 # Optional SECTION.key=value overrides with --set
common_flags:              # Additional arguments shared by the stage entry points
pretrain_flags:            # Additional arguments passed only to pretraining

Set the model and corpus

mira_config selects a named architecture: mira-xs, mira-s, mira-m or mira-l. pretrain_dir must point to your prepared local corpus. When changing the named architecture, also review the context lengths, batch size and epoch setting already written into the YAML; changing the name does not erase explicit values in the file.

Set sequence lengths and memory use

context_window: 4096 uses one fixed training context of 4,096 tokens. For four H100 GPUs with 80 GB each, this configuration uses a logical batch: 16 with a single context_window: 4096. Under fsdp-channel all four ranks process the same logical batch with channels partitioned across GPUs; the batch is not multiplied by four. Each optimizer step processes the complete batch of 16 sequences, or 65,536 input tokens, across the channel-sharded model. Gradient checkpointing is enabled and loss chunking is disabled. The vocabulary projection and loss are replicated across ranks, so four GPUs do not provide four times the memory for those tensors. This batch is a conservative memory-based configuration; its peak memory and throughput still require validation on the target H100 system. mixed_precision: 1 enables mixed precision; grad_checkpoint: 1 enables gradient checkpointing if required. loss_chunk: 0 keeps loss chunking disabled.

Choose CUDA and distributed execution

mira_cuda_bin selects the CUDA package in YAML, and mira_cuda_kernel points to the compiled-kernel directory. Build and verify it before training; see Code.

ModeWhat is distributedData and batch
ddpEvery GPU holds a complete model; gradients are averaged before each optimizer update.Each GPU reads different examples. With four GPUs, the global batch is four times the configured batch.
fsdp-layerOne process places consecutive model layers on different GPUs and transfers activations between them.One logical batch passes through the layers in sequence. The implementation does not overlap pipeline stages, so devices can wait for each other.
fsdp-channelOne process per GPU splits lattice channels and feed-forward hidden units across ranks; partial results are combined within each layer.All ranks read the same logical batch. Four GPUs cooperate on batch 16; the global batch remains 16.

The configured four-H100 run uses fsdp-channel. Embeddings, the output head and some other tensors remain replicated. The names fsdp-layer and fsdp-channel describe this project’s layer and channel partitioning; they are not interchangeable with conventional fully sharded data parallelism.

Set the run budget and persistence

The trainer writes checkpoint.pt and history.json under checkpoints/pretrain/<run>/, and figures under outputs/plots/pretrain/. The run name is derived from the selected model and architecture. These roots are derived by the current facility, so they are shown as path comments rather than unsupported YAML settings.

pretrain_epochs: 1 requests exactly one corpus pass. Leave pretrain_steps blank: the trainer derives the optimizer-step count from the verified token counts, fixed context and logical batch. Each within-stream next-token target is used once; the first token supplies context, and incomplete final windows or batches are padded and masked. An epoch checkpoint records the corpus signatures and batching plan, and resume rejects a different plan. ckpt_every: 2000 and plot_every: 2000 save checkpoints and figures every 2,000 steps. Keep no_resume: 0 to resume an existing run; no_resume: 1 disables resume.

The trainer applies built-in defaults, then the selected YAML, then explicit command-line settings. A nonblank value in YAML overrides the corresponding default. Optimization settings such as the learning-rate schedule remain in the supplied template.

1. Download data

Input: the dataset entry selected in the downloader. Output: the raw corpus in a local directory. The example uses FineWeb-Edu sample-100BT, matching pretrain_dir in the YAML.

shell
python -m tools.download_data --only llm-pretrain-fineweb-edu-100BT

This stage downloads data. It does not train the tokenizer or the language model. The downloader resumes supported partial downloads.

2. Train BPE

Input: the downloaded text corpus. Output: the BPE tokenizer, including its vocabulary and merge rules. This stage learns how text is represented as token IDs; it does not update language-model weights.

shell
python -m tokenizer.run --config config/pretrain/config_pretrain.yaml

A fresh run trains the tokenizer. An interrupted tokenizer build resumes from its saved state; an existing compatible tokenizer is reused. Use the same tokenizer for corpus tokenization and pretraining.

3. Tokenize data

Input: the raw corpus and trained BPE tokenizer. Output: token shards and manifests under cache/tokens/. This stage encodes the corpus once so pretraining can read it without repeating tokenization.

shell
python -m tokenize_data.run --config config/pretrain/config_pretrain.yaml

Pretraining verifies the recorded content signatures before opening the token streams as read-only memory maps. A damaged stream stops the run instead of being repaired by the trainer.

4. Pretrain MIRA

Input: the verified token streams, matching tokenizer and configured model. Output: model checkpoints, training history and figures. For four H100 GPUs with 80 GB each, launch one process per GPU:

shell
torchrun --standalone --nproc_per_node=4 -m pretrain.run --config config/pretrain/config_pretrain.yaml

pretrain_epochs: 1 derives the optimizer-step count from the corpus and uses every within-stream next-token target exactly once. Padding in the final partial batch is masked. The initial token of each stream supplies context. With fsdp-channel, all four GPUs cooperate on the same batch of 16 sequences at 4,096 tokens; the logical batch is not multiplied by four.

On a fresh run, the model is randomly initialized. With no_resume: 0, an existing compatible checkpoint resumes at the next unconsumed part of the epoch. The checkpoint records the corpus signatures and plan so a different dataset or batch plan cannot silently continue the run. For a custom architecture, see Customise LLM.