efe-T/LatentMqa-eperiment-A

Latent MQA Experiment A

Status: trainer audited and ready; continued-pretraining weights have not been produced yet.

This repository is the training and checkpoint destination for a controlled continued-pretraining experiment based on efe-T/Experiment-1-A. The run uses pure trainable xIELU and moves value-embedding projections into per-layer 128-wide MQA KV-latent tables.

The standalone GptTrainer.py recipe is pinned to an NVIDIA A100 (sm_80), BF16, the audited base revision, and FineWeb-Edu shards 000011 through 000031. It consumes exactly 2,000,158,720 new target tokens in monotonic order without wrapping.

Run in Colab

1. Select an A100 GPU runtime.
2. Add a private Colab secret named HF_TOKEN with write access to this repo.
3. Upload and run the complete GptTrainer.py file (or paste the complete file into one cell).

On every start, the trainer scans every complete checkpoint manifest in this repository and resumes the one with the greatest verified token cursor. The checkpoints/latest/manifest.json pointer is checked, but a stale pointer cannot force an older checkpoint or replay an already checkpointed range. Divergent or invalid checkpoint histories stop the run instead of guessing.

The next prefetched-but-unconsumed microbatch, optimizer and scheduler state, RNG state, and a per-step first/last data-range ledger are saved together. A SIGINT/SIGTERM finishes the current optimizer step and waits for the checkpoint and pointer to be committed before exiting. Once the planned token budget is complete, later launches exit before opening a dataset shard.

Otherwise, when no checkpoint exists, the trainer downloads and verifies the pinned base model before starting.

The eventual model is custom PyTorch code and raw safetensors, not a drop-in Transformers model.