Training Pipeline & Checkpoint Provenance

Unlike projects that release unverified random weights, Q-TensorFormer provides a reproducible causal language model training pipeline, a canonical tokenizer, and verified trained checkpoints with SHA-256 provenance.

1. The Canonical Tokenizer (QTensorFormerTokenizer)

The canonical tokenizer (src/tokenizer.py) provides deterministic, self-contained subword and character tokenization without external tokenization dependencies:

Fixed Special Tokens: <pad> = 0, <bos> = 1, <eos> = 2, <unk> = 3.
Printable ASCII Foundation: Slots 4 to 101 deterministically map all standard whitespace and printable ASCII characters.
Subword Longest-Match: Remaining vocabulary slots up to vocab_size = 256 are built from frequent space-prefixed words and n-grams from the corpus.
Lossless Round-Trip: Printable ASCII text decodes back to the exact input string (verified in tests/test_tokenizer_and_training.py).
JSON Persistence: Saved to checkpoints/tokenizer/tokenizer.json with SHA-256 integrity checksums.

2. Educational Pretraining Corpus

The canonical checkpoint was trained on data/qtensorformer_corpus.txt, an open-source technical text corpus explaining Transformer attention, tensor decompositions, memory hierarchies, and edge AI deployment.

Corpus Statistics:

Total Sentences: 46
Total Characters: 5,453
Total Tokenized Tokens: 2,905
Deterministic Splits: 70% Train (2,033 tokens), 15% Validation (435 tokens), 15% Test (437 tokens)

3. Optimization Hyperparameters

Training is executed via LanguageModelTrainer (src/training.py) with the following verified hyperparameters:

optimizer: AdamW (learning_rate = 1e-3, weight_decay = 0.01)
lr_scheduler: Cosine with 10 warmup steps (lr_min_factor = 0.1)
gradient_clipping: max_grad_norm = 1.0
batch_size: 8 (sequence_length = 64, stride = 16)
epochs: 12
loss_function: CrossEntropyLoss(ignore_index = 0) # Pads ignored

Training Outcome: Over 12 epochs, test loss dropped from 5.8646 (perplexity 352.35) to 3.8086 (perplexity 45.09), achieving an 87.2% reduction in perplexity.

4. Knowledge Distillation Pipeline

For training small student models, DistillationTrainer (scripts/distill.py) pairs a dense teacher (e.g. DenseBaseline) with a compressed Q-TensorFormer student, optimizing a joint loss:

Loss = α * T^2 * KL(Student / T || Teacher / T) + (1 - α) * CrossEntropy(Student, y)

Tested with temperature T = 3.0 and distillation weight α = 0.5.

5. How to Retrain the Canonical Model

Run the training pipeline with one command:

python scripts/train_and_evaluate.py

This deterministically retrains the model, verifies loss reduction, saves checkpoints/qtensorformer_canonical_trained.pt, and exports the Hugging Face package to checkpoints/hf_canonical_trained/.