Text Generation
Transformers
Safetensors
PyTorch
ONNX
English
qtensorformer
q-tensorformer
high-throughput-inference
tensor-train
tensor-networks
tensorrt-llm
vllm
continuous-batching
paged-attention
speculative-decoding
zero-svd
adaptive-computation
dynamic-neural-networks
kv-cache
kv-cache-quantization
attention-sinks
lyapunov-stability
control-theory
contextual-bandits
edge-ai
edge-inference
on-device-ai
model-compression
hopper-h100
blackwell-b200
jetson-orin
jetson-thor
grouped-query-attention
rotary-position-embeddings
custom_code
academic-paper
ieee
neurips
vit-ap-university
deepseek
deepseek-r1
reasoning
chain-of-thought
gsm8k
openai-compatible
fastapi
rest-api
sse-streaming
onnx-runtime
embedded-ai
hardware-telemetry
power-efficiency
green-ai
raspberry-pi
smollm2
qwen2.5
llama-3.2
apple-silicon
Premchandyadav369
feat(seo): maximize discoverability, Google Scholar metadata, interactive HF widget, LLM crawler schemas, zero em dashes, and one-click quickstart downloads
5175adb | <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Training Pipeline & Provenance - Q-TensorFormer</title> | |
| <meta name="description" content="Causal language model training pipeline, tokenizer architecture, knowledge distillation, and checkpoint provenance for Q-TensorFormer."> | |
| <link rel="canonical" href="https://premchan369.github.io/Q-TensorFormer/training.html"> | |
| <meta property="og:type" content="article"> | |
| <meta property="og:site_name" content="Q-TensorFormer Research Project"> | |
| <meta property="og:title" content="Training Pipeline & Provenance - Q-TensorFormer"> | |
| <meta property="og:description" content="Reproducible language model training, distillation, and dataset provenance."> | |
| <meta property="og:url" content="https://premchan369.github.io/Q-TensorFormer/training.html"> | |
| <script type="application/ld+json"> | |
| { | |
| "@context": "https://schema.org", | |
| "@type": "TechArticle", | |
| "headline": "Q-TensorFormer Language Model Training Pipeline and Provenance", | |
| "description": "Details on the training loop, AdamW schedule, knowledge distillation, and tokenizer validation.", | |
| "url": "https://premchan369.github.io/Q-TensorFormer/training.html", | |
| "author": { "@type": "Person", "name": "Premchand Yadav" }, | |
| "datePublished": "2026-10-13" | |
| } | |
| </script> | |
| <link rel="stylesheet" href="assets/style.css"> | |
| </head> | |
| <body> | |
| <header> | |
| <div class="header-container"> | |
| <div class="logo"> | |
| <a href="index.html"><span>Q-TensorFormer</span> Research</a> | |
| </div> | |
| <nav> | |
| <a href="index.html">Overview</a> | |
| <a href="architecture.html">Architecture</a> | |
| <a href="methodology.html">Methodology</a> | |
| <a href="benchmarks.html">Benchmarks</a> | |
| <a href="edge-inference.html">Edge Inference</a> | |
| <a href="training.html" class="active">Training</a> | |
| <a href="documentation.html">Docs & API</a> | |
| <a href="research.html">Research</a> | |
| <a href="limitations.html">Limitations</a> | |
| <a href="contributing.html">Contributing</a> | |
| </nav> | |
| </div> | |
| </header> | |
| <main class="container"> | |
| <h1>Training Pipeline & Checkpoint Provenance</h1> | |
| <p class="lead"> | |
| Unlike projects that release unverified random weights, Q-TensorFormer provides a reproducible causal language model training pipeline, a canonical tokenizer, and verified trained checkpoints with SHA-256 provenance. | |
| </p> | |
| <section> | |
| <h2>1. The Canonical Tokenizer (QTensorFormerTokenizer)</h2> | |
| <p> | |
| The canonical tokenizer (<code>src/tokenizer.py</code>) provides deterministic, self-contained subword and character tokenization without external tokenization dependencies: | |
| </p> | |
| <ul> | |
| <li><strong>Fixed Special Tokens:</strong> <code><pad> = 0</code>, <code><bos> = 1</code>, <code><eos> = 2</code>, <code><unk> = 3</code>.</li> | |
| <li><strong>Printable ASCII Foundation:</strong> Slots 4 to 101 deterministically map all standard whitespace and printable ASCII characters.</li> | |
| <li><strong>Subword Longest-Match:</strong> Remaining vocabulary slots up to <code>vocab_size = 256</code> are built from frequent space-prefixed words and n-grams from the corpus.</li> | |
| <li><strong>Lossless Round-Trip:</strong> Printable ASCII text decodes back to the exact input string (verified in <code>tests/test_tokenizer_and_training.py</code>).</li> | |
| <li><strong>JSON Persistence:</strong> Saved to <code>checkpoints/tokenizer/tokenizer.json</code> with SHA-256 integrity checksums.</li> | |
| </ul> | |
| </section> | |
| <section> | |
| <h2>2. Educational Pretraining Corpus</h2> | |
| <p> | |
| The canonical checkpoint was trained on <code>data/qtensorformer_corpus.txt</code>, an open-source technical text corpus explaining Transformer attention, tensor decompositions, memory hierarchies, and edge AI deployment. | |
| </p> | |
| <div class="callout"> | |
| <p><strong>Corpus Statistics:</strong></p> | |
| <ul> | |
| <li>Total Sentences: 46</li> | |
| <li>Total Characters: 5,453</li> | |
| <li>Total Tokenized Tokens: 2,905</li> | |
| <li>Deterministic Splits: 70% Train (2,033 tokens), 15% Validation (435 tokens), 15% Test (437 tokens)</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>3. Optimization Hyperparameters</h2> | |
| <p> | |
| Training is executed via <code>LanguageModelTrainer</code> (<code>src/training.py</code>) with the following verified hyperparameters: | |
| </p> | |
| <pre><code>optimizer: AdamW (learning_rate = 1e-3, weight_decay = 0.01) | |
| lr_scheduler: Cosine with 10 warmup steps (lr_min_factor = 0.1) | |
| gradient_clipping: max_grad_norm = 1.0 | |
| batch_size: 8 (sequence_length = 64, stride = 16) | |
| epochs: 12 | |
| loss_function: CrossEntropyLoss(ignore_index = 0) # Pads ignored</code></pre> | |
| <p> | |
| <strong>Training Outcome:</strong> Over 12 epochs, test loss dropped from <code>5.8646</code> (perplexity <code>352.35</code>) to <code>3.8086</code> (perplexity <code>45.09</code>), achieving an <strong>87.2% reduction in perplexity</strong>. | |
| </p> | |
| </section> | |
| <section> | |
| <h2>4. Knowledge Distillation Pipeline</h2> | |
| <p> | |
| For training small student models, <code>DistillationTrainer</code> (<code>scripts/distill.py</code>) pairs a dense teacher (e.g. <code>DenseBaseline</code>) with a compressed Q-TensorFormer student, optimizing a joint loss: | |
| </p> | |
| <pre><code>Loss = α * T^2 * KL(Student / T || Teacher / T) + (1 - α) * CrossEntropy(Student, y)</code></pre> | |
| <p> | |
| Tested with temperature <code>T = 3.0</code> and distillation weight <code>α = 0.5</code>. | |
| </p> | |
| </section> | |
| <section> | |
| <h2>5. How to Retrain the Canonical Model</h2> | |
| <p>Run the training pipeline with one command:</p> | |
| <pre><code>python scripts/train_and_evaluate.py</code></pre> | |
| <p> | |
| This deterministically retrains the model, verifies loss reduction, saves <code>checkpoints/qtensorformer_canonical_trained.pt</code>, and exports the Hugging Face package to <code>checkpoints/hf_canonical_trained/</code>. | |
| </p> | |
| </section> | |
| </main> | |
| <footer> | |
| <div class="footer-links"> | |
| <a href="index.html">Home</a> | |
| <a href="architecture.html">Architecture</a> | |
| <a href="methodology.html">Methodology</a> | |
| <a href="benchmarks.html">Benchmarks</a> | |
| <a href="documentation.html">Docs</a> | |
| <a href="limitations.html">Limitations</a> | |
| <a href="sitemap.xml">Sitemap</a> | |
| <a href="llms.txt">llms.txt</a> | |
| <a href="https://huggingface.co/Premchan369/Q-TensorFormer">Hugging Face</a> | |
| </div> | |
| <p>© 2026 Premchand Yadav. Licensed under Apache-2.0.</p> | |
| </footer> | |
| </body> | |
| </html> | |