Empirical Benchmarks & Evaluation
All figures reported below are generated from actual execution runs on host hardware (Intel 20-thread CPU, PyTorch 2.10.0+cpu, Windows 10) under test identifier RUN-20261013-QTF-HARDENING.
Scientific Integrity Classification: Every reported metric is tagged as MEASURED (recorded from real hardware timers and memory allocation), SIMULATED (statevector emulation), or ESTIMATED (analytical cost models). Zero metrics are fabricated or projected.
1. Language Model Training & Quality Validation
Evaluated on data/qtensorformer_corpus.txt across 70% train, 15% validation, and 15% test splits (source: results/raw/training_and_quality_validation.json):
| Architecture | Parameters | Initial Test Loss | Initial Perplexity | Final Test Loss | Final Perplexity | Perplexity Drop | Label |
|---|---|---|---|---|---|---|---|
| QTensorFormer (Canonical) | 68,604 | 5.8646 | 352.35 | 3.8086 | 45.09 | -87.2% | MEASURED |
| DenseBaseline (Matched) | 116,480 | 5.7282 | 307.40 | 3.8242 | 45.80 | -85.1% | MEASURED |
2. Layer-Level Tensor-Train Rank Sweep (512 → 2048 Projection)
40 timed runs per rank level; zero-SVD rank slicing overhead measured at 98.29 μs (source: results/raw/tt_compression_audit.json):
| Layer / Rank | Active Params | Weight Bytes | Compression Ratio | Param Reduction | Frobenius Error | Prefill (B=4, T=64) | Label |
|---|---|---|---|---|---|---|---|
nn.Linear (Dense) |
1,050,624 | 4,202,496 B | 1.00x | 0.00% | 0.0 | 3.59 ms | MEASURED |
TTLinear (r=16) |
266,304 | 1,065,216 B | 3.95x | 74.65% | 1.3e-6 | 13.49 ms | MEASURED |
TTLinear (r=8) |
133,664 | 534,656 B | 7.86x | 87.28% | 5.0e-7 | 7.71 ms | MEASURED |
TTLinear (r=4) |
67,728 | 270,912 B | 15.51x | 93.55% | 4.0e-7 | 5.35 ms | MEASURED |
TTLinear (r=2) |
34,856 | 139,424 B | 30.14x | 96.68% | 1.0e-7 | 4.17 ms | MEASURED |
TTLinear (r=1) |
18,444 | 73,776 B | 56.96x | 98.24% | 0.0 | 5.93 ms | MEASURED |
3. Adaptive KV-Cache Scaling at Sequence Length L=1024
4 sink tokens protected, maximum eviction capacity 128 (source: results/raw/kv_cache_audit.json):
| Cache Configuration | Physical Cache Bytes | Memory Reduction vs FP16 | Output Cosine Similarity | Evaluation Finding | Label |
|---|---|---|---|---|---|
FP16 Full Cache |
524,288 B | 0.00% | 1.000000 | Baseline uncompressed reference | MEASURED |
INT8 Full Cache |
278,528 B | -46.88% (1.88x) | 0.999973 | Near-lossless quantization | MEASURED |
INT4 Packed Full Cache |
163,840 B | -68.75% (3.20x) | 0.995593 | High compression with minimal drift | MEASURED |
Sink-Protected INT8 (cap=128) |
34,816 B | -93.36% (15.06x) | 0.996337 | Bounded memory ceiling, stable attention | MEASURED |
Sliding Window without Sinks |
262,144 B | -50.00% | 0.004714 | Severe attention collapse | MEASURED |
4. Reproduction Commands
To reproduce all benchmark figures directly on your hardware:
# Run full scientific benchmark suite
python scripts/run_scientific_benchmark.py
# Run training quality validation
python scripts/train_and_evaluate.py
# Inspect results
cat results/processed/benchmark_summary.csv