Blame for site/benchmarks.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Empirical Benchmarks &amp; Evaluation - Q-TensorFormer</title>
  <meta name="description" content="Reproducible empirical benchmarks for Q-TensorFormer: Tensor-Train parameter compression, CPU latency, TTFT, cold-start loading, and multi-precision KV caching.">
  <link rel="canonical" href="https://premchan369.github.io/Q-TensorFormer/benchmarks.html">
  
  <meta property="og:type" content="article">
  <meta property="og:site_name" content="Q-TensorFormer Research Project">
  <meta property="og:title" content="Empirical Benchmarks &amp; Evaluation - Q-TensorFormer">
  <meta property="og:description" content="Detailed reproducible hardware measurements across latency, memory, parameters, and perplexity.">
  <meta property="og:url" content="https://premchan369.github.io/Q-TensorFormer/benchmarks.html">

  <script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "TechArticle",
    "headline": "Q-TensorFormer Empirical Benchmarks and Performance Evaluation",
    "description": "Verifiable latency, memory bandwidth, parameter compression, and perplexity measurements.",
    "url": "https://premchan369.github.io/Q-TensorFormer/benchmarks.html",
    "author": { "@type": "Person", "name": "Premchand Yadav" },
    "datePublished": "2026-10-13"
  }
  </script>
  <link rel="stylesheet" href="assets/style.css">
</head>
<body>
  <header>
    <div class="header-container">
      <div class="logo">
        <a href="index.html"><span>Q-TensorFormer</span> Research</a>
      </div>
      <nav>
        <a href="index.html">Overview</a>
        <a href="architecture.html">Architecture</a>
        <a href="methodology.html">Methodology</a>
        <a href="benchmarks.html" class="active">Benchmarks</a>
        <a href="edge-inference.html">Edge Inference</a>
        <a href="training.html">Training</a>
        <a href="documentation.html">Docs & API</a>
        <a href="research.html">Research</a>
        <a href="limitations.html">Limitations</a>
        <a href="contributing.html">Contributing</a>
      </nav>
    </div>
  </header>

  <main class="container">
    <h1>Empirical Benchmarks &amp; Evaluation</h1>
    <p class="lead">
      All figures reported below are generated from actual execution runs on host hardware (Intel 20-thread CPU, PyTorch 2.10.0+cpu, Windows 10) under test identifier <code>RUN-20261013-QTF-HARDENING</code>.
    </p>

    <div class="callout">
      <p><strong>Scientific Integrity Classification:</strong> Every reported metric is tagged as <code>MEASURED</code> (recorded from real hardware timers and memory allocation), <code>SIMULATED</code> (statevector emulation), or <code>ESTIMATED</code> (analytical cost models). Zero metrics are fabricated or projected.</p>
    </div>

    <section>
      <h2>1. Language Model Training &amp; Quality Validation</h2>
      <p>
        Evaluated on <code>data/qtensorformer_corpus.txt</code> across 70% train, 15% validation, and 15% test splits (source: <code>results/raw/training_and_quality_validation.json</code>):
      </p>
      <table>
        <thead>
          <tr>
            <th>Architecture</th>
            <th>Parameters</th>
            <th>Initial Test Loss</th>
            <th>Initial Perplexity</th>
            <th>Final Test Loss</th>
            <th>Final Perplexity</th>
            <th>Perplexity Drop</th>
            <th>Label</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td><strong>QTensorFormer (Canonical)</strong></td>
            <td>68,604</td>
            <td>5.8646</td>
            <td>352.35</td>
            <td><strong>3.8086</strong></td>
            <td><strong>45.09</strong></td>
            <td><strong>-87.2%</strong></td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><strong>DenseBaseline (Matched)</strong></td>
            <td>116,480</td>
            <td>5.7282</td>
            <td>307.40</td>
            <td><strong>3.8242</strong></td>
            <td><strong>45.80</strong></td>
            <td><strong>-85.1%</strong></td>
            <td>MEASURED</td>
          </tr>
        </tbody>
      </table>
    </section>

    <section>
      <h2>2. Layer-Level Tensor-Train Rank Sweep (512 &rarr; 2048 Projection)</h2>
      <p>
        40 timed runs per rank level; zero-SVD rank slicing overhead measured at 98.29 &mu;s (source: <code>results/raw/tt_compression_audit.json</code>):
      </p>
      <table>
        <thead>
          <tr>
            <th>Layer / Rank</th>
            <th>Active Params</th>
            <th>Weight Bytes</th>
            <th>Compression Ratio</th>
            <th>Param Reduction</th>
            <th>Frobenius Error</th>
            <th>Prefill (B=4, T=64)</th>
            <th>Label</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td><code>nn.Linear</code> (Dense)</td>
            <td>1,050,624</td>
            <td>4,202,496 B</td>
            <td>1.00x</td>
            <td>0.00%</td>
            <td>0.0</td>
            <td>3.59 ms</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>TTLinear</code> (r=16)</td>
            <td>266,304</td>
            <td>1,065,216 B</td>
            <td>3.95x</td>
            <td>74.65%</td>
            <td>1.3e-6</td>
            <td>13.49 ms</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>TTLinear</code> (r=8)</td>
            <td>133,664</td>
            <td>534,656 B</td>
            <td>7.86x</td>
            <td>87.28%</td>
            <td>5.0e-7</td>
            <td>7.71 ms</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>TTLinear</code> (r=4)</td>
            <td>67,728</td>
            <td>270,912 B</td>
            <td>15.51x</td>
            <td>93.55%</td>
            <td>4.0e-7</td>
            <td>5.35 ms</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>TTLinear</code> (r=2)</td>
            <td>34,856</td>
            <td>139,424 B</td>
            <td>30.14x</td>
            <td>96.68%</td>
            <td>1.0e-7</td>
            <td>4.17 ms</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>TTLinear</code> (r=1)</td>
            <td>18,444</td>
            <td>73,776 B</td>
            <td><strong>56.96x</strong></td>
            <td><strong>98.24%</strong></td>
            <td>0.0</td>
            <td>5.93 ms</td>
            <td>MEASURED</td>
          </tr>
        </tbody>
      </table>
    </section>

    <section>
      <h2>3. Adaptive KV-Cache Scaling at Sequence Length L=1024</h2>
      <p>
        4 sink tokens protected, maximum eviction capacity 128 (source: <code>results/raw/kv_cache_audit.json</code>):
      </p>
      <table>
        <thead>
          <tr>
            <th>Cache Configuration</th>
            <th>Physical Cache Bytes</th>
            <th>Memory Reduction vs FP16</th>
            <th>Output Cosine Similarity</th>
            <th>Evaluation Finding</th>
            <th>Label</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td><code>FP16 Full Cache</code></td>
            <td>524,288 B</td>
            <td>0.00%</td>
            <td>1.000000</td>
            <td>Baseline uncompressed reference</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>INT8 Full Cache</code></td>
            <td>278,528 B</td>
            <td>-46.88% (1.88x)</td>
            <td>0.999973</td>
            <td>Near-lossless quantization</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>INT4 Packed Full Cache</code></td>
            <td>163,840 B</td>
            <td>-68.75% (3.20x)</td>
            <td>0.995593</td>
            <td>High compression with minimal drift</td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>Sink-Protected INT8 (cap=128)</code></td>
            <td><strong>34,816 B</strong></td>
            <td><strong>-93.36% (15.06x)</strong></td>
            <td><strong>0.996337</strong></td>
            <td><strong>Bounded memory ceiling, stable attention</strong></td>
            <td>MEASURED</td>
          </tr>
          <tr>
            <td><code>Sliding Window without Sinks</code></td>
            <td>262,144 B</td>
            <td>-50.00%</td>
            <td>0.004714</td>
            <td>Severe attention collapse</td>
            <td>MEASURED</td>
          </tr>
        </tbody>
      </table>
    </section>

    <section>
      <h2>4. Reproduction Commands</h2>
      <p>To reproduce all benchmark figures directly on your hardware:</p>
      <pre><code># Run full scientific benchmark suite
python scripts/run_scientific_benchmark.py

# Run training quality validation
python scripts/train_and_evaluate.py

# Inspect results
cat results/processed/benchmark_summary.csv</code></pre>
    </section>
  </main>

  <footer>
    <div class="footer-links">
      <a href="index.html">Home</a>
      <a href="architecture.html">Architecture</a>
      <a href="methodology.html">Methodology</a>
      <a href="benchmarks.html">Benchmarks</a>
      <a href="documentation.html">Docs</a>
      <a href="limitations.html">Limitations</a>
      <a href="sitemap.xml">Sitemap</a>
      <a href="llms.txt">llms.txt</a>
      <a href="https://huggingface.co/Premchan369/Q-TensorFormer">Hugging Face</a>
    </div>
    <p>&copy; 2026 Premchand Yadav. Licensed under Apache-2.0.</p>
  </footer>
</body>
</html>