| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <title>FinDendro · Do LLMs Know What They Know? · The Fin AI</title> |
| <meta name="description" content="FinDendro dates what LLMs know about financial history, and checks it against what they say they know." /> |
| <link rel="preconnect" href="https://fonts.googleapis.com" /> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> |
| <link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,500;0,9..144,600;1,9..144,500&family=Newsreader:ital,opsz,wght@0,6..72,400;0,6..72,500;1,6..72,400;1,6..72,500&family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" /> |
| <link rel="icon" href="./assets/finai-logo.png" /> |
| <link rel="stylesheet" href="./style.css" /> |
| </head> |
| <body> |
| <div class="progress" id="progress"></div> |
| <div class="glow" aria-hidden="true"></div> |
| <div class="edge edge-l" aria-hidden="true"><span class="e-name">FinDendro</span></div> |
| <div class="edge edge-r" aria-hidden="true"><span class="e-name">The Fin AI</span></div> |
| <div class="tip" id="tip" role="tooltip"></div> |
|
|
| <header class="header" id="header"> |
| <div class="wrap nav"> |
| <a href="https://thefin.ai/" class="brand" target="_blank" rel="noreferrer"><img src="./assets/finai-logo.png" alt="" />The Fin AI<span class="brand-sep">/</span><span class="brand-fd">FinDendro</span></a> |
| <div class="nav-right"> |
| <nav class="links" id="navLinks"> |
| <a href="#replay" data-sec="replay">Replay</a> |
| <a href="#leaderboard" data-sec="leaderboard">Leaderboard</a> |
| <a href="#core" data-sec="core">Core sample</a> |
| <a href="#date" data-sec="date">Declared date</a> |
| <a href="#benchmark" data-sec="benchmark">Benchmark</a> |
| </nav> |
| <a class="btn btn--sm btn--ghost nav-data" href="https://huggingface.co/datasets/TheFinAI/FinDendro" target="_blank" rel="noreferrer">Data <span class="arrow">↗</span></a> |
| <a class="btn btn--sm" href="https://arxiv.org/abs/2611.06808" target="_blank" rel="noreferrer">Paper <span class="arrow">↗</span></a> |
| </div> |
| </div> |
| </header> |
|
|
| <main id="top"> |
| <section class="hero" id="hero"> |
| <div class="wrap hero-grid"> |
| <div class="hero-copy"> |
| <a class="award reveal" href="#replay"><span class="medal" id="bird" aria-hidden="true"></span>Temporal self-knowledge · <b>1999 to 2026, quarter by quarter</b> →</a> |
| <h1 class="wordmark reveal">FinDendro</h1> |
| <p class="hero-sub reveal">A tree records every year it lived.<br />An LLM <em>can't tell you</em> what it remembers.</p> |
| <p class="lead reveal">Dendrochronologists date wood by reading its rings. We date models the same way. Every model, <span class="rotator" id="rotator"></span>, answers dated questions about markets and says whether it knows each answer. Then we check both.</p> |
| <div class="stats reveal" id="stats"></div> |
| <div class="ctas reveal"> |
| <a class="btn btn--lg" href="#replay">Try to beat a model</a> |
| <a class="btn btn--lg btn--ghost" href="#leaderboard">See who knows</a> |
| </div> |
| <p class="paper-links reveal"><a href="https://arxiv.org/abs/2611.06808" target="_blank" rel="noreferrer">arXiv 2611.06808 ↗</a><a href="https://huggingface.co/papers/2611.06808" target="_blank" rel="noreferrer">Hugging Face Papers ↗</a></p> |
| </div> |
|
|
| <figure class="hero-viz reveal"> |
| <figcaption><span>Cross-section · one ring per period</span><a href="#core">Read the core →</a></figcaption> |
| <div class="xs-chips" id="xsChips" role="tablist" aria-label="Choose a model"></div> |
| <div class="xs-wrap"> |
| <div class="xs-side xs-side--l"><b>Says</b><span>claims it knows</span></div> |
| <div class="xs" id="xs"></div> |
| <div class="xs-side xs-side--r"><b>Knows</b><span>answers correctly</span></div> |
| </div> |
| <div class="xs-ends" id="xsEnds"></div> |
| <p class="hero-claim">“I don't know” is <em>not</em> evidence that a model doesn't know.</p> |
| </figure> |
| </div> |
| <div class="ticker" id="ticker" aria-label="Leaderboard ticker"><div class="ticker-track" id="tickerTrack"></div></div> |
| </section> |
|
|
| <section class="section" id="replay"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Replay</span> |
| <h2>It said it <em>didn't know</em>. Then it named the price.</h2> |
| <p class="note">Real questions from the benchmark and the models' recorded answers. Make your pick first, then see what each model chose and whether it claimed to know.</p> |
| </div> |
| <div class="game reveal"> |
| <article class="q-card"> |
| <div class="q-top"><span class="q-kick" id="qKick"></span><span class="q-n" id="qN"></span></div> |
| <p class="q-text" id="qText"></p> |
| <div class="q-opts" id="qOpts"></div> |
| <div class="q-foot"> |
| <button class="btn btn--ghost btn--sm" id="qReveal" type="button">Reveal without guessing</button> |
| <button class="btn btn--sm" id="qNext" type="button" hidden>Next fact →</button> |
| </div> |
| </article> |
| <aside class="a-card"> |
| <div class="a-head"><span>What the models said</span><span class="a-score" id="aScore"></span></div> |
| <div class="a-rows" id="aRows"></div> |
| <p class="a-punch" id="aPunch" hidden></p> |
| </aside> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section section--tint" id="leaderboard"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Leaderboard</span> |
| <h2>Who <em>knows</em>, and who <em>admits</em> it</h2> |
| <p class="note">Open a row for the breakdown. Click a heading to rank by it.</p> |
| </div> |
| <div class="board reveal"><div class="scroll"><table class="lb" id="lb"></table></div></div> |
| <div class="keys reveal"> |
| <p><b>Knows</b> accuracy on 1,340 controlled facts, chance about 0.27. <b>Says</b> share of answers claiming knowledge. <b>Right when denying</b> accuracy on facts the model says it does not know. <b>Gap</b> how much better it does when it claims to know, inside each period.</p> |
| <p class="ramp-key">Darker green means higher. <span class="ramp"></span></p> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section" id="core"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Core sample</span> |
| <h2>The last ring and the last <em>claim</em> can be years apart</h2> |
| <p class="note">One rule places every boundary. Knowledge ends where accuracy falls halfway to chance. Claims end where the claim rate halves.</p> |
| </div> |
| <div class="cal card reveal"> |
| <div class="legend"> |
| <span><i class="mk mk--k"></i>Knowledge ends</span><span><i class="mk mk--c"></i>Claims end</span><span><i class="mk mk--s"></i>Says when asked</span><span><i class="mk mk--v"></i>Vendor cutoff</span><span><i class="band"></i>Claiming without knowing</span> |
| </div> |
| <div id="calendar"></div> |
| </div> |
| <div class="big card reveal"> |
| <div class="big-top"> |
| <div class="pick" id="coreChips" role="tablist" aria-label="Choose a model"></div> |
| <div class="legend"><span><i class="ln ln--k"></i>Accuracy</span><span><i class="ln ln--c"></i>Claim rate</span><span><i class="tick"></i>Chance</span></div> |
| </div> |
| <div class="chart" id="coreChart"></div> |
| <p class="note" id="coreNote"></p> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section section--tint" id="grain"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Grain</span> |
| <h2>Models admit the <em>Fed</em> and hide the <em>prices</em></h2> |
| <p class="note">Frontier models claim knowledge of most Fed decisions and almost no stock prices, yet they recall prices well above what they admit.</p> |
| </div> |
| <div class="fam reveal" id="fam"></div> |
| </div> |
| </section> |
|
|
| <section class="section" id="date"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Declared date</span> |
| <h2>A date changes the <em>words</em>, not the <em>knowledge</em></h2> |
| <p class="note">Backtests fix the forecast origin with one line. Add it and watch what moves.</p> |
| </div> |
| <div class="prompt-line reveal"> |
| <button class="sw" id="dateSw" type="button" role="switch" aria-checked="false"><i></i></button> |
| <code id="dateCode"><span id="datePrefix">Assume that today is May 31, 2023. Answer as you would on that date.</span></code> |
| </div> |
| <div class="decl reveal"> |
| <div class="card"><h3>Claims to already know</h3><div class="hb" id="declC"></div></div> |
| <div class="card"><h3>Accuracy on the value</h3><div class="hb" id="declA"></div></div> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section section--tint" id="pit"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Saplings</span> |
| <h2>Clean by construction, <em>empty</em> by construction</h2> |
| <p class="note">Point-in-time models stop reading at a cutoff, so they cannot leak. At 1 to 1.5B parameters they also retain almost none of these facts. Scored by candidate likelihood, accuracy sits near chance on both sides of the cutoff.</p> |
| </div> |
| <div class="board reveal"><div class="scroll"><table class="lb pit" id="pitT"></table></div></div> |
| </div> |
| </section> |
|
|
| <section class="section" id="benchmark"> |
| <div class="wrap"> |
| <div class="sec-head reveal"> |
| <span class="kicker">Benchmark</span> |
| <h2><em>One</em> fact, <em>one</em> choice, two answers</h2> |
| <p class="note">Every option pairs a value with a statement of knowledge, so a single letter reveals both what the model thinks and whether it claims to know.</p> |
| </div> |
| <div class="how reveal"> |
| <div class="eg card"> |
| <p class="eg-q">What was the 12-month percent change in the US CPI for June 2023?</p> |
| <div class="eg-grid"> |
| <span class="eg-h">“I know this value”</span><span class="eg-h">“I do not know, best guess”</span> |
| <span>A. 2.8%</span><span>E. 2.8%</span><span class="eg-hit">B. 3.0%</span><span class="eg-hit">F. 3.0%</span><span>C. 3.2%</span><span>G. 3.2%</span><span>D. 3.4%</span><span>H. 3.4%</span> |
| </div> |
| <p class="note">The published value is shaded. The model picks one letter out of eight.</p> |
| </div> |
| <div class="m-grid"> |
| <div><h3>Facts</h3><p>2,226 dated U.S. facts drawn from a pool of 20,972. Twelve FRED series, month-end prices of 50 companies and Fed decisions, scored against the value published at the time.</p></div> |
| <div><h3>Questions</h3><p>Five prompts per fact. One asks for the value alone. Four offer the paired options, as a recall or a prediction, with and without a declared date.</p></div> |
| <div><h3>Controls</h3><p>47% of numerical facts fall to repeating last month's value, so they do not count toward knowledge. Fed decisions use balanced accuracy.</p></div> |
| <div><h3>Boundaries</h3><p>One rule for every model, from two-quarter rolling accuracy and the claim rate, each against its 2015 to 2020 level.</p></div> |
| </div> |
| </div> |
| </div> |
| </section> |
| </main> |
|
|
| <footer class="foot"><div class="wrap foot-in"><span>FinDendro · The Fin AI</span><span class="foot-links"><a href="https://arxiv.org/abs/2611.06808" target="_blank" rel="noreferrer">Paper · arXiv 2611.06808 ↗</a><a href="https://huggingface.co/papers/2611.06808" target="_blank" rel="noreferrer">Hugging Face Papers ↗</a><a href="https://huggingface.co/datasets/TheFinAI/FinDendro" target="_blank" rel="noreferrer">Questions and raw results ↗</a></span></div></footer> |
| <script src="./data.js"></script> |
| <script src="./app.js"></script> |
| </body> |
| </html> |
|
|
