Spaces:
Running
Running
Download index.html from Battam/training-data-foundation: direct link, hf CLI and curl.
- Browser
- Download file 23.3 kB
-
https://huggingface.co/spaces/Battam/training-data-foundation/resolve/main/index.html
- Command line
-
hf download hf://spaces/Battam/training-data-foundation/index.html
-
curl -L -o index.html https://huggingface.co/spaces/Battam/training-data-foundation/resolve/main/index.html
23.3 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title data-i18n="page.title">Training Data Foundation</title> | |
| <link rel="icon" href="data:,"> | |
| <link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents"> | |
| <meta name="description" content="Real public data for pre-training, mid-training and post-training from Battam/training-data-foundation, the public index of the Training Data Foundation: pick by training stage, domain and form, and take it with one command." data-i18n-attr="content:page.description"> | |
| <!-- #2422: both themes, by the reader's own setting (style.css prefers-color-scheme) --> | |
| <meta name="color-scheme" content="light dark"> | |
| <meta name="theme-color" content="#08131b" media="(prefers-color-scheme: dark)"> | |
| <meta name="theme-color" content="#e9eff3" media="(prefers-color-scheme: light)"> | |
| <link rel="preconnect" href="https://huggingface.co"> | |
| <link rel="preconnect" href="https://cdn.jsdelivr.net" crossorigin> | |
| <link rel="preload" href="locales/en.json" as="fetch" crossorigin> | |
| <link rel="preload" href="locales/zh.json" as="fetch" crossorigin> | |
| <link rel="preload" href="fonts/plex-sans-400.woff2" as="font" type="font/woff2" crossorigin> | |
| <link rel="stylesheet" href="style.css"> | |
| <script type="module" src="first.js" async></script> | |
| <script type="module" src="app.js"></script> | |
| </head> | |
| <body> | |
| <nav class="topnav" aria-label="Page sections" data-i18n-attr="aria-label:nav.sections"> | |
| <div class="nav-row"> | |
| <!-- the mark: three strata, in the colors of the three training stages --> | |
| <a class="brand" href="#top"><svg class="mark" width="26" height="26" viewBox="0 0 26 26" aria-hidden="true"><path d="M13 3 23 8 13 13 3 8z" fill="var(--post)"/><path d="M3 12.5 13 17.5 23 12.5" fill="none" stroke="var(--mid)" stroke-width="2.2" stroke-linejoin="round"/><path d="M3 17.5 13 22.5 23 17.5" fill="none" stroke="var(--pre)" stroke-width="2.2" stroke-linejoin="round"/></svg><span data-i18n="page.title">Training Data Foundation</span></a> | |
| <!-- #2422: the page's three parts: the foundation (the working screen), the sources and Get the data --> | |
| <ol class="progress"> | |
| <li><a href="#top" data-part="stages"><span class="pl pl-long" data-i18n="nav.stages">The foundation</span><span class="pl pl-short" data-i18n="nav.stagesShort">Foundation</span><span class="track"><span class="fill"></span></span></a></li> | |
| <li><a href="#browse" data-part="pick"><span class="pl pl-long" data-i18n="nav.pick">Sources</span><span class="pl pl-short" data-i18n="nav.pickShort">Sources</span><span class="track"><span class="fill"></span></span></a></li> | |
| <li><a href="#get" data-part="get"><span class="pl pl-long" data-i18n="nav.get">Get the data</span><span class="pl pl-short" data-i18n="nav.getShort">Get</span><span class="track"><span class="fill"></span></span></a></li> | |
| </ol> | |
| <div class="nav-links"> | |
| <a href="https://huggingface.co/datasets/Battam/training-data-foundation" data-i18n="link.dataset">Dataset</a> | |
| <a href="#manual" data-i18n="nav.manual">Manual</a> | |
| <a href="llms.txt" data-i18n="nav.agents">For agents</a> | |
| <a class="btn solid small-btn" href="https://huggingface.co/datasets/Battam/training-data-foundation/discussions/new" data-i18n="nav.request">Request data</a> | |
| <!-- the language switch: the other language's name, in that language (i18n.js) --> | |
| <button class="btn small-btn lang-btn" id="lang-switch" type="button" hidden></button> | |
| </div> | |
| </div> | |
| </nav> | |
| <main id="page"> | |
| <!-- #2422, second round ("Strata"): the first screen is a working screen. Its ground is the field (field.js, drawn | |
| with WebGL): every source of the index one point, colored by its stage group and sized by its deduplicated size, | |
| in three arrangements. Glass panels float on it from 1100 px: left, the name, the sentence, the stages of the | |
| stage table as a column (first.js drawStages, app.js renderStageFig: a click on one sets the filter by stage) | |
| and the filters; right, the count of the selection and its sources (app.js renderWork); at the bottom, the one | |
| command that takes the selection (renderDock). Below 1100 px the field is on top and the panels follow in the | |
| flow. Without WebGL the panels keep their places and one line says what the ground would show --> | |
| <header class="work" id="top" data-part="stages"> | |
| <div class="field" id="field"> | |
| <!-- the outline of four strata while the scene is loading, so the ground is never bare --> | |
| <svg class="field-ghost" viewBox="0 0 700 470" aria-hidden="true"><path d="M20 95 190 10 680 65 590 180z"/><path d="M20 190 190 105 680 160 590 275z"/><path d="M20 285 190 200 680 255 590 370z"/><path d="M20 380 190 295 680 350 590 465z"/></svg> | |
| <canvas role="img" tabindex="0" aria-label="Every source of the index as one point, layered by training stage" data-i18n-attr="aria-label:field.label"></canvas> | |
| <!-- the ring of the point under the pointer and the two rings of the open source (field.js places them) --> | |
| <svg class="field-ov" aria-hidden="true"><g class="field-ring" visibility="hidden"><circle/></g><g class="field-open" visibility="hidden"><circle class="a"/><circle class="b"/></g></svg> | |
| <!-- the labels are a second way to the stage bands and the domain filter, for the pointer only --> | |
| <div class="field-labs" aria-hidden="true"></div> | |
| <div class="field-call" hidden></div> | |
| <div class="field-tip" hidden></div> | |
| </div> | |
| <p class="field-empty" id="field-empty" hidden><span data-i18n="sources.none">No source matches</span><button class="link" id="field-clear" type="button" data-i18n="filter.clear">Clear</button></p> | |
| <!-- what a point's light says --> | |
| <p class="field-legend" id="field-legend" hidden><span><i class="l1"></i><span data-i18n="field.legendOn">in the selection</span></span><span><i class="l2"></i><span data-i18n="field.legendHalf">kept, in the other layer</span></span><span><i class="l3"></i><span data-i18n="field.legendOff">filtered out</span></span></p> | |
| <p class="field-off" id="field-off" hidden data-i18n="field.off">The ground of this screen is a field of points, one per source, drawn with WebGL, which this browser does not offer. Everything else on the page works without it.</p> | |
| <div class="panel views" id="views" hidden> | |
| <span class="seg" role="group" aria-label="Arrangement" data-i18n-attr="aria-label:field.views" id="view-seg"> | |
| <button type="button" data-view="strata" aria-pressed="true" data-i18n="field.strata">Strata</button> | |
| <button type="button" data-view="domains" aria-pressed="false" data-i18n="field.domains">Domains</button> | |
| <button type="button" data-view="scale" aria-pressed="false" data-i18n="field.scale">Scale</button> | |
| </span> | |
| <span class="seg zoom" role="group" aria-label="Zoom" data-i18n-attr="aria-label:field.zoom"> | |
| <button type="button" id="zoom-out" aria-label="Zoom out" data-i18n-attr="aria-label:field.zoomOut">−</button> | |
| <button type="button" id="zoom-in" aria-label="Zoom in" data-i18n-attr="aria-label:field.zoomIn">+</button> | |
| <button type="button" id="zoom-reset" aria-label="Back to the first view" title="Back to the first view" data-i18n-attr="aria-label:field.reset;title:field.reset"><svg width="13" height="13" viewBox="0 0 16 16" aria-hidden="true"><path d="M3 8a5 5 0 1 0 1.6-3.7M3 2.5v3h3" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round"/></svg></button> | |
| </span> | |
| </div> | |
| <p class="field-hint" id="field-hint" hidden data-i18n="field.hint">Drag to turn 路 Ctrl + scroll to zoom 路 point at a source</p> | |
| <aside class="panel work-left"> | |
| <div class="pad"> | |
| <p class="eyebrow" data-i18n="page.eyebrow">The public index</p> | |
| <h1 data-i18n="page.title">Training Data Foundation</h1> | |
| <p class="lede" data-i18n="hero.lede">Real public data for pre-training, mid-training and post-training: each file with its address, size, hashes and license, fetched with one command.</p> | |
| <!-- the release the page reads, in one line: its tag, its day and a link to every release (first.js drawRelease) --> | |
| <p class="release-line" id="release-line"><span class="sk sk-line"></span></p> | |
| </div> | |
| <!-- the stratigraphic column: one band per stage of the stage table, top to bottom as the field stacks them, each | |
| with its funnel, and under them what no stage counts, by reason --> | |
| <figure class="stage-fig" aria-labelledby="stage-fig-h"> | |
| <figcaption class="eyebrow" id="stage-fig-h" data-i18n="stage.figTitle">What each training stage can use</figcaption> | |
| <ul class="column" id="stage-tiles"><li><div class="stratum sk-tile"></div></li><li><div class="stratum sk-tile"></div></li><li><div class="stratum sk-tile"></div></li></ul> | |
| <!-- a missing input, one short line each (app.js renderStageFig) --> | |
| <p class="stage-why" id="stage-why" hidden></p> | |
| </figure> | |
| <!-- the words the page counts in and sorts by (material set, corpus, unit, domain): each a button that opens its | |
| meaning in place, as the index's vocabulary gives it (app.js renderTerms); the manual's glossary has the rest; | |
| #2614: the popover they open follows the manual's view, outside the page, as the manual's start has the line too --> | |
| <p class="terms" id="terms" hidden></p> | |
| <!-- #2422: the filters: the search, then the domain, the form, the size, the training permit and the personal-data | |
| mark (two labels every source carries, which the page filters on and counts and never leaves anything out of a | |
| count for) and the file types. The stage is set by its band above, so its select stays out of sight. A filter | |
| the panel does not show (collection, subdomain, license tier, redistribution, status) still holds from the | |
| address, as a chip that removes it --> | |
| <div class="filters" id="filters" role="group" aria-label="Filters" data-i18n-attr="aria-label:filter.title"> | |
| <button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row"> | |
| <span data-i18n="filter.title">Filters</span><span class="state" id="filter-state" data-i18n="filter.noneOn">none on</span> | |
| </button> | |
| <div class="filter-row" id="filter-row"> | |
| <p class="eyebrow filter-title" data-i18n="filter.title">Filters</p> | |
| <label class="wide"><span data-i18n="filter.search">Search</span><span class="q-wrap"><input id="f-q" type="search" placeholder="Title or text" data-i18n-attr="placeholder:filter.searchHint" autocomplete="off"><kbd aria-hidden="true">/</kbd></span></label> | |
| <!-- #2006: the training stages of the stage table (catalog/stages.yaml), computed when the page is built | |
| (stages.json); the bands above set it --> | |
| <label hidden><span data-i18n="filter.stage">Training stage</span><select id="f-stage"><option value="" data-i18n="filter.all">All</option></select><span id="f-stage-note" hidden></span></label> | |
| <label><span data-i18n="filter.domain">Domain</span><select id="f-domain"><option value="" data-i18n="filter.all">All</option></select></label> | |
| <label><span data-i18n="filter.form">Form</span><select id="f-form"><option value="" data-i18n="filter.all">All</option></select></label> | |
| <label><span data-i18n="filter.size">Size</span><select id="f-size"> | |
| <option value="" data-i18n="filter.any">Any</option> | |
| <option value="lt1g" data-i18n="filter.size.lt1g">Under 1 GB</option> | |
| <option value="1g1t" data-i18n="filter.size.1g1t">1 GB to 1 TB</option> | |
| <option value="ge1t" data-i18n="filter.size.ge1t">1 TB or more</option> | |
| <option value="unknown" data-i18n="filter.size.unknown">Not published</option> | |
| </select></label> | |
| <label><span data-i18n="filter.training">Training</span><select id="f-train"> | |
| <option value="" data-i18n="filter.any">Any</option> | |
| <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option> | |
| <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option> | |
| </select></label> | |
| <!-- #2422: the personal-data mark of a source (its catalog pii.level) --> | |
| <label><span data-i18n="filter.pii">Personal data</span><select id="f-pii"> | |
| <option value="" data-i18n="filter.any">Any</option> | |
| <option value="none" data-i18n="filter.pii.none">None found</option> | |
| <option value="marked" data-i18n="filter.pii.marked">Present or suspected</option> | |
| <option value="unknown" data-i18n="filter.pii.unknown">Not judged yet</option> | |
| </select></label> | |
| <!-- #1524: the file types of the layer shown with their rows (app.js fillExt), any number checked; folded | |
| while none is checked, its summary saying how many are --> | |
| <details class="ext-fold" id="ext-fold"> | |
| <summary><span data-i18n="filter.ext">File type</span><span class="ext-on" id="ext-on"></span></summary> | |
| <div class="ext-list" id="f-ext" role="group" aria-label="File type" data-i18n-attr="aria-label:filter.ext"></div> | |
| </details> | |
| <div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled data-i18n="filter.clear">Clear</button></div> | |
| </div> | |
| </div> | |
| <p class="work-foot"> | |
| <span class="chip" id="preview-chip" hidden></span> | |
| <span class="notice" id="notice" hidden></span> | |
| </p> | |
| </aside> | |
| <aside class="panel work-right" aria-labelledby="sel-h"> | |
| <div class="pad"> | |
| <div class="sel-top"><p class="eyebrow" id="sel-h" data-i18n="chip.selection">Selection</p> | |
| <!-- the layer switch (#1083): the list, the charts and the command take one layer, material sets by default --> | |
| <span class="seg" role="group" aria-label="Show" data-i18n-attr="aria-label:layer.label" id="layer-seg"> | |
| <button type="button" data-kind="material_set" aria-pressed="true" data-i18n="kind.material_set">Material sets</button> | |
| <button type="button" data-kind="corpus" aria-pressed="false" data-i18n="kind.corpus">Corpora</button> | |
| </span> | |
| </div> | |
| <p class="count" aria-live="polite"><b class="num" id="sel-n"><span class="sk sk-num"></span></b><span id="sel-word"></span></p> | |
| <p class="meta" id="sel-meta"></p> | |
| <!-- the selection: one removable chip per filter that is on (app.js renderChips) --> | |
| <div class="sel-chips" id="sel-chips"></div> | |
| </div> | |
| <!-- the open source, in short (app.js renderCard); its whole detail is in the list below the screen --> | |
| <div id="sel-card" hidden></div> | |
| <ul class="src-list" id="sel-list"><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li></ul> | |
| <p class="panel-foot"><a href="#sources" id="sel-all"></a></p> | |
| </aside> | |
| <section class="panel work-dock" aria-labelledby="dock-h"> | |
| <div class="dock-top"> | |
| <p class="eyebrow" id="dock-h" data-i18n="get.take">Take</p> | |
| <span class="dock-word" id="dock-units" data-i18n="get.units">Units per source, at most</span> | |
| <span class="seg" role="group" aria-labelledby="dock-units" id="dock-sample"></span> | |
| <button class="btn solid small-btn" id="dock-copy" type="button" data-i18n="dock.copy">Copy the command</button> | |
| </div> | |
| <pre class="dock-cmd" id="dock-cmd" tabindex="0"><span class="sk sk-line"></span></pre> | |
| <p class="dock-foot"><a href="#get" data-i18n="dock.more">SQL, CSV and what it writes</a></p> | |
| </section> | |
| </header> | |
| <!-- #2422: the sources: the list of the selection, each source with its fit, its details and a preview of its rows, | |
| and the domain by form map with the bars by domain and by form, linked to the filters --> | |
| <div class="band" id="browse" data-part="pick"> | |
| <div class="wrap browse-main"> | |
| <section id="sources" aria-labelledby="sources-h"> | |
| <div class="band-head reveal"><h2 id="sources-h"><span class="sk sk-line"></span></h2></div> | |
| <div class="card table-card" id="sources-body"><div class="sk sk-row"></div><div class="sk sk-row"></div><div class="sk sk-row"></div><div class="sk sk-row"></div></div> | |
| </section> | |
| <section id="map" aria-labelledby="map-h"> | |
| <div class="band-head row reveal"> | |
| <h2 id="map-h"><span class="sk sk-line"></span></h2> | |
| <span class="seg-wrap" id="metric-wrap"><span class="seg-label" id="metric-label" data-i18n="metric.label">Count</span> | |
| <span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg"> | |
| <button type="button" data-metric="sets" aria-pressed="true" data-i18n="metric.sets">Material sets</button> | |
| <button type="button" data-metric="units" aria-pressed="false" data-i18n="metric.units">Units</button> | |
| </span> | |
| </span> | |
| </div> | |
| <div class="map-cards"> | |
| <figure class="card chart-card" id="heat-fig"><div class="sk sk-block tall"></div></figure> | |
| <div class="card chart-card side-card"> | |
| <figure class="chart" id="dom-fig"><div class="sk sk-block"></div></figure> | |
| <figure class="chart" id="form-fig"></figure> | |
| </div> | |
| </div> | |
| </section> | |
| </div> | |
| </div> | |
| <!-- #2422: Get the data: the selection's one command, what it writes and what the default profile of the fetch tool | |
| leaves out with the flag that takes each back, and the selection file (app.js renderGet) --> | |
| <section class="band band-alt" id="get" data-part="get" aria-labelledby="get-h"> | |
| <div class="wrap"> | |
| <div class="band-head reveal"><h2 id="get-h"><span class="sk sk-line"></span></h2></div> | |
| <div id="get-body"><div class="sk sk-block tall"></div></div> | |
| <!-- #1498, #1524: the usage example video from media/ of the dataset, another origin, hence crossorigin for its | |
| track; app.js renderVideo plays the video and the subtitles of the page's language. Its pictures carry no | |
| words, so the track is the default one. The dataset card and the project page link to this block by its id. | |
| #2066: beside it the second usage video, for people who build data from this page, added the same way | |
| (catalog/media.tsv: usage-people.<en or zh>.mp4 and .vtt) --> | |
| <div class="video-grid"> | |
| <div class="card video-card reveal" id="usage-video"> | |
| <h3 data-i18n="video.title">Usage example video</h3> | |
| <p class="small" data-i18n="video.text">The index in use from this page, in the page's language, with subtitles.</p> | |
| <video id="video-player" data-lang="en" controls preload="metadata" crossorigin="anonymous" src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-example.en.mp4"> | |
| <track kind="subtitles" srclang="en" default src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-example.en.vtt" label="English"> | |
| </video> | |
| </div> | |
| <div class="card video-card reveal" id="usage-people"> | |
| <h3 data-i18n="video.peopleTitle">Usage video for people who build data</h3> | |
| <p class="small" data-i18n="video.peopleText">The manual's worked example on this page, from one need to the files, in the page's language, with subtitles.</p> | |
| <video id="people-player" data-lang="en" controls preload="metadata" crossorigin="anonymous" src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-people.en.mp4"> | |
| <track kind="subtitles" srclang="en" default src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-people.en.vtt" label="English"> | |
| </video> | |
| </div> | |
| </div> | |
| <div class="card agents-card lift reveal" id="agents"> | |
| <span class="agents-ico" data-icon="robot" data-size="36"></span> | |
| <div class="agents-text"> | |
| <h3 id="agents-h" data-i18n="nav.agents">For agents</h3> | |
| <p class="small" data-i18n="agents.text">The index's columns 路 the one command 路 the default profile 路 what to keep out of the agent's environment</p> | |
| </div> | |
| <div class="agents-actions"> | |
| <a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a> | |
| <a class="btn" href="https://huggingface.co/datasets/Battam/training-data-foundation/blob/main/docs/for-agents.md" data-i18n-attr="href:url.forAgents" data-icon="doc" data-size="16"><span data-i18n="agents.readme">Index docs: For agents</span></a> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| </main> | |
| <!-- #1617: the user manual, a view of its own at the address #manual (app.js showView, manual.js), in place of the page | |
| above: manual/manual.<language>.md of this folder, its numbers and command outputs filled at deploy | |
| (03_data_line/steps/portal.py), its images manual/shots/<output>, taken on this page after each deploy | |
| (03_data_line/steps/shots.py); #2614: its contents beside it from 1024 px, a menu above it below that --> | |
| <main class="manual-view" id="manual" hidden aria-label="User manual" data-i18n-attr="aria-label:manual.title"> | |
| <div class="wrap manual-wrap"> | |
| <p class="manual-back"><a href="#top" data-i18n="manual.back">Back to the page</a></p> | |
| <div class="manual-grid"> | |
| <nav class="manual-toc" id="manual-toc" aria-label="Contents" data-i18n-attr="aria-label:manual.contents"></nav> | |
| <article class="card manual-body" id="manual-body"><div class="sk sk-block tall"></div></article> | |
| </div> | |
| </div> | |
| </main> | |
| <div class="term-pop" id="term-pop" popover></div> | |
| <footer class="site-foot" id="request" data-part="get"> | |
| <div class="wrap foot-grid"> | |
| <div class="foot-cta"> | |
| <span class="foot-ico" data-icon="request" data-size="32"></span> | |
| <div> | |
| <h2 data-i18n="foot.missing">Missing data?</h2> | |
| <p data-i18n="foot.ask">Ask for a domain, a form or a source.</p> | |
| </div> | |
| <a class="btn on-dark-solid" href="https://huggingface.co/datasets/Battam/training-data-foundation/discussions/new" data-i18n="nav.request">Request data</a> | |
| </div> | |
| <ul class="foot-links"> | |
| <li><a href="https://huggingface.co/datasets/Battam/training-data-foundation" data-i18n="link.dataset">Dataset</a></li> | |
| <li><a href="https://huggingface.co/datasets/Battam/training-data-foundation/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Index fields</a></li> | |
| <li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li> | |
| <li><a href="#manual" data-i18n="nav.manual">Manual</a></li> | |
| <li><a href="#top" data-i18n="foot.top">Back to top</a></li> | |
| </ul> | |
| <p class="foot-pin mono" id="foot-pin"></p> | |
| </div> | |
| </footer> | |
| </body> | |
| </html> | |