Battam's picture
portal: data page of repository commit 544a0b41b806
8c4f803 verified
Raw History Blame Contribute Delete
23.3 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title data-i18n="page.title">Training Data Foundation</title>
<link rel="icon" href="data:,">
<link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents">
<meta name="description" content="Real public data for pre-training, mid-training and post-training from Battam/training-data-foundation, the public index of the Training Data Foundation: pick by training stage, domain and form, and take it with one command." data-i18n-attr="content:page.description">
<!-- #2422: both themes, by the reader's own setting (style.css prefers-color-scheme) -->
<meta name="color-scheme" content="light dark">
<meta name="theme-color" content="#08131b" media="(prefers-color-scheme: dark)">
<meta name="theme-color" content="#e9eff3" media="(prefers-color-scheme: light)">
<link rel="preconnect" href="https://huggingface.co">
<link rel="preconnect" href="https://cdn.jsdelivr.net" crossorigin>
<link rel="preload" href="locales/en.json" as="fetch" crossorigin>
<link rel="preload" href="locales/zh.json" as="fetch" crossorigin>
<link rel="preload" href="fonts/plex-sans-400.woff2" as="font" type="font/woff2" crossorigin>
<link rel="stylesheet" href="style.css">
<script type="module" src="first.js" async></script>
<script type="module" src="app.js"></script>
</head>
<body>
<nav class="topnav" aria-label="Page sections" data-i18n-attr="aria-label:nav.sections">
<div class="nav-row">
<!-- the mark: three strata, in the colors of the three training stages -->
<a class="brand" href="#top"><svg class="mark" width="26" height="26" viewBox="0 0 26 26" aria-hidden="true"><path d="M13 3 23 8 13 13 3 8z" fill="var(--post)"/><path d="M3 12.5 13 17.5 23 12.5" fill="none" stroke="var(--mid)" stroke-width="2.2" stroke-linejoin="round"/><path d="M3 17.5 13 22.5 23 17.5" fill="none" stroke="var(--pre)" stroke-width="2.2" stroke-linejoin="round"/></svg><span data-i18n="page.title">Training Data Foundation</span></a>
<!-- #2422: the page's three parts: the foundation (the working screen), the sources and Get the data -->
<ol class="progress">
<li><a href="#top" data-part="stages"><span class="pl pl-long" data-i18n="nav.stages">The foundation</span><span class="pl pl-short" data-i18n="nav.stagesShort">Foundation</span><span class="track"><span class="fill"></span></span></a></li>
<li><a href="#browse" data-part="pick"><span class="pl pl-long" data-i18n="nav.pick">Sources</span><span class="pl pl-short" data-i18n="nav.pickShort">Sources</span><span class="track"><span class="fill"></span></span></a></li>
<li><a href="#get" data-part="get"><span class="pl pl-long" data-i18n="nav.get">Get the data</span><span class="pl pl-short" data-i18n="nav.getShort">Get</span><span class="track"><span class="fill"></span></span></a></li>
</ol>
<div class="nav-links">
<a href="https://huggingface.co/datasets/Battam/training-data-foundation" data-i18n="link.dataset">Dataset</a>
<a href="#manual" data-i18n="nav.manual">Manual</a>
<a href="llms.txt" data-i18n="nav.agents">For agents</a>
<a class="btn solid small-btn" href="https://huggingface.co/datasets/Battam/training-data-foundation/discussions/new" data-i18n="nav.request">Request data</a>
<!-- the language switch: the other language's name, in that language (i18n.js) -->
<button class="btn small-btn lang-btn" id="lang-switch" type="button" hidden></button>
</div>
</div>
</nav>
<main id="page">
<!-- #2422, second round ("Strata"): the first screen is a working screen. Its ground is the field (field.js, drawn
with WebGL): every source of the index one point, colored by its stage group and sized by its deduplicated size,
in three arrangements. Glass panels float on it from 1100 px: left, the name, the sentence, the stages of the
stage table as a column (first.js drawStages, app.js renderStageFig: a click on one sets the filter by stage)
and the filters; right, the count of the selection and its sources (app.js renderWork); at the bottom, the one
command that takes the selection (renderDock). Below 1100 px the field is on top and the panels follow in the
flow. Without WebGL the panels keep their places and one line says what the ground would show -->
<header class="work" id="top" data-part="stages">
<div class="field" id="field">
<!-- the outline of four strata while the scene is loading, so the ground is never bare -->
<svg class="field-ghost" viewBox="0 0 700 470" aria-hidden="true"><path d="M20 95 190 10 680 65 590 180z"/><path d="M20 190 190 105 680 160 590 275z"/><path d="M20 285 190 200 680 255 590 370z"/><path d="M20 380 190 295 680 350 590 465z"/></svg>
<canvas role="img" tabindex="0" aria-label="Every source of the index as one point, layered by training stage" data-i18n-attr="aria-label:field.label"></canvas>
<!-- the ring of the point under the pointer and the two rings of the open source (field.js places them) -->
<svg class="field-ov" aria-hidden="true"><g class="field-ring" visibility="hidden"><circle/></g><g class="field-open" visibility="hidden"><circle class="a"/><circle class="b"/></g></svg>
<!-- the labels are a second way to the stage bands and the domain filter, for the pointer only -->
<div class="field-labs" aria-hidden="true"></div>
<div class="field-call" hidden></div>
<div class="field-tip" hidden></div>
</div>
<p class="field-empty" id="field-empty" hidden><span data-i18n="sources.none">No source matches</span><button class="link" id="field-clear" type="button" data-i18n="filter.clear">Clear</button></p>
<!-- what a point's light says -->
<p class="field-legend" id="field-legend" hidden><span><i class="l1"></i><span data-i18n="field.legendOn">in the selection</span></span><span><i class="l2"></i><span data-i18n="field.legendHalf">kept, in the other layer</span></span><span><i class="l3"></i><span data-i18n="field.legendOff">filtered out</span></span></p>
<p class="field-off" id="field-off" hidden data-i18n="field.off">The ground of this screen is a field of points, one per source, drawn with WebGL, which this browser does not offer. Everything else on the page works without it.</p>
<div class="panel views" id="views" hidden>
<span class="seg" role="group" aria-label="Arrangement" data-i18n-attr="aria-label:field.views" id="view-seg">
<button type="button" data-view="strata" aria-pressed="true" data-i18n="field.strata">Strata</button>
<button type="button" data-view="domains" aria-pressed="false" data-i18n="field.domains">Domains</button>
<button type="button" data-view="scale" aria-pressed="false" data-i18n="field.scale">Scale</button>
</span>
<span class="seg zoom" role="group" aria-label="Zoom" data-i18n-attr="aria-label:field.zoom">
<button type="button" id="zoom-out" aria-label="Zoom out" data-i18n-attr="aria-label:field.zoomOut">&minus;</button>
<button type="button" id="zoom-in" aria-label="Zoom in" data-i18n-attr="aria-label:field.zoomIn">+</button>
<button type="button" id="zoom-reset" aria-label="Back to the first view" title="Back to the first view" data-i18n-attr="aria-label:field.reset;title:field.reset"><svg width="13" height="13" viewBox="0 0 16 16" aria-hidden="true"><path d="M3 8a5 5 0 1 0 1.6-3.7M3 2.5v3h3" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round"/></svg></button>
</span>
</div>
<p class="field-hint" id="field-hint" hidden data-i18n="field.hint">Drag to turn 路 Ctrl + scroll to zoom 路 point at a source</p>
<aside class="panel work-left">
<div class="pad">
<p class="eyebrow" data-i18n="page.eyebrow">The public index</p>
<h1 data-i18n="page.title">Training Data Foundation</h1>
<p class="lede" data-i18n="hero.lede">Real public data for pre-training, mid-training and post-training: each file with its address, size, hashes and license, fetched with one command.</p>
<!-- the release the page reads, in one line: its tag, its day and a link to every release (first.js drawRelease) -->
<p class="release-line" id="release-line"><span class="sk sk-line"></span></p>
</div>
<!-- the stratigraphic column: one band per stage of the stage table, top to bottom as the field stacks them, each
with its funnel, and under them what no stage counts, by reason -->
<figure class="stage-fig" aria-labelledby="stage-fig-h">
<figcaption class="eyebrow" id="stage-fig-h" data-i18n="stage.figTitle">What each training stage can use</figcaption>
<ul class="column" id="stage-tiles"><li><div class="stratum sk-tile"></div></li><li><div class="stratum sk-tile"></div></li><li><div class="stratum sk-tile"></div></li></ul>
<!-- a missing input, one short line each (app.js renderStageFig) -->
<p class="stage-why" id="stage-why" hidden></p>
</figure>
<!-- the words the page counts in and sorts by (material set, corpus, unit, domain): each a button that opens its
meaning in place, as the index's vocabulary gives it (app.js renderTerms); the manual's glossary has the rest;
#2614: the popover they open follows the manual's view, outside the page, as the manual's start has the line too -->
<p class="terms" id="terms" hidden></p>
<!-- #2422: the filters: the search, then the domain, the form, the size, the training permit and the personal-data
mark (two labels every source carries, which the page filters on and counts and never leaves anything out of a
count for) and the file types. The stage is set by its band above, so its select stays out of sight. A filter
the panel does not show (collection, subdomain, license tier, redistribution, status) still holds from the
address, as a chip that removes it -->
<div class="filters" id="filters" role="group" aria-label="Filters" data-i18n-attr="aria-label:filter.title">
<button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row">
<span data-i18n="filter.title">Filters</span><span class="state" id="filter-state" data-i18n="filter.noneOn">none on</span>
</button>
<div class="filter-row" id="filter-row">
<p class="eyebrow filter-title" data-i18n="filter.title">Filters</p>
<label class="wide"><span data-i18n="filter.search">Search</span><span class="q-wrap"><input id="f-q" type="search" placeholder="Title or text" data-i18n-attr="placeholder:filter.searchHint" autocomplete="off"><kbd aria-hidden="true">/</kbd></span></label>
<!-- #2006: the training stages of the stage table (catalog/stages.yaml), computed when the page is built
(stages.json); the bands above set it -->
<label hidden><span data-i18n="filter.stage">Training stage</span><select id="f-stage"><option value="" data-i18n="filter.all">All</option></select><span id="f-stage-note" hidden></span></label>
<label><span data-i18n="filter.domain">Domain</span><select id="f-domain"><option value="" data-i18n="filter.all">All</option></select></label>
<label><span data-i18n="filter.form">Form</span><select id="f-form"><option value="" data-i18n="filter.all">All</option></select></label>
<label><span data-i18n="filter.size">Size</span><select id="f-size">
<option value="" data-i18n="filter.any">Any</option>
<option value="lt1g" data-i18n="filter.size.lt1g">Under 1 GB</option>
<option value="1g1t" data-i18n="filter.size.1g1t">1 GB to 1 TB</option>
<option value="ge1t" data-i18n="filter.size.ge1t">1 TB or more</option>
<option value="unknown" data-i18n="filter.size.unknown">Not published</option>
</select></label>
<label><span data-i18n="filter.training">Training</span><select id="f-train">
<option value="" data-i18n="filter.any">Any</option>
<option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
<option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
</select></label>
<!-- #2422: the personal-data mark of a source (its catalog pii.level) -->
<label><span data-i18n="filter.pii">Personal data</span><select id="f-pii">
<option value="" data-i18n="filter.any">Any</option>
<option value="none" data-i18n="filter.pii.none">None found</option>
<option value="marked" data-i18n="filter.pii.marked">Present or suspected</option>
<option value="unknown" data-i18n="filter.pii.unknown">Not judged yet</option>
</select></label>
<!-- #1524: the file types of the layer shown with their rows (app.js fillExt), any number checked; folded
while none is checked, its summary saying how many are -->
<details class="ext-fold" id="ext-fold">
<summary><span data-i18n="filter.ext">File type</span><span class="ext-on" id="ext-on"></span></summary>
<div class="ext-list" id="f-ext" role="group" aria-label="File type" data-i18n-attr="aria-label:filter.ext"></div>
</details>
<div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled data-i18n="filter.clear">Clear</button></div>
</div>
</div>
<p class="work-foot">
<span class="chip" id="preview-chip" hidden></span>
<span class="notice" id="notice" hidden></span>
</p>
</aside>
<aside class="panel work-right" aria-labelledby="sel-h">
<div class="pad">
<div class="sel-top"><p class="eyebrow" id="sel-h" data-i18n="chip.selection">Selection</p>
<!-- the layer switch (#1083): the list, the charts and the command take one layer, material sets by default -->
<span class="seg" role="group" aria-label="Show" data-i18n-attr="aria-label:layer.label" id="layer-seg">
<button type="button" data-kind="material_set" aria-pressed="true" data-i18n="kind.material_set">Material sets</button>
<button type="button" data-kind="corpus" aria-pressed="false" data-i18n="kind.corpus">Corpora</button>
</span>
</div>
<p class="count" aria-live="polite"><b class="num" id="sel-n"><span class="sk sk-num"></span></b><span id="sel-word"></span></p>
<p class="meta" id="sel-meta"></p>
<!-- the selection: one removable chip per filter that is on (app.js renderChips) -->
<div class="sel-chips" id="sel-chips"></div>
</div>
<!-- the open source, in short (app.js renderCard); its whole detail is in the list below the screen -->
<div id="sel-card" hidden></div>
<ul class="src-list" id="sel-list"><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li><li class="sk sk-row"></li></ul>
<p class="panel-foot"><a href="#sources" id="sel-all"></a></p>
</aside>
<section class="panel work-dock" aria-labelledby="dock-h">
<div class="dock-top">
<p class="eyebrow" id="dock-h" data-i18n="get.take">Take</p>
<span class="dock-word" id="dock-units" data-i18n="get.units">Units per source, at most</span>
<span class="seg" role="group" aria-labelledby="dock-units" id="dock-sample"></span>
<button class="btn solid small-btn" id="dock-copy" type="button" data-i18n="dock.copy">Copy the command</button>
</div>
<pre class="dock-cmd" id="dock-cmd" tabindex="0"><span class="sk sk-line"></span></pre>
<p class="dock-foot"><a href="#get" data-i18n="dock.more">SQL, CSV and what it writes</a></p>
</section>
</header>
<!-- #2422: the sources: the list of the selection, each source with its fit, its details and a preview of its rows,
and the domain by form map with the bars by domain and by form, linked to the filters -->
<div class="band" id="browse" data-part="pick">
<div class="wrap browse-main">
<section id="sources" aria-labelledby="sources-h">
<div class="band-head reveal"><h2 id="sources-h"><span class="sk sk-line"></span></h2></div>
<div class="card table-card" id="sources-body"><div class="sk sk-row"></div><div class="sk sk-row"></div><div class="sk sk-row"></div><div class="sk sk-row"></div></div>
</section>
<section id="map" aria-labelledby="map-h">
<div class="band-head row reveal">
<h2 id="map-h"><span class="sk sk-line"></span></h2>
<span class="seg-wrap" id="metric-wrap"><span class="seg-label" id="metric-label" data-i18n="metric.label">Count</span>
<span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
<button type="button" data-metric="sets" aria-pressed="true" data-i18n="metric.sets">Material sets</button>
<button type="button" data-metric="units" aria-pressed="false" data-i18n="metric.units">Units</button>
</span>
</span>
</div>
<div class="map-cards">
<figure class="card chart-card" id="heat-fig"><div class="sk sk-block tall"></div></figure>
<div class="card chart-card side-card">
<figure class="chart" id="dom-fig"><div class="sk sk-block"></div></figure>
<figure class="chart" id="form-fig"></figure>
</div>
</div>
</section>
</div>
</div>
<!-- #2422: Get the data: the selection's one command, what it writes and what the default profile of the fetch tool
leaves out with the flag that takes each back, and the selection file (app.js renderGet) -->
<section class="band band-alt" id="get" data-part="get" aria-labelledby="get-h">
<div class="wrap">
<div class="band-head reveal"><h2 id="get-h"><span class="sk sk-line"></span></h2></div>
<div id="get-body"><div class="sk sk-block tall"></div></div>
<!-- #1498, #1524: the usage example video from media/ of the dataset, another origin, hence crossorigin for its
track; app.js renderVideo plays the video and the subtitles of the page's language. Its pictures carry no
words, so the track is the default one. The dataset card and the project page link to this block by its id.
#2066: beside it the second usage video, for people who build data from this page, added the same way
(catalog/media.tsv: usage-people.<en or zh>.mp4 and .vtt) -->
<div class="video-grid">
<div class="card video-card reveal" id="usage-video">
<h3 data-i18n="video.title">Usage example video</h3>
<p class="small" data-i18n="video.text">The index in use from this page, in the page's language, with subtitles.</p>
<video id="video-player" data-lang="en" controls preload="metadata" crossorigin="anonymous" src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-example.en.mp4">
<track kind="subtitles" srclang="en" default src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-example.en.vtt" label="English">
</video>
</div>
<div class="card video-card reveal" id="usage-people">
<h3 data-i18n="video.peopleTitle">Usage video for people who build data</h3>
<p class="small" data-i18n="video.peopleText">The manual's worked example on this page, from one need to the files, in the page's language, with subtitles.</p>
<video id="people-player" data-lang="en" controls preload="metadata" crossorigin="anonymous" src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-people.en.mp4">
<track kind="subtitles" srclang="en" default src="https://huggingface.co/datasets/Battam/training-data-foundation/resolve/main/media/usage-people.en.vtt" label="English">
</video>
</div>
</div>
<div class="card agents-card lift reveal" id="agents">
<span class="agents-ico" data-icon="robot" data-size="36"></span>
<div class="agents-text">
<h3 id="agents-h" data-i18n="nav.agents">For agents</h3>
<p class="small" data-i18n="agents.text">The index's columns 路 the one command 路 the default profile 路 what to keep out of the agent's environment</p>
</div>
<div class="agents-actions">
<a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a>
<a class="btn" href="https://huggingface.co/datasets/Battam/training-data-foundation/blob/main/docs/for-agents.md" data-i18n-attr="href:url.forAgents" data-icon="doc" data-size="16"><span data-i18n="agents.readme">Index docs: For agents</span></a>
</div>
</div>
</div>
</section>
</main>
<!-- #1617: the user manual, a view of its own at the address #manual (app.js showView, manual.js), in place of the page
above: manual/manual.<language>.md of this folder, its numbers and command outputs filled at deploy
(03_data_line/steps/portal.py), its images manual/shots/<output>, taken on this page after each deploy
(03_data_line/steps/shots.py); #2614: its contents beside it from 1024 px, a menu above it below that -->
<main class="manual-view" id="manual" hidden aria-label="User manual" data-i18n-attr="aria-label:manual.title">
<div class="wrap manual-wrap">
<p class="manual-back"><a href="#top" data-i18n="manual.back">Back to the page</a></p>
<div class="manual-grid">
<nav class="manual-toc" id="manual-toc" aria-label="Contents" data-i18n-attr="aria-label:manual.contents"></nav>
<article class="card manual-body" id="manual-body"><div class="sk sk-block tall"></div></article>
</div>
</div>
</main>
<div class="term-pop" id="term-pop" popover></div>
<footer class="site-foot" id="request" data-part="get">
<div class="wrap foot-grid">
<div class="foot-cta">
<span class="foot-ico" data-icon="request" data-size="32"></span>
<div>
<h2 data-i18n="foot.missing">Missing data?</h2>
<p data-i18n="foot.ask">Ask for a domain, a form or a source.</p>
</div>
<a class="btn on-dark-solid" href="https://huggingface.co/datasets/Battam/training-data-foundation/discussions/new" data-i18n="nav.request">Request data</a>
</div>
<ul class="foot-links">
<li><a href="https://huggingface.co/datasets/Battam/training-data-foundation" data-i18n="link.dataset">Dataset</a></li>
<li><a href="https://huggingface.co/datasets/Battam/training-data-foundation/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Index fields</a></li>
<li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li>
<li><a href="#manual" data-i18n="nav.manual">Manual</a></li>
<li><a href="#top" data-i18n="foot.top">Back to top</a></li>
</ul>
<p class="foot-pin mono" id="foot-pin"></p>
</div>
</footer>
</body>
</html>