Spaces:
Running
Running
data-line commited on
Commit ·
fcd1156
1
Parent(s): 442062d
portal: catalog page of repository commit 2da961963825
Browse filesFiles of catalog/portal/ (12), digest 4e64fccdd34a52a2ad12f75da2950a09c332c6acc32c3fe2706eb47f22ff476e.
- README.md +28 -1
- app.js +267 -245
- charts.js +39 -27
- data.js +5 -3
- first.js +12 -13
- first.json +7 -7
- i18n.js +67 -0
- index.html +72 -68
- llms.txt +2 -2
- locales/en.json +379 -0
- locales/zh.json +343 -0
- portal.json +4 -4
- style.css +7 -2
README.md
CHANGED
|
@@ -84,12 +84,36 @@ the code. Maps from code to name, and nested `children`, are read too.
|
|
| 84 |
| `vocabulary=<url>` | read the names of domains and forms from this URL instead of `meta/vocabulary.json` |
|
| 85 |
| `domain`, `leaf`, `form`, `train`, `redist`, `tier`, `size`, `status`, `q` | the filters (`train` and `redist`: `yes` for every row allowed, `cond` for allowed or with conditions; `status` is the kind: `ready`, `raw` or `flagged`) |
|
| 86 |
| `metric`, `sort`, `dir`, `open` | the map's count, the sort of the source list (default: fit), the open source |
|
|
|
|
| 87 |
|
| 88 |
The filters scope the map, the source list and "Get the data"; the overview, the history and "Not included" describe the
|
| 89 |
whole index. The licence filters keep a source only when every one of its rows has that permission
|
| 90 |
(`meta/sources.json`, `license_rows`), so the counts, the SQL and the CSV agree without a per-row licence condition.
|
| 91 |
"Source size" is the total known size of a source (referenced).
|
| 92 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
## Files
|
| 94 |
|
| 95 |
- `index.html`, `style.css`: the frame, in layers: a tinted canvas, a deep slate hero band with the one big number,
|
|
@@ -104,6 +128,9 @@ whole index. The licence filters keep a source only when every one of its rows h
|
|
| 104 |
the same index commit they are the same numbers; when main has moved on, the changed ones are briefly highlighted.
|
| 105 |
With `?rev=`, `?catalog=` or `?vocabulary=` the hero waits for the index (`first.json` describes main).
|
| 106 |
- `data.js`: reading the commit (hyparquet for the parquet files, byte ranges for data files).
|
|
|
|
|
|
|
|
|
|
| 107 |
- `charts.js`: thin bars, columns, the stacked split, the heatmap, the tooltip, the chart and table toggle, and `vt`
|
| 108 |
(views and tabs switch with the View Transitions API where there is one and motion is welcome).
|
| 109 |
- `icons.js`: the pictograms (inline SVG, one stroke style) for the forms, the domains, the facts and the four parts.
|
|
@@ -129,7 +156,7 @@ Fonts: Inter (400 to 800) for text and numbers, Noto Sans SC for CJK, IBM Plex M
|
|
| 129 |
Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
|
| 130 |
|
| 131 |
Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
|
| 132 |
-
are zstd-compressed). There is no build step. To look at it locally: `python -m http.server` in this folder (without a
|
| 133 |
`first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
|
| 134 |
writes one).
|
| 135 |
|
|
|
|
| 84 |
| `vocabulary=<url>` | read the names of domains and forms from this URL instead of `meta/vocabulary.json` |
|
| 85 |
| `domain`, `leaf`, `form`, `train`, `redist`, `tier`, `size`, `status`, `q` | the filters (`train` and `redist`: `yes` for every row allowed, `cond` for allowed or with conditions; `status` is the kind: `ready`, `raw` or `flagged`) |
|
| 86 |
| `metric`, `sort`, `dir`, `open` | the map's count, the sort of the source list (default: fit), the open source |
|
| 87 |
+
| `lng=<en or zh>` | open the page in English or in Simplified Chinese (and keep that choice) |
|
| 88 |
|
| 89 |
The filters scope the map, the source list and "Get the data"; the overview, the history and "Not included" describe the
|
| 90 |
whole index. The licence filters keep a source only when every one of its rows has that permission
|
| 91 |
(`meta/sources.json`, `license_rows`), so the counts, the SQL and the CSV agree without a per-row licence condition.
|
| 92 |
"Source size" is the total known size of a source (referenced).
|
| 93 |
|
| 94 |
+
## Languages
|
| 95 |
+
|
| 96 |
+
The page writes its own words in English or in Simplified Chinese, with i18next and its browser language detector.
|
| 97 |
+
Every word the page itself writes comes from i18next's `t` by key: headings, labels, buttons, notes, tooltips, `title`
|
| 98 |
+
and `aria-label` attributes, table headers, chart titles, axes and legends, the first screen, the empty, loading and
|
| 99 |
+
error states, the document title and the meta description, and the labels the page maps codes to (licence tiers,
|
| 100 |
+
reasons for leaving out, kinds of unit, the fallback names of domains).
|
| 101 |
+
The English words are in `locales/en.json`, the Chinese ones in `locales/zh.json`, with the same keys (flat keys, since
|
| 102 |
+
a key may hold a code with dots such as `domain.D10.2`).
|
| 103 |
+
Counts use i18next's plurals (`_one` and `_other` in English; Chinese has only `_other`);
|
| 104 |
+
numbers and dates are formatted with `Intl.NumberFormat` and `Intl.DateTimeFormat` in the current language.
|
| 105 |
+
What the page reads from the dataset or from `first.json` stays as it is in both languages: source names, titles,
|
| 106 |
+
summaries, descriptions, licence names, statements and notes, and the names of domains and forms from the vocabulary.
|
| 107 |
+
The issue a "Request" link opens is prefilled in English, the language of the hub.
|
| 108 |
+
|
| 109 |
+
The switch is one button in the header that names the other language in that language.
|
| 110 |
+
It changes the language without a reload and sets `<html lang>` (`en` or `zh-CN`; Chinese text then uses Noto Sans SC
|
| 111 |
+
first, `:lang(zh)` in `style.css`).
|
| 112 |
+
The detector reads the query string first (`?lng=zh`), then localStorage, and keeps the choice in localStorage
|
| 113 |
+
(`i18nextLng`), so the page opens in English unless the reader chose Chinese;
|
| 114 |
+
the switch drops a `lng` from the address, so the kept choice holds on the next visit.
|
| 115 |
+
`llms.txt` and this README stay in English.
|
| 116 |
+
|
| 117 |
## Files
|
| 118 |
|
| 119 |
- `index.html`, `style.css`: the frame, in layers: a tinted canvas, a deep slate hero band with the one big number,
|
|
|
|
| 128 |
the same index commit they are the same numbers; when main has moved on, the changed ones are briefly highlighted.
|
| 129 |
With `?rev=`, `?catalog=` or `?vocabulary=` the hero waits for the index (`first.json` describes main).
|
| 130 |
- `data.js`: reading the commit (hyparquet for the parquet files, byte ranges for data files).
|
| 131 |
+
- `i18n.js`: the page's languages (i18next with its browser language detector, the words from `locales/en.json` and
|
| 132 |
+
`locales/zh.json`), the words of `index.html` (`data-i18n` sets an element's text, `data-i18n-attr` its attributes)
|
| 133 |
+
and the language switch; the other modules draw their text again on i18next's `languageChanged`.
|
| 134 |
- `charts.js`: thin bars, columns, the stacked split, the heatmap, the tooltip, the chart and table toggle, and `vt`
|
| 135 |
(views and tabs switch with the View Transitions API where there is one and motion is welcome).
|
| 136 |
- `icons.js`: the pictograms (inline SVG, one stroke style) for the forms, the domains, the facts and the four parts.
|
|
|
|
| 156 |
Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
|
| 157 |
|
| 158 |
Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
|
| 159 |
+
are zstd-compressed), `i18next@26.4.5` and `i18next-browser-languagedetector@8.2.1`. There is no build step. To look at it locally: `python -m http.server` in this folder (without a
|
| 160 |
`first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
|
| 161 |
writes one).
|
| 162 |
|
app.js
CHANGED
|
@@ -2,24 +2,16 @@
|
|
| 2 |
// at a glance, take it with one command. Graphics first; every chart keeps a short label and a table view.
|
| 3 |
import { DATASET_URL, fileUrl, loadBatchDates, loadIndex, num, parquetRows, readColumns, REPO, topOf } from './data.js';
|
| 4 |
import {
|
| 5 |
-
clear, columns, el, figureWithToggle, fmtBytes, fmtCompact, fmtInt, fmtPct, fmtShort, heatmap,
|
| 6 |
-
segmented, stack, table, vt, withTip,
|
| 7 |
} from './charts.js';
|
| 8 |
import { decorate, drawHero, markFromIndex, setNumber } from './first.js';
|
|
|
|
| 9 |
import { domainIcon, formIcon, icon } from './icons.js';
|
| 10 |
|
| 11 |
-
// The categories the group asks for most, shown first.
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
['D3.1', 'ETL and multi-source merging'], ['D3.6', 'Visualisation and reports'], ['D2.5', 'File systems and recovery'],
|
| 15 |
-
['D2.6', 'Crash and performance analysis'], ['D2.7', 'Virtualisation and emulators'],
|
| 16 |
-
['D2.8', 'Large text and log editing'], ['D8.2', 'Reverse engineering and binaries'],
|
| 17 |
-
];
|
| 18 |
-
// names of the top-level domains until meta/vocabulary.json is published
|
| 19 |
-
const TOP_NAMES = {
|
| 20 |
-
D1: 'Software engineering', D2: 'System administration', D3: 'Data science', D4: 'Machine learning', D5: 'Science',
|
| 21 |
-
D6: 'Operations', D7: 'Hardware', D8: 'Security', D9: 'Media', D10: 'Misc',
|
| 22 |
-
};
|
| 23 |
const GITHUB = 'https://github.com/msra-sg/agentic-env-pipeline';
|
| 24 |
const CSV_MAX_ROWS = 200000;
|
| 25 |
const ROW_COLUMNS = ['source', 'batch', 'uri', 'range_offset', 'range_length', 'member_path', 'size_bytes', 'sha256'];
|
|
@@ -31,45 +23,41 @@ const MAX_FILE = 20e6;
|
|
| 31 |
const MAX_LISTED = 30;
|
| 32 |
const $ = (id) => document.getElementById(id);
|
| 33 |
|
| 34 |
-
// ---- plain names (codes only inside the SQL and the commands)
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
'no-training': 'No training', 'internal-only': 'Internal only', mixed: 'Mixed', unknown: 'Not stated',
|
| 38 |
-
};
|
| 39 |
-
const PERMIT = { yes: 'Allowed', no: 'Not allowed', conditional: 'Conditional', unknown: 'Not stated' };
|
| 40 |
const PERMIT_ORDER = ['yes', 'no', 'conditional', 'unknown'];
|
| 41 |
const PERMIT_COLOR = { yes: 'var(--ours)', no: 'var(--problem)', conditional: 'var(--fill-3)', unknown: 'var(--other)' };
|
| 42 |
-
const
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
other: 'Other',
|
| 47 |
-
};
|
| 48 |
-
const UNIT_KIND = { file: 'a file', 'API response': 'an API response', 'WARC record': 'a record in a web archive', 'archive member': 'a file inside an archive', 'byte range': 'a byte range of a bigger file' };
|
| 49 |
-
// file type -> the tool that opens it
|
| 50 |
const TOOLS = {
|
| 51 |
xlsx: 'openpyxl', xlsm: 'openpyxl', xls: 'LibreOffice', ods: 'LibreOffice', csv: 'DuckDB', tsv: 'DuckDB', parquet: 'DuckDB',
|
| 52 |
json: 'jq', jsonl: 'jq', ndjson: 'jq', pdf: 'pdftotext', doc: 'LibreOffice', docx: 'python-docx', ppt: 'LibreOffice',
|
| 53 |
pptx: 'python-pptx', gz: 'gzip', tgz: 'tar', tar: 'tar', bz2: 'bzip2', xz: 'xz', zst: 'zstd', zip: 'unzip', whl: 'unzip',
|
| 54 |
jar: 'unzip', '7z': '7-Zip', crate: 'tar', deb: 'dpkg-deb', rpm: 'rpm2cpio', warc: 'warcio', arc: 'warcio', pgn: 'python-chess',
|
| 55 |
-
py: 'Python',
|
| 56 |
-
yaml: 'yq', yml: 'yq',
|
| 57 |
safetensors: 'safetensors', final: 'TensorBoard', ipynb: 'Jupyter', eml: 'Python email', mbox: 'Python mailbox',
|
| 58 |
ics: 'icalendar', e01: 'The Sleuth Kit', img: 'The Sleuth Kit', dd: 'The Sleuth Kit', raw: 'The Sleuth Kit',
|
| 59 |
ys: 'Yosys', v: 'Icarus Verilog', sv: 'Verilator', go: 'Go', rs: 'Cargo', zig: 'Zig', lua: 'Lua', rb: 'Ruby', c: 'gcc',
|
| 60 |
-
tcl: 'tclsh',
|
| 61 |
};
|
|
|
|
|
|
|
| 62 |
const NONE = '_none';
|
| 63 |
const MULTI = '_multi';
|
| 64 |
const OTHER = '_other';
|
| 65 |
const human = (code) => String(code).replace(/[_-]+/g, ' ').replace(/^./, (c) => c.toUpperCase());
|
| 66 |
function sentence(s) {
|
| 67 |
-
let
|
| 68 |
-
if (!/\s/.test(
|
| 69 |
-
|
| 70 |
-
return
|
| 71 |
}
|
| 72 |
-
|
|
|
|
| 73 |
const sum = (xs, f) => xs.reduce((a, x) => a + f(x), 0);
|
| 74 |
|
| 75 |
let IX = null;
|
|
@@ -99,15 +87,14 @@ const vocabName = (sec, code) => {
|
|
| 99 |
const n = IX.vocab[sec].names.get(code);
|
| 100 |
return n ? sentence(n) : null;
|
| 101 |
};
|
| 102 |
-
function domainName(key) {
|
| 103 |
-
if (key === NONE) return '
|
| 104 |
-
if (key === MULTI) return vocabName('domain', 'multi') || '
|
| 105 |
-
|
| 106 |
-
return vocabName('domain', key) || fallback || key;
|
| 107 |
}
|
| 108 |
-
function formName(key) {
|
| 109 |
-
if (key === NONE || key == null) return '
|
| 110 |
-
if (key === OTHER) return '
|
| 111 |
return vocabName('form', key) || sentence(key);
|
| 112 |
}
|
| 113 |
const domainKeysOf = (s) => (s.domainKind === 'codes' ? s.tops : [s.domainKind === 'multi' ? MULTI : NONE]);
|
|
@@ -121,17 +108,18 @@ function sourceDomainText(s) {
|
|
| 121 |
}
|
| 122 |
const inCategory = (s, code) => (code.includes('.') ? s.codes.includes(code) : s.tops.includes(code));
|
| 123 |
|
| 124 |
-
// ---- fit: what decides whether a source can feed a task (yes, part, no, unknown)
|
| 125 |
const FIT = [
|
| 126 |
-
{ k: 'ship',
|
| 127 |
-
{ k: 'bench',
|
| 128 |
-
{ k: 'pii',
|
| 129 |
-
{ k: 'size',
|
| 130 |
-
{ k: 'alone',
|
| 131 |
-
{ k: 'answers',
|
| 132 |
-
{ k: 'login',
|
| 133 |
];
|
| 134 |
-
const
|
|
|
|
| 135 |
function fitOf(s) {
|
| 136 |
const q = s.info.size_quantiles_upper_bytes || {};
|
| 137 |
const c = s.contamination?.level;
|
|
@@ -149,14 +137,14 @@ function fitOf(s) {
|
|
| 149 |
};
|
| 150 |
}
|
| 151 |
function fitDot(state, label) {
|
| 152 |
-
return el('span', { class: `fit fit-${state}`, role: 'img', 'aria-label':
|
| 153 |
}
|
| 154 |
function fitStrip(s) {
|
| 155 |
const f = fitOf(s);
|
| 156 |
-
return el('span', { class: 'fit-strip' }, FIT.map((x) => withTip(fitDot(f[x.k], x.
|
| 157 |
}
|
| 158 |
function fitKey() {
|
| 159 |
-
return el('span', { class: 'fit-key' }, ['yes', 'part', 'no', 'unknown'].map((k) => el('span', {}, el('span', { class: `fit fit-${k}`, 'aria-hidden': 'true' }),
|
| 160 |
}
|
| 161 |
|
| 162 |
// ---- filters
|
|
@@ -193,22 +181,23 @@ function passes(s, skip = {}) {
|
|
| 193 |
const selected = () => IX.sources.filter((s) => passes(s));
|
| 194 |
const FILTER_KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q'];
|
| 195 |
const activeFilters = () => FILTER_KEYS.filter((k) => F[k]);
|
| 196 |
-
|
|
|
|
| 197 |
const out = [];
|
| 198 |
-
const
|
| 199 |
-
if (F.domain) out.push(['
|
| 200 |
-
if (F.form) out.push(['
|
| 201 |
-
if (F.train) out.push(['
|
| 202 |
-
if (F.redist) out.push(['
|
| 203 |
-
if (F.tier) out.push(['
|
| 204 |
-
if (F.size) out.push(['
|
| 205 |
-
if (F.status) out.push(['
|
| 206 |
-
if (F.q) out.push(['
|
| 207 |
return out;
|
| 208 |
}
|
| 209 |
function metricOf(s, m = F.metric) { return m === 'sources' ? 1 : m === 'rows' ? s.rows : s.distinct || 0; }
|
| 210 |
const metricFmt = (m = F.metric) => (m === 'bytes' ? fmtBytes : (v) => fmtInt(v));
|
| 211 |
-
const
|
| 212 |
|
| 213 |
function setFilter(patch) {
|
| 214 |
Object.assign(F, patch);
|
|
@@ -241,7 +230,7 @@ function selectionArgs(sel) {
|
|
| 241 |
return { args: commandArgs(sel, { domain, form: F.form }), listed: false };
|
| 242 |
}
|
| 243 |
async function copy(text, state) {
|
| 244 |
-
try { await navigator.clipboard.writeText(text); state.textContent = '
|
| 245 |
setTimeout(() => { state.textContent = ''; }, 2400);
|
| 246 |
}
|
| 247 |
function copyButton(label, text, { dark = false, iconName = 'copy' } = {}) {
|
|
@@ -265,24 +254,18 @@ function catStats(code) {
|
|
| 265 |
return { sources: S.length, readySources: S.filter((s) => s.isTask).length, ready, raw, onReady, ship, shipOf: onReady ? ready : raw };
|
| 266 |
}
|
| 267 |
const bySize = (a, b) => b.st.ready - a.st.ready || b.st.raw - a.st.raw || b.st.sources - a.st.sources;
|
| 268 |
-
function requestUrl(
|
| 269 |
const req = new URL(`${GITHUB}/issues/new`);
|
| 270 |
req.searchParams.set('template', 'data_request.yml');
|
| 271 |
-
req.searchParams.set('what',
|
| 272 |
return req.toString();
|
| 273 |
}
|
| 274 |
-
/** 8970000 -> ['8.97', 'million']; small numbers whole. */
|
| 275 |
-
function bigParts(v) {
|
| 276 |
-
const t = fmtCompact(v);
|
| 277 |
-
const m = t.match(/^([\d.,]+) (thousand|million|billion)$/);
|
| 278 |
-
return m ? [m[1], m[2]] : [fmtInt(v), ''];
|
| 279 |
-
}
|
| 280 |
/** The largest true number of a category: task-ready units, raw rows or sources. */
|
| 281 |
function leadOf(st) {
|
| 282 |
const c = [
|
| 283 |
-
{ v: st.ready, k: 'ready', word:
|
| 284 |
-
{ v: st.raw, k: 'raw', word: 'raw
|
| 285 |
-
{ v: st.sources, k: 'sources', word:
|
| 286 |
].filter((x) => x.v > 0);
|
| 287 |
return c.sort((x, y) => y.v - x.v)[0];
|
| 288 |
}
|
|
@@ -297,35 +280,35 @@ function catCard({ code, st }) {
|
|
| 297 |
el('div', { class: 'cat-main' },
|
| 298 |
el('span', { class: 'cat-name', title: name }, name),
|
| 299 |
el('span', { class: 'wanted-art', 'aria-hidden': 'true' }, domainIcon(code, { size: 44 })),
|
| 300 |
-
el('span', { class: 'cat-wanted' }, icon('target', { size: 16 }), '
|
| 301 |
-
el('div', { class: 'cat-foot' }, el('a', { class: 'btn', href: requestUrl(
|
| 302 |
}
|
| 303 |
const cur = F.leaf === code;
|
| 304 |
const pick = () => setFilter(cur ? { domain: '', leaf: '' } : { domain: topOf(code), leaf: code });
|
| 305 |
const lead = leadOf(st);
|
| 306 |
-
const [n, scale] =
|
| 307 |
const num = el('b', { class: 'cat-num' });
|
| 308 |
setNumber(num, `cat:${code}`, lead.v, () => n);
|
| 309 |
const meta = [];
|
| 310 |
-
if (lead.k !== 'sources') meta.push(el('span', {}, plural(
|
| 311 |
-
if (lead.k !== 'raw' && st.raw) meta.push(el('span', { class: 'muted' },
|
| 312 |
-
const readyText = st.ready ?
|
| 313 |
return el('li', { class: `card cat lift${cur ? ' current' : ''}` },
|
| 314 |
el('button', { type: 'button', class: 'cat-main', 'aria-pressed': String(cur), onclick: pick },
|
| 315 |
el('span', { class: 'cat-head' }, el('span', { class: 'cat-ico' }, domainIcon(code, { size: 22 })),
|
| 316 |
cur ? el('span', { class: 'cat-check' }, icon('check', { size: 18 })) : null),
|
| 317 |
el('span', { class: 'cat-name', title: name }, name),
|
| 318 |
-
el('span', { class: 'cat-lead' }, num, el('span', { class: 'cat-num-label' }, scale ?
|
| 319 |
el('span', { class: 'cat-meta' }, meta),
|
| 320 |
el('span', { class: 'cat-bars' },
|
| 321 |
-
smallBar('
|
| 322 |
-
|
| 323 |
-
smallBar(st.onReady ? '
|
| 324 |
-
st.onReady ? '
|
| 325 |
-
el('div', { class: 'cat-foot' }, copyButton('
|
| 326 |
}
|
| 327 |
function renderCategories() {
|
| 328 |
-
const cats = MOST_REQUESTED.map((
|
| 329 |
const have = cats.filter((c) => c.st.sources).sort(bySize);
|
| 330 |
const wanted = cats.filter((c) => !c.st.sources);
|
| 331 |
clear($('req-cards')).append(...have.map(catCard), ...wanted.map(catCard));
|
|
@@ -338,15 +321,15 @@ function renderCategories() {
|
|
| 338 |
const STACK_NAMED = 3;
|
| 339 |
// parts ranked by size, largest darkest (one ordinal ramp of the slate blue); not classified yet is gray stripes
|
| 340 |
const STEPS = ['var(--ours)', '#6f8797', '#8ea2ae', '#a8b6c0'];
|
| 341 |
-
function capParts(entries,
|
| 342 |
const named = entries.filter((e) => e.key !== NONE && e.value > 0).sort((x, y) => y.value - x.value);
|
| 343 |
const none = entries.find((e) => e.key === NONE && e.value > 0);
|
| 344 |
const head = named.length > STACK_NAMED + 1 ? named.slice(0, STACK_NAMED) : named;
|
| 345 |
const rest = named.slice(head.length);
|
| 346 |
const parts = head.map((e, i) => ({ ...e, color: STEPS[i] }));
|
| 347 |
-
if (rest.length) parts.push({ key: OTHER, label:
|
| 348 |
-
if (none) parts.push({ ...none, label: '
|
| 349 |
-
return parts.map((p) => ({ ...p, sub:
|
| 350 |
}
|
| 351 |
function groupReady(T, keyOf) {
|
| 352 |
const m = new Map();
|
|
@@ -373,11 +356,12 @@ function renderWhere() {
|
|
| 373 |
tipLabel: [s.title, formName(formKeyOf(s))], onClick: () => openSource(s.id) }));
|
| 374 |
if (rest.length) {
|
| 375 |
const v = sum(rest, (s) => s.readyUnits);
|
| 376 |
-
|
|
|
|
| 377 |
}
|
| 378 |
-
figureWithToggle($('where-src'), 'where-src', '
|
| 379 |
-
chart: () => (items.length ? hbars(items) : el('p', { class: 'small muted' }, '
|
| 380 |
-
table: () => table(['
|
| 381 |
ranked.map((s) => [s.title, formName(formKeyOf(s)), fmtInt(s.readyUnits)])),
|
| 382 |
});
|
| 383 |
}
|
|
@@ -393,46 +377,50 @@ function renderReady() {
|
|
| 393 |
const inside = ready - answered;
|
| 394 |
const raw = sum(R, (s) => s.rawRows);
|
| 395 |
const parts = [
|
| 396 |
-
{ value: T.length, color: 'var(--ours)', label: '
|
| 397 |
-
{ value: R.length, color: 'var(--other-light)', label: '
|
| 398 |
];
|
| 399 |
const inner = [
|
| 400 |
-
{ value: answered, color: 'var(--ours)', label: '
|
| 401 |
-
{ value: inside, color: 'var(--fill-3)', label: '
|
| 402 |
];
|
| 403 |
const forms = groupReady(T, formKeyOf);
|
| 404 |
const doms = groupReady(T, oneDomain);
|
| 405 |
-
const formParts = capParts(forms.map((e) => ({ ...e, label: formName(e.key) })), '
|
| 406 |
-
const domParts = capParts(doms.map((e) => ({ ...e, label: domainName(e.key) })), '
|
| 407 |
const bar = (ps, twoRows = false) => el('div', { class: 'split' },
|
| 408 |
el('div', { class: 'stack', role: 'img', 'aria-label': ps.map((p) => `${p.label} ${p.sub}`).join(', ') },
|
| 409 |
ps.filter((p) => p.value > 0).map((p) => withTip(el('span', { tabindex: '0', style: { flex: `${p.value} 1 0`, background: p.color } }), p.sub, p.label))),
|
| 410 |
el('div', { class: `split-labels fixed${twoRows ? ' two-rows' : ''}` }, ps.map((p) => el('span', { title: `${p.label}: ${p.sub}` },
|
| 411 |
el('span', { class: 'sw', style: { background: p.color } }), el('b', {}, p.label), el('span', { class: 'small' }, p.sub)))));
|
| 412 |
const only = el('button', { type: 'button', class: 'btn small-btn', 'aria-pressed': String(F.status === 'ready'),
|
| 413 |
-
onclick: () => setFilter({ status: F.status === 'ready' ? '' : 'ready' }) }, icon('pick', { size: 16 }), F.status === 'ready' ? '
|
| 414 |
const mini = (ico, text, first) => el('p', { class: `mini-title${first ? ' first' : ''}` }, icon(ico, { size: 16 }), text);
|
| 415 |
const rows = (list, kind) => [...list].sort((x, y) => y.value - x.value).map((e) => [kind, e.label ?? '', fmtInt(e.sources), fmtInt(e.value)]);
|
| 416 |
-
figureWithToggle($('ready-card'), 'ready', '
|
| 417 |
-
chart: () => el('div', {}, mini('sources', '
|
| 418 |
-
mini('stack', '
|
| 419 |
-
table: () => table(['
|
| 420 |
-
['
|
| 421 |
-
['
|
| 422 |
-
['
|
| 423 |
-
['
|
| 424 |
-
...rows(forms.map((e) => ({ ...e, label: formName(e.key) })), '
|
| 425 |
-
...rows(doms.map((e) => ({ ...e, label: domainName(e.key) })), '
|
| 426 |
]),
|
| 427 |
}, () => el('div', { class: 'card-foot' }, only));
|
| 428 |
}
|
| 429 |
-
// ---- filter row
|
| 430 |
function option(value, text) { return el('option', { value }, text); }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 431 |
function fillFilterRow() {
|
| 432 |
-
|
| 433 |
-
clear($('f-form')).append(option('', 'All'), formsByCount().map((k) => option(k, formName(k))));
|
| 434 |
-
const tiers = [...new Set(IX.sources.map((s) => s.tier))].sort((a, b) => (TIER[a] || a).localeCompare(TIER[b] || b));
|
| 435 |
-
clear($('f-tier')).append(option('', 'All'), tiers.map((t) => option(t, TIER[t] || human(t))));
|
| 436 |
const on = (id, key) => $(id).addEventListener('change', (e) => {
|
| 437 |
const patch = { [key]: e.target.value };
|
| 438 |
if (key === 'domain') patch.leaf = '';
|
|
@@ -440,8 +428,8 @@ function fillFilterRow() {
|
|
| 440 |
});
|
| 441 |
on('f-domain', 'domain'); on('f-leaf', 'leaf'); on('f-form', 'form'); on('f-train', 'train'); on('f-redist', 'redist');
|
| 442 |
on('f-tier', 'tier'); on('f-size', 'size'); on('f-status', 'status');
|
| 443 |
-
let
|
| 444 |
-
$('f-q').addEventListener('input', (e) => { clearTimeout(
|
| 445 |
$('f-clear').addEventListener('click', () => setFilter(Object.fromEntries(FILTER_KEYS.map((k) => [k, '']))));
|
| 446 |
$('filter-toggle').addEventListener('click', () => {
|
| 447 |
const open = !$('filters').classList.contains('open');
|
|
@@ -456,13 +444,13 @@ function syncFilterRow(sel) {
|
|
| 456 |
if (document.activeElement !== $('f-q')) $('f-q').value = F.q;
|
| 457 |
const leaf = $('f-leaf');
|
| 458 |
const leaves = F.domain && F.domain !== NONE && F.domain !== MULTI ? leafKeys(F.domain) : [];
|
| 459 |
-
clear(leaf).append(option('', '
|
| 460 |
leaf.disabled = !leaves.length;
|
| 461 |
leaf.value = F.leaf;
|
| 462 |
const n = activeFilters().length;
|
| 463 |
$('f-clear').disabled = !n;
|
| 464 |
-
$('filter-state').textContent = n ? `${n
|
| 465 |
-
$('f-count').textContent = plural(
|
| 466 |
}
|
| 467 |
|
| 468 |
// ---- axes of the map
|
|
@@ -525,28 +513,28 @@ function renderMap() {
|
|
| 525 |
// gray row and column of the map, never the headline)
|
| 526 |
const total = sum(base, (s) => metricOf(s, m));
|
| 527 |
const covered = new Set(base.flatMap((s) => s.tops)).size;
|
| 528 |
-
const amount = m === 'sources' ? plural(
|
| 529 |
-
$('map-h').textContent = !base.length ? '
|
| 530 |
for (const b of $('metric-seg').querySelectorAll('button')) b.setAttribute('aria-pressed', String(b.dataset.metric === m));
|
| 531 |
|
| 532 |
const current = F.domain && F.form ? { row: F.domain, col: F.form } : null;
|
| 533 |
const rowObjs = rows.map((k) => ({ key: k, label: domainName(k), muted: k === NONE }));
|
| 534 |
const colObjs = cols.map((k) => ({ key: k, label: formName(k), head: formIcon(k, { size: 20 }), muted: k === NONE }));
|
| 535 |
-
figureWithToggle($('heat-fig'), 'heat',
|
| 536 |
chart: () => {
|
| 537 |
const h = heatmap({
|
| 538 |
rows: rowObjs, cols: colObjs, value: (r, c) => cell.get(`${r.key}|${c.key}`) || 0, fmt, current,
|
| 539 |
-
metricName:
|
| 540 |
onPick: (r, c) => {
|
| 541 |
const form = c.key === OTHER ? '' : c.key;
|
| 542 |
if (F.domain === r.key && F.form === form) setFilter({ domain: '', leaf: '', form: '' });
|
| 543 |
else setFilter({ domain: r.key, leaf: '', form });
|
| 544 |
},
|
| 545 |
});
|
| 546 |
-
const key = el('ul', { class: 'icon-key', 'aria-label': '
|
| 547 |
return el('div', { class: 'heat-body' }, h.grid, key, h.legend);
|
| 548 |
},
|
| 549 |
-
table: () => table(['
|
| 550 |
rowObjs.flatMap((r) => colObjs.map((c) => [r, c, cell.get(`${r.key}|${c.key}`) || 0])).filter((x) => x[2] > 0)
|
| 551 |
.map(([r, c, v]) => [r.label, c.label, fmt(v)])),
|
| 552 |
});
|
|
@@ -573,9 +561,9 @@ function renderMap() {
|
|
| 573 |
}
|
| 574 |
}
|
| 575 |
const domMax = Math.max(0, ...rows.map((r) => domVal.get(r)));
|
| 576 |
-
figureWithToggle($('dom-fig'), 'dom', '
|
| 577 |
chart: () => hbars(items, { max: domMax }),
|
| 578 |
-
table: () => table(['
|
| 579 |
});
|
| 580 |
|
| 581 |
const formBase = IX.sources.filter((s) => passes(s, { form: true }));
|
|
@@ -585,22 +573,22 @@ function renderMap() {
|
|
| 585 |
const fShown = UI.allForms ? fAll : fAll.slice(0, FORM_BARS);
|
| 586 |
const fItems = fShown.map((c) => ({ key: c, label: formName(c), icon: formIcon(c, { size: 16 }), value: fv.get(c) || 0, text: fmt(fv.get(c) || 0),
|
| 587 |
current: F.form === c, muted: c === NONE, tipLabel: formName(c), onClick: () => setFilter({ form: F.form === c ? '' : c }) }));
|
| 588 |
-
figureWithToggle($('form-fig'), 'form', '
|
| 589 |
chart: () => hbars(fItems),
|
| 590 |
-
table: () => table(['
|
| 591 |
}, () => (fAll.length > FORM_BARS ? el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.allForms),
|
| 592 |
-
onclick: () => { UI.allForms = !UI.allForms; renderMap(); } }, UI.allForms ? '
|
| 593 |
}
|
| 594 |
|
| 595 |
-
// ---- 4. sources
|
| 596 |
const COLS = [
|
| 597 |
-
{ key: 'title'
|
| 598 |
-
{ key: 'domain'
|
| 599 |
-
{ key: 'form'
|
| 600 |
-
{ key: 'units',
|
| 601 |
-
{ key: 'distinct',
|
| 602 |
-
{ key: 'tier'
|
| 603 |
-
{ key: 'fit'
|
| 604 |
];
|
| 605 |
function fitScore(s) { const f = fitOf(s); return sum(FIT, (x) => ({ yes: 2, part: 1, unknown: 0.5, no: 0 })[f[x.k]]); }
|
| 606 |
function sortValue(s, k) {
|
|
@@ -611,28 +599,24 @@ function sortValue(s, k) {
|
|
| 611 |
// task-ready units above raw rows (never compared as one count): a raw corpus has fewer than 1e12 rows
|
| 612 |
case 'units': return s.isTask ? 1e12 + s.readyUnits : s.rawRows;
|
| 613 |
case 'distinct': return s.distinct ?? -1;
|
| 614 |
-
case 'tier': return (
|
| 615 |
case 'fit': return fitScore(s);
|
| 616 |
default: return 0;
|
| 617 |
}
|
| 618 |
}
|
| 619 |
-
const distinctText = (s) => (s.distinct == null ? '
|
| 620 |
-
// #364: the same words as the index README, section Size (artpool/projection_templates/text.json, size_text)
|
| 621 |
-
|
| 622 |
-
+ '(the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the '
|
| 623 |
-
+ 'first source by registry order), counted at every release, per source in the catalog column distinct_bytes; archive '
|
| 624 |
-
+ 'members whose archive the same source also lists whole have other hashes, so they are not subtracted. tools/fetch.py '
|
| 625 |
-
+ 'removes duplicates over your own selection when it reads it.';
|
| 626 |
/** Distinct content and rows without a size of the sources S: the rule of index_projection.size_totals. */
|
| 627 |
function sizeTotals(S) {
|
| 628 |
const known = S.filter((s) => s.distinct != null);
|
| 629 |
return { distinct: known.length ? sum(known, (s) => s.distinct) : null, unsized: sum(S, (s) => s.rowsNoSize) };
|
| 630 |
}
|
| 631 |
function sizeNote(S) {
|
| 632 |
-
const
|
| 633 |
-
if (
|
| 634 |
-
const
|
| 635 |
-
return
|
| 636 |
}
|
| 637 |
|
| 638 |
function renderSources(sel) {
|
|
@@ -643,7 +627,7 @@ function renderSources(sel) {
|
|
| 643 |
return (x < y ? -1 : x > y ? 1 : 0) * dir || a.title.localeCompare(b.title);
|
| 644 |
});
|
| 645 |
const nReady = sel.filter((s) => s.isTask).length;
|
| 646 |
-
$('sources-h').textContent = sel.length ?
|
| 647 |
const body = clear($('sources-body'));
|
| 648 |
if (!sel.length) return;
|
| 649 |
const shown = UI.showAll ? list : list.slice(0, FIRST_ROWS);
|
|
@@ -655,7 +639,7 @@ function renderSources(sel) {
|
|
| 655 |
el('button', { type: 'button', onclick: () => {
|
| 656 |
const d = on ? (F.dir === 'asc' ? 'desc' : 'asc') : (c.r || c.key === 'fit' ? 'desc' : 'asc');
|
| 657 |
F.sort = c.key; F.dir = d; writeUrl(); renderSources(selected());
|
| 658 |
-
} }, c.
|
| 659 |
}));
|
| 660 |
const tbody = el('tbody');
|
| 661 |
for (const s of shown) {
|
|
@@ -665,23 +649,23 @@ function renderSources(sel) {
|
|
| 665 |
el('td', { class: 'title' },
|
| 666 |
el('button', { type: 'button', 'aria-expanded': String(open), 'aria-controls': `d-${s.id}`, onclick: toggle },
|
| 667 |
el('span', { class: 'caret', 'aria-hidden': 'true' }, open ? '▾' : '▸'), s.title),
|
| 668 |
-
s.isTask ? el('span', { class: 'ready-tag' }, icon('check', { size: 14 }), '
|
| 669 |
-
el('td', { 'data-label': '
|
| 670 |
-
el('td', { 'data-label': '
|
| 671 |
-
el('td', { class: 'r num', 'data-label': s.isTask ? '
|
| 672 |
-
? [fmtInt(s.readyUnits), el('span', { class: 'unit-word' },
|
| 673 |
-
: el('span', { class: 'muted' }, fmtCompact(s.rawRows), el('span', { class: 'unit-word' }, '
|
| 674 |
-
el('td', { class: 'r num', 'data-label': '
|
| 675 |
-
el('td', { 'data-label': '
|
| 676 |
el('td', { class: 'fit-cell' }, fitStrip(s))));
|
| 677 |
if (open) tbody.append(el('tr', { class: 'detail-row' }, el('td', { colspan: String(COLS.length), id: `d-${s.id}` }, detail(s))));
|
| 678 |
}
|
| 679 |
-
body.append(el('div', { class: 'list-key' }, el('span', { class: 'small' }, '
|
| 680 |
-
body.append(el('table', { class: 'src-table' }, el('caption', { class: 'visually-hidden' }, '
|
| 681 |
body.append(el('p', { class: 'small size-note' }, sizeNote(sel)));
|
| 682 |
if (list.length > FIRST_ROWS) {
|
| 683 |
body.append(el('p', { class: 'more' }, el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.showAll),
|
| 684 |
-
onclick: () => { UI.showAll = !UI.showAll; renderSources(selected()); } }, UI.showAll ?
|
| 685 |
}
|
| 686 |
}
|
| 687 |
|
|
@@ -707,12 +691,12 @@ function sizeStrip(q) {
|
|
| 707 |
const lo = Math.log10(Math.min(1e3, v('p10')));
|
| 708 |
const hi = Math.log10(Math.max(v('p99'), v('p90') * 10, MAX_FILE * 2));
|
| 709 |
const at = (x) => `${(100 * (Math.log10(x) - lo)) / (hi - lo)}%`;
|
| 710 |
-
return el('div', { class: 'strip', role: 'img', 'aria-label':
|
| 711 |
el('span', { class: 'strip-axis' }),
|
| 712 |
el('span', { class: 'strip-band', style: { left: at(v('p10')), width: `calc(${at(v('p90'))} - ${at(v('p10'))})` } }),
|
| 713 |
el('span', { class: 'strip-limit', style: { left: at(MAX_FILE) } }),
|
| 714 |
el('span', { class: 'strip-dot', style: { left: at(v('p50')) } }),
|
| 715 |
-
el('span', { class: 'strip-lab', style: { left: at(v('p50')) } },
|
| 716 |
el('span', { class: 'strip-lab limit', style: { left: at(MAX_FILE) } }, '20 MB'));
|
| 717 |
}
|
| 718 |
function item(iconName, label, ...value) {
|
|
@@ -743,48 +727,55 @@ function detail(s) {
|
|
| 743 |
|
| 744 |
// fit at a glance
|
| 745 |
box.append(el('ul', { class: 'fit-row' }, FIT.map((x) => el('li', { class: `fit-item fit-${fit[x.k]}-item` },
|
| 746 |
-
icon(x.icon, { size: 18 }), el('span', {}, x.
|
| 747 |
|
| 748 |
const dl = el('dl', { class: 'd-list' });
|
| 749 |
// 1 what it is
|
| 750 |
const text = s.summary || s.description;
|
| 751 |
-
dl.append(item('doc', '
|
| 752 |
// 2 what one unit is
|
|
|
|
|
|
|
|
|
|
|
|
|
| 753 |
const unitWhat = s.isTask
|
| 754 |
-
? [el('span', {}, s.answered ? '
|
| 755 |
-
el('span', { class: 'chip' }, plural(s.roles.unit
|
| 756 |
-
s.roles.answer ? el('span', { class: 'chip' }, plural(s.roles.answer
|
| 757 |
-
: [el('span', {}, sentence(s.kinds.map((k) => UNIT_KIND[k]
|
| 758 |
-
|
|
|
|
| 759 |
// 3 where it comes from
|
| 760 |
const origins = [...new Set(metas.map((m) => m.pool_source?.origin).filter(Boolean))];
|
| 761 |
const hosts = [...new Set(rows.map((r) => hostOf(r.uri)).filter(Boolean))];
|
| 762 |
const from = origins.length ? origins : hosts;
|
| 763 |
-
dl.append(item('pin', '
|
| 764 |
// 4 self-contained or companions
|
| 765 |
const comps = [...new Set(rows.filter((r, i) => metas[i].role === 'companion').map((r) => r.uri.split('/').pop()))];
|
| 766 |
-
dl.append(item('layers', '
|
| 767 |
-
? (s.roles.companion ? [el('span', {}, '
|
| 768 |
-
: fit.alone === 'yes' ? '
|
| 769 |
// 5 where the answers are
|
| 770 |
-
dl.append(item('check', '
|
| 771 |
-
: s.isTask ?
|
| 772 |
// 6 tools to open it
|
| 773 |
const exts = Object.entries(info.by_ext || {}).map(([k, v]) => [k, num(v.rows) || 0]).sort((a, b) => b[1] - a[1]).map(([k]) => k);
|
| 774 |
const tools = [];
|
| 775 |
-
for (const e of exts) {
|
| 776 |
-
dl.append(item('tool', '
|
| 777 |
// 7 licence
|
| 778 |
const lr = info.license_rows || {};
|
| 779 |
-
const permit = (field) => PERMIT_ORDER.map((k) => ({ k, value: num(lr[field]?.[k]?.rows) || 0, color: PERMIT_COLOR[k], label:
|
| 780 |
-
dl.append(item('training', '
|
| 781 |
el('span', { class: 'lic-bars' },
|
| 782 |
-
el('span', { class: 'mini-title' }, '
|
| 783 |
-
el('span', { class: 'mini-title' }, '
|
| 784 |
// 8 size and units
|
| 785 |
const q = info.size_quantiles_upper_bytes || {};
|
| 786 |
-
|
| 787 |
-
|
|
|
|
|
|
|
| 788 |
q.p50 ? sizeStrip(q) : null));
|
| 789 |
// 9 one example file
|
| 790 |
let ex = null;
|
|
@@ -796,21 +787,22 @@ function detail(s) {
|
|
| 796 |
ex = el('span', { class: 'example' }, href ? el('a', { href, rel: 'noopener', class: 'mono' }, name) : el('span', { class: 'mono' }, name),
|
| 797 |
r.size_bytes != null ? el('span', { class: 'small' }, ` · ${fmtBytes(num(r.size_bytes))}`) : null);
|
| 798 |
}
|
| 799 |
-
dl.append(item('link', '
|
| 800 |
// 10 the command that takes this source
|
| 801 |
const cmd = command(commandArgs(null, { sources: [s.id] }));
|
| 802 |
-
dl.append(item('run', '
|
| 803 |
box.append(dl);
|
| 804 |
|
| 805 |
// batches and the rows read, on request
|
| 806 |
const b = st.batches;
|
| 807 |
-
const more = el('details', { class: 'd-more' }, el('summary', {}, '
|
| 808 |
if (Array.isArray(b)) {
|
| 809 |
-
more.append(table(['
|
| 810 |
-
b.map((x) => [x.batch, x.rows == null ? '
|
|
|
|
| 811 |
}
|
| 812 |
if (rows.length) {
|
| 813 |
-
more.append(el('div', { class: 'table-wrap sample' }, table(['
|
| 814 |
rows.map((r) => [r.member_path ? `${r.uri} ! ${r.member_path}` : r.range_offset != null ? `${r.uri} @ ${num(r.range_offset)}+${num(r.range_length)}` : r.uri,
|
| 815 |
r.size_bytes == null ? '' : fmtInt(num(r.size_bytes)), hex(r.sha256).slice(0, 16) + (r.sha256 ? '…' : '')]), st.file)));
|
| 816 |
}
|
|
@@ -844,8 +836,8 @@ function selectionSql(sel) {
|
|
| 844 |
const paths = all ? [`${root}/data/*/*/*.parquet`] : sel.map((s) => `${root}/data/${s.id}/*/*.parquet`);
|
| 845 |
const chips = filterChips();
|
| 846 |
return [
|
| 847 |
-
`--
|
| 848 |
-
|
| 849 |
`SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
|
| 850 |
'FROM read_parquet([',
|
| 851 |
paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
|
|
@@ -884,50 +876,51 @@ function folderTree(sel) {
|
|
| 884 |
const src = first ? first.id : '<source>';
|
| 885 |
const line = (depth, iconName, name, note) => el('li', { style: { paddingLeft: `${depth * 20}px` } }, icon(iconName, { size: 16 }),
|
| 886 |
el('span', { class: 'tree-name' }, el('span', { class: 'mono' }, name), note ? el('span', { class: 'small' }, note) : null));
|
| 887 |
-
return el('figure', { class: 'card tree-fig' }, el('p', { class: 'chart-title' }, '
|
| 888 |
el('ul', { class: 'tree' },
|
| 889 |
line(0, 'folder', 'data/'),
|
| 890 |
-
line(1, 'folder', `<domain>/${src}/`, 'units
|
| 891 |
-
line(1, 'folder', `answer/${src}/`, 'answers
|
| 892 |
line(1, 'folder', '_meta/'),
|
| 893 |
-
line(2, 'rows', 'manifest.jsonl', '
|
| 894 |
-
line(2, 'rows', 'sizes.tsv', '
|
| 895 |
-
el('p', { class: 'small tree-note' }, el('span', { class: 'mono' }, '<domain>'), '
|
| 896 |
-
el('p', { class: 'chart-title profile-title' }, '
|
| 897 |
-
el('ul', { class: 'chips profile' }, ['
|
| 898 |
-
.map((
|
| 899 |
el('p', { class: 'small agent-links' }, icon('robot', { size: 16 }),
|
| 900 |
-
el('a', { href: `${DATASET_URL}#for-agents` }, '
|
| 901 |
}
|
| 902 |
function renderGet(sel) {
|
| 903 |
const ready = sum(sel, (s) => s.readyUnits);
|
| 904 |
const raw = sum(sel, (s) => s.rawRows);
|
| 905 |
-
$('get-h').textContent = !sel.length ? '
|
|
|
|
| 906 |
const body = clear($('get-body'));
|
| 907 |
if (!sel.length) return;
|
| 908 |
const chips = filterChips();
|
| 909 |
-
const tabs = segmented('', [['cmd', '
|
| 910 |
const pickRow = el('div', { class: 'pick-row' },
|
| 911 |
-
el('span', { class: 'fact-line' }, icon('sources', { size: 18 }), el('b', {}, fmtInt(sel.length)),
|
| 912 |
-
el('span', { class: 'fact-line' }, icon('check', { size: 18 }), el('b', {}, fmtInt(ready)),
|
| 913 |
-
raw ? el('span', { class: 'fact-line muted' }, icon('rows', { size: 18 }), el('b', {}, fmtCompact(raw)), 'raw
|
| 914 |
-
el('span', { class: 'chips' }, chips.length ? chips.map(([k, v]) => el('span', { class: 'chip', title: k }, v)) : el('span', { class: 'chip' }, '
|
| 915 |
-
el('a', { class: 'small', href: '#filters' }, '
|
| 916 |
body.append(pickRow);
|
| 917 |
const codeCard = (iconName, title, code, n) => el('div', { class: 'code-card' },
|
| 918 |
el('div', { class: 'code-head' }, el('span', { class: 'code-ico' }, icon(iconName, { size: 20 })),
|
| 919 |
n ? el('span', { class: 'step-n' }, String(n)) : null, el('span', { class: 'code-title' }, title),
|
| 920 |
-
copyButton('
|
| 921 |
el('pre', { class: 'code' }, el('code', {}, code)));
|
| 922 |
if (UI.tab === 'cmd') {
|
| 923 |
const sa = selectionArgs(sel);
|
| 924 |
const dry = command(sa.args, true);
|
| 925 |
const take = command(sa.args, false);
|
| 926 |
body.append(el('div', { class: 'get-grid' },
|
| 927 |
-
el('div', { class: 'cmd-steps' }, codeCard('search', '
|
| 928 |
folderTree(sel)));
|
| 929 |
} else if (UI.tab === 'sql') {
|
| 930 |
-
body.append(codeCard('database', '
|
| 931 |
} else {
|
| 932 |
const rows = sum(sel, (s) => s.rows);
|
| 933 |
const key = sel.map((s) => s.id).join(',');
|
|
@@ -935,18 +928,18 @@ function renderGet(sel) {
|
|
| 935 |
const tooMany = rows > CSV_MAX_ROWS;
|
| 936 |
const btn = el('button', { type: 'button', class: 'btn solid', disabled: tooMany, onclick: async () => {
|
| 937 |
btn.disabled = true;
|
| 938 |
-
const say = (
|
| 939 |
try {
|
| 940 |
-
const csv = await buildCsv(sel, (d, n) => say(
|
| 941 |
const blob = new Blob([csv], { type: 'text/csv' });
|
| 942 |
const a = el('a', { href: URL.createObjectURL(blob), download: `agentic-env-index-${IX.sha.slice(0, 7)}-${sel.length === 1 ? sel[0].id : `${sel.length}-sources`}.csv` });
|
| 943 |
document.body.append(a); a.click(); a.remove();
|
| 944 |
setTimeout(() => URL.revokeObjectURL(a.href), 10000);
|
| 945 |
-
say(
|
| 946 |
} catch (e) { say(e.message); }
|
| 947 |
btn.disabled = false;
|
| 948 |
-
} }, icon('csv', { size: 16 }),
|
| 949 |
-
body.append(el('div', { class: 'card csv-card copy-row' }, icon('csv', { size: 28 }), btn, tooMany ? el('span', { class: 'small' },
|
| 950 |
}
|
| 951 |
}
|
| 952 |
|
|
@@ -956,19 +949,19 @@ function renderExcluded() {
|
|
| 956 |
const by = new Map();
|
| 957 |
for (const r of rows) by.set(r.category, (by.get(r.category) || 0) + 1);
|
| 958 |
const ranked = [...by].sort((a, b) => b[1] - a[1]);
|
| 959 |
-
const name =
|
| 960 |
-
$('excluded-h').textContent = rows.length ?
|
| 961 |
const fig = clear($('excluded-fig'));
|
| 962 |
if (!rows.length) return;
|
| 963 |
const items = ranked.map(([k, v]) => ({ key: k, label: name(k), value: v, text: fmtInt(v), tipLabel: IX.requirements.get(k) || name(k) }));
|
| 964 |
-
figureWithToggle(fig, 'excluded', '
|
| 965 |
chart: () => hbars(items),
|
| 966 |
-
table: () => table(['
|
| 967 |
}, () => {
|
| 968 |
const list = [...rows].sort((a, b) => (a.category || '').localeCompare(b.category || '') || (a.source || '').localeCompare(b.source || ''));
|
| 969 |
-
return el('details', { class: 'long' }, el('summary', {},
|
| 970 |
-
el('div', { class: 'table-wrap' }, table(['
|
| 971 |
-
list.map((r) => [r.source, r.level === 'source' ? '
|
| 972 |
});
|
| 973 |
}
|
| 974 |
function snapshots() {
|
|
@@ -985,10 +978,11 @@ function snapshots() {
|
|
| 985 |
}
|
| 986 |
function renderHistory() {
|
| 987 |
const snaps = snapshots();
|
| 988 |
-
figureWithToggle($('history-fig'), 'history', '
|
| 989 |
-
chart: () => columns(snaps.map((x) => ({ key: x.date, label: x.date
|
| 990 |
-
|
| 991 |
-
|
|
|
|
| 992 |
e.kind === 'update' ? fmtInt(e.sourcesAdded) : e.kind === 'start' ? fmtInt(e.sources) : '',
|
| 993 |
e.kind === 'update' ? fmtInt(e.rowsAdded) : e.kind === 'start' ? fmtInt(e.rows) : ''])),
|
| 994 |
});
|
|
@@ -1007,16 +1001,17 @@ function renderHeader() {
|
|
| 1007 |
});
|
| 1008 |
if (IX.overrides.catalog || IX.overrides.vocabulary) {
|
| 1009 |
const chip = $('preview-chip');
|
| 1010 |
-
clear(chip).append(el('span', { class: 'dot other', 'aria-hidden': 'true' }), '
|
| 1011 |
chip.title = [IX.overrides.catalog, IX.overrides.vocabulary].filter(Boolean).join(' ');
|
| 1012 |
chip.hidden = false;
|
| 1013 |
}
|
| 1014 |
}
|
|
|
|
| 1015 |
function updateRequestLinks() {
|
| 1016 |
-
const chips = filterChips();
|
| 1017 |
const u = new URL(`${GITHUB}/issues/new`);
|
| 1018 |
u.searchParams.set('template', 'data_request.yml');
|
| 1019 |
-
if (chips.length) u.searchParams.set('what',
|
| 1020 |
for (const a of document.querySelectorAll('.request-link')) a.href = u.toString();
|
| 1021 |
}
|
| 1022 |
function progress() {
|
|
@@ -1057,8 +1052,36 @@ function renderScoped() {
|
|
| 1057 |
updateRequestLinks();
|
| 1058 |
}
|
| 1059 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1060 |
async function main() {
|
| 1061 |
readUrl();
|
|
|
|
|
|
|
|
|
|
| 1062 |
decorate();
|
| 1063 |
reveal();
|
| 1064 |
let ticking = false;
|
|
@@ -1068,11 +1091,10 @@ async function main() {
|
|
| 1068 |
addEventListener('resize', onScroll);
|
| 1069 |
progress();
|
| 1070 |
try {
|
| 1071 |
-
IX = await loadIndex(new URLSearchParams(location.search), (
|
| 1072 |
} catch (e) {
|
| 1073 |
-
|
| 1074 |
-
|
| 1075 |
-
n.hidden = false;
|
| 1076 |
console.error(e);
|
| 1077 |
return;
|
| 1078 |
}
|
|
|
|
| 2 |
// at a glance, take it with one command. Graphics first; every chart keeps a short label and a table view.
|
| 3 |
import { DATASET_URL, fileUrl, loadBatchDates, loadIndex, num, parquetRows, readColumns, REPO, topOf } from './data.js';
|
| 4 |
import {
|
| 5 |
+
clear, columns, compactParts, el, figureWithToggle, fmtBytes, fmtCompact, fmtDate, fmtInt, fmtOr, fmtPct, fmtShort, heatmap,
|
| 6 |
+
hbars, hideTip, meter, segmented, stack, table, vt, withTip,
|
| 7 |
} from './charts.js';
|
| 8 |
import { decorate, drawHero, markFromIndex, setNumber } from './first.js';
|
| 9 |
+
import { onLanguage, setUpSwitch, t, tEn, translatePage } from './i18n.js';
|
| 10 |
import { domainIcon, formIcon, icon } from './icons.js';
|
| 11 |
|
| 12 |
+
// The categories the group asks for most, shown first. Their names, and those of the top-level domains, are fallbacks
|
| 13 |
+
// in the locale files (domain.<code>) for when the vocabulary has none.
|
| 14 |
+
const MOST_REQUESTED = ['D10.2', 'D4.1', 'D4.5', 'D3.1', 'D3.6', 'D2.5', 'D2.6', 'D2.7', 'D2.8', 'D8.2'];
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
const GITHUB = 'https://github.com/msra-sg/agentic-env-pipeline';
|
| 16 |
const CSV_MAX_ROWS = 200000;
|
| 17 |
const ROW_COLUMNS = ['source', 'batch', 'uri', 'range_offset', 'range_length', 'member_path', 'size_bytes', 'sha256'];
|
|
|
|
| 23 |
const MAX_LISTED = 30;
|
| 24 |
const $ = (id) => document.getElementById(id);
|
| 25 |
|
| 26 |
+
// ---- plain names (codes only inside the SQL and the commands); the words are in the locale files, by code:
|
| 27 |
+
// tier.<tier>, permit.<answer>, reason.<category>, unitKind.<kind> and unitCount.<kind>
|
| 28 |
+
const tierName = (k, tt = t) => tt(`tier.${k}`, { defaultValue: human(k) });
|
|
|
|
|
|
|
|
|
|
| 29 |
const PERMIT_ORDER = ['yes', 'no', 'conditional', 'unknown'];
|
| 30 |
const PERMIT_COLOR = { yes: 'var(--ours)', no: 'var(--problem)', conditional: 'var(--fill-3)', unknown: 'var(--other)' };
|
| 31 |
+
const reasonName = (k) => t(`reason.${k}`, { defaultValue: human(k) });
|
| 32 |
+
// the kinds of unit of a raw corpus (sources.json unit) -> their key in the locale files
|
| 33 |
+
const UNIT_KIND = { file: 'file', 'API response': 'apiResponse', 'WARC record': 'warcRecord', 'archive member': 'archiveMember', 'byte range': 'byteRange' };
|
| 34 |
+
// file type -> the tool that opens it (TOOL_WORDS: not a tool's name, a key of the locale files)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
const TOOLS = {
|
| 36 |
xlsx: 'openpyxl', xlsm: 'openpyxl', xls: 'LibreOffice', ods: 'LibreOffice', csv: 'DuckDB', tsv: 'DuckDB', parquet: 'DuckDB',
|
| 37 |
json: 'jq', jsonl: 'jq', ndjson: 'jq', pdf: 'pdftotext', doc: 'LibreOffice', docx: 'python-docx', ppt: 'LibreOffice',
|
| 38 |
pptx: 'python-pptx', gz: 'gzip', tgz: 'tar', tar: 'tar', bz2: 'bzip2', xz: 'xz', zst: 'zstd', zip: 'unzip', whl: 'unzip',
|
| 39 |
jar: 'unzip', '7z': '7-Zip', crate: 'tar', deb: 'dpkg-deb', rpm: 'rpm2cpio', warc: 'warcio', arc: 'warcio', pgn: 'python-chess',
|
| 40 |
+
py: 'Python', '(none)': 'file(1)', xml: 'xmllint',
|
| 41 |
+
yaml: 'yq', yml: 'yq', evtx: 'python-evtx', log: 'grep', rego: 'OPA', sql: 'DuckDB', db: 'SQLite',
|
| 42 |
safetensors: 'safetensors', final: 'TensorBoard', ipynb: 'Jupyter', eml: 'Python email', mbox: 'Python mailbox',
|
| 43 |
ics: 'icalendar', e01: 'The Sleuth Kit', img: 'The Sleuth Kit', dd: 'The Sleuth Kit', raw: 'The Sleuth Kit',
|
| 44 |
ys: 'Yosys', v: 'Icarus Verilog', sv: 'Verilator', go: 'Go', rs: 'Cargo', zig: 'Zig', lua: 'Lua', rb: 'Ruby', c: 'gcc',
|
| 45 |
+
tcl: 'tclsh', smt2: 'Z3', mzn: 'MiniZinc', dzn: 'MiniZinc', lp: 'HiGHS', mps: 'HiGHS',
|
| 46 |
};
|
| 47 |
+
const TOOL_WORDS = { md: 'tool.editor', txt: 'tool.editor', toml: 'tool.editor', html: 'tool.browser', htm: 'tool.browser', test: 'tool.suiteRunner' };
|
| 48 |
+
const toolOf = (ext) => TOOLS[ext] || (TOOL_WORDS[ext] ? t(TOOL_WORDS[ext]) : null);
|
| 49 |
const NONE = '_none';
|
| 50 |
const MULTI = '_multi';
|
| 51 |
const OTHER = '_other';
|
| 52 |
const human = (code) => String(code).replace(/[_-]+/g, ' ').replace(/^./, (c) => c.toUpperCase());
|
| 53 |
function sentence(s) {
|
| 54 |
+
let x = String(s).trim();
|
| 55 |
+
if (!/\s/.test(x)) x = x.replace(/[-_]+/g, ' ');
|
| 56 |
+
x = x.replace(/\b(ci|ctf|etl|ml|qa|sql|api)\b/gi, (w) => (w.toLowerCase() === 'qa' ? 'Q&A' : w.toUpperCase()));
|
| 57 |
+
return x.charAt(0).toUpperCase() + x.slice(1);
|
| 58 |
}
|
| 59 |
+
/** A count with its noun: key_one and key_other in the locale files, {{n}} the formatted number. */
|
| 60 |
+
const plural = (key, n, tt = t) => tt(key, { count: n, n: fmtInt(n) });
|
| 61 |
const sum = (xs, f) => xs.reduce((a, x) => a + f(x), 0);
|
| 62 |
|
| 63 |
let IX = null;
|
|
|
|
| 87 |
const n = IX.vocab[sec].names.get(code);
|
| 88 |
return n ? sentence(n) : null;
|
| 89 |
};
|
| 90 |
+
function domainName(key, tt = t) {
|
| 91 |
+
if (key === NONE) return tt('name.notClassified');
|
| 92 |
+
if (key === MULTI) return vocabName('domain', 'multi') || tt('name.severalDomains');
|
| 93 |
+
return vocabName('domain', key) || tt(`domain.${key}`, { defaultValue: key });
|
|
|
|
| 94 |
}
|
| 95 |
+
function formName(key, tt = t) {
|
| 96 |
+
if (key === NONE || key == null) return tt('name.notClassified');
|
| 97 |
+
if (key === OTHER) return tt('name.otherForms');
|
| 98 |
return vocabName('form', key) || sentence(key);
|
| 99 |
}
|
| 100 |
const domainKeysOf = (s) => (s.domainKind === 'codes' ? s.tops : [s.domainKind === 'multi' ? MULTI : NONE]);
|
|
|
|
| 108 |
}
|
| 109 |
const inCategory = (s, code) => (code.includes('.') ? s.codes.includes(code) : s.tops.includes(code));
|
| 110 |
|
| 111 |
+
// ---- fit: what decides whether a source can feed a task (yes, part, no, unknown); labels fit.<k>, states fitWord.<state>
|
| 112 |
const FIT = [
|
| 113 |
+
{ k: 'ship', icon: 'share' },
|
| 114 |
+
{ k: 'bench', icon: 'target' },
|
| 115 |
+
{ k: 'pii', icon: 'person' },
|
| 116 |
+
{ k: 'size', icon: 'size' },
|
| 117 |
+
{ k: 'alone', icon: 'doc' },
|
| 118 |
+
{ k: 'answers', icon: 'check' },
|
| 119 |
+
{ k: 'login', icon: 'link' },
|
| 120 |
];
|
| 121 |
+
const fitLabel = (k) => t(`fit.${k}`);
|
| 122 |
+
const fitWord = (state) => t(`fitWord.${state}`);
|
| 123 |
function fitOf(s) {
|
| 124 |
const q = s.info.size_quantiles_upper_bytes || {};
|
| 125 |
const c = s.contamination?.level;
|
|
|
|
| 137 |
};
|
| 138 |
}
|
| 139 |
function fitDot(state, label) {
|
| 140 |
+
return el('span', { class: `fit fit-${state}`, role: 'img', 'aria-label': t('fit.dot', { label, state: fitWord(state) }) });
|
| 141 |
}
|
| 142 |
function fitStrip(s) {
|
| 143 |
const f = fitOf(s);
|
| 144 |
+
return el('span', { class: 'fit-strip' }, FIT.map((x) => withTip(fitDot(f[x.k], fitLabel(x.k)), fitWord(f[x.k]), fitLabel(x.k))));
|
| 145 |
}
|
| 146 |
function fitKey() {
|
| 147 |
+
return el('span', { class: 'fit-key' }, ['yes', 'part', 'no', 'unknown'].map((k) => el('span', {}, el('span', { class: `fit fit-${k}`, 'aria-hidden': 'true' }), fitWord(k))));
|
| 148 |
}
|
| 149 |
|
| 150 |
// ---- filters
|
|
|
|
| 181 |
const selected = () => IX.sources.filter((s) => passes(s));
|
| 182 |
const FILTER_KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q'];
|
| 183 |
const activeFilters = () => FILTER_KEYS.filter((k) => F[k]);
|
| 184 |
+
/** The filters that are on, as [label, value] in the words of tt (t, or tEn for the issue the page prefills). */
|
| 185 |
+
function filterChips(tt = t) {
|
| 186 |
const out = [];
|
| 187 |
+
const word = (key) => tt(key, { defaultValue: '' });
|
| 188 |
+
if (F.domain) out.push([tt('chip.domain'), F.leaf ? domainName(F.leaf, tt) : domainName(F.domain, tt)]);
|
| 189 |
+
if (F.form) out.push([tt('chip.form'), formName(F.form, tt)]);
|
| 190 |
+
if (F.train) out.push([tt('chip.training'), word(`filter.permit.${F.train}`)]);
|
| 191 |
+
if (F.redist) out.push([tt('chip.redistribution'), word(`filter.permit.${F.redist}`)]);
|
| 192 |
+
if (F.tier) out.push([tt('chip.tier'), tierName(F.tier, tt)]);
|
| 193 |
+
if (F.size) out.push([tt('chip.size'), word(`filter.size.${F.size}`)]);
|
| 194 |
+
if (F.status) out.push([tt('chip.kind'), word(`filter.status.${F.status}`)]);
|
| 195 |
+
if (F.q) out.push([tt('chip.search'), F.q]);
|
| 196 |
return out;
|
| 197 |
}
|
| 198 |
function metricOf(s, m = F.metric) { return m === 'sources' ? 1 : m === 'rows' ? s.rows : s.distinct || 0; }
|
| 199 |
const metricFmt = (m = F.metric) => (m === 'bytes' ? fmtBytes : (v) => fmtInt(v));
|
| 200 |
+
const metricWord = (m) => t(`metric.${m}`);
|
| 201 |
|
| 202 |
function setFilter(patch) {
|
| 203 |
Object.assign(F, patch);
|
|
|
|
| 230 |
return { args: commandArgs(sel, { domain, form: F.form }), listed: false };
|
| 231 |
}
|
| 232 |
async function copy(text, state) {
|
| 233 |
+
try { await navigator.clipboard.writeText(text); state.textContent = t('copy.done'); } catch { state.textContent = t('copy.select'); }
|
| 234 |
setTimeout(() => { state.textContent = ''; }, 2400);
|
| 235 |
}
|
| 236 |
function copyButton(label, text, { dark = false, iconName = 'copy' } = {}) {
|
|
|
|
| 254 |
return { sources: S.length, readySources: S.filter((s) => s.isTask).length, ready, raw, onReady, ship, shipOf: onReady ? ready : raw };
|
| 255 |
}
|
| 256 |
const bySize = (a, b) => b.st.ready - a.st.ready || b.st.raw - a.st.raw || b.st.sources - a.st.sources;
|
| 257 |
+
function requestUrl(code) {
|
| 258 |
const req = new URL(`${GITHUB}/issues/new`);
|
| 259 |
req.searchParams.set('template', 'data_request.yml');
|
| 260 |
+
req.searchParams.set('what', tEn('request.more', { name: domainName(code, tEn) }));
|
| 261 |
return req.toString();
|
| 262 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 263 |
/** The largest true number of a category: task-ready units, raw rows or sources. */
|
| 264 |
function leadOf(st) {
|
| 265 |
const c = [
|
| 266 |
+
{ v: st.ready, k: 'ready', word: t('cat.lead.ready', { count: st.ready }) },
|
| 267 |
+
{ v: st.raw, k: 'raw', word: t('cat.lead.raw', { count: st.raw }) },
|
| 268 |
+
{ v: st.sources, k: 'sources', word: t('cat.lead.sources', { count: st.sources }) },
|
| 269 |
].filter((x) => x.v > 0);
|
| 270 |
return c.sort((x, y) => y.v - x.v)[0];
|
| 271 |
}
|
|
|
|
| 280 |
el('div', { class: 'cat-main' },
|
| 281 |
el('span', { class: 'cat-name', title: name }, name),
|
| 282 |
el('span', { class: 'wanted-art', 'aria-hidden': 'true' }, domainIcon(code, { size: 44 })),
|
| 283 |
+
el('span', { class: 'cat-wanted' }, icon('target', { size: 16 }), t('cat.wanted'))),
|
| 284 |
+
el('div', { class: 'cat-foot' }, el('a', { class: 'btn', href: requestUrl(code) }, icon('request', { size: 16 }), t('cat.request'))));
|
| 285 |
}
|
| 286 |
const cur = F.leaf === code;
|
| 287 |
const pick = () => setFilter(cur ? { domain: '', leaf: '' } : { domain: topOf(code), leaf: code });
|
| 288 |
const lead = leadOf(st);
|
| 289 |
+
const [n, scale] = compactParts(lead.v);
|
| 290 |
const num = el('b', { class: 'cat-num' });
|
| 291 |
setNumber(num, `cat:${code}`, lead.v, () => n);
|
| 292 |
const meta = [];
|
| 293 |
+
if (lead.k !== 'sources') meta.push(el('span', {}, plural('n.source', st.sources)));
|
| 294 |
+
if (lead.k !== 'raw' && st.raw) meta.push(el('span', { class: 'muted' }, t('cat.plusRaw', { count: st.raw, n: fmtCompact(st.raw) })));
|
| 295 |
+
const readyText = st.ready ? plural('n.unit', st.ready) : t('common.noneYet');
|
| 296 |
return el('li', { class: `card cat lift${cur ? ' current' : ''}` },
|
| 297 |
el('button', { type: 'button', class: 'cat-main', 'aria-pressed': String(cur), onclick: pick },
|
| 298 |
el('span', { class: 'cat-head' }, el('span', { class: 'cat-ico' }, domainIcon(code, { size: 22 })),
|
| 299 |
cur ? el('span', { class: 'cat-check' }, icon('check', { size: 18 })) : null),
|
| 300 |
el('span', { class: 'cat-name', title: name }, name),
|
| 301 |
+
el('span', { class: 'cat-lead' }, num, el('span', { class: 'cat-num-label' }, scale ? t('cat.scaled', { scale, word: lead.word }) : lead.word)),
|
| 302 |
el('span', { class: 'cat-meta' }, meta),
|
| 303 |
el('span', { class: 'cat-bars' },
|
| 304 |
+
smallBar(t('kind.ready'), st.sources ? st.readySources / st.sources : 0, readyText,
|
| 305 |
+
t('cat.readyTip', { count: st.sources, ready: fmtInt(st.readySources), n: fmtInt(st.sources) })),
|
| 306 |
+
smallBar(st.onReady ? t('cat.shippable') : t('cat.shippableRaw'), st.shipOf ? st.ship / st.shipOf : 0, fmtPct(st.ship, st.shipOf),
|
| 307 |
+
st.onReady ? t('cat.shipTipReady') : t('cat.shipTipRaw')))),
|
| 308 |
+
el('div', { class: 'cat-foot' }, copyButton(t('cat.sample'), command(commandArgs(null, { domain: code })))));
|
| 309 |
}
|
| 310 |
function renderCategories() {
|
| 311 |
+
const cats = MOST_REQUESTED.map((code) => ({ code, st: catStats(code) }));
|
| 312 |
const have = cats.filter((c) => c.st.sources).sort(bySize);
|
| 313 |
const wanted = cats.filter((c) => !c.st.sources);
|
| 314 |
clear($('req-cards')).append(...have.map(catCard), ...wanted.map(catCard));
|
|
|
|
| 321 |
const STACK_NAMED = 3;
|
| 322 |
// parts ranked by size, largest darkest (one ordinal ramp of the slate blue); not classified yet is gray stripes
|
| 323 |
const STEPS = ['var(--ours)', '#6f8797', '#8ea2ae', '#a8b6c0'];
|
| 324 |
+
function capParts(entries, otherKey) {
|
| 325 |
const named = entries.filter((e) => e.key !== NONE && e.value > 0).sort((x, y) => y.value - x.value);
|
| 326 |
const none = entries.find((e) => e.key === NONE && e.value > 0);
|
| 327 |
const head = named.length > STACK_NAMED + 1 ? named.slice(0, STACK_NAMED) : named;
|
| 328 |
const rest = named.slice(head.length);
|
| 329 |
const parts = head.map((e, i) => ({ ...e, color: STEPS[i] }));
|
| 330 |
+
if (rest.length) parts.push({ key: OTHER, label: plural(otherKey, rest.length), value: sum(rest, (e) => e.value), sources: sum(rest, (e) => e.sources), color: STEPS[3] });
|
| 331 |
+
if (none) parts.push({ ...none, label: t('stack.notClassified'), color: 'var(--unclassified)' });
|
| 332 |
+
return parts.map((p) => ({ ...p, sub: plural('n.unit', p.value) }));
|
| 333 |
}
|
| 334 |
function groupReady(T, keyOf) {
|
| 335 |
const m = new Map();
|
|
|
|
| 356 |
tipLabel: [s.title, formName(formKeyOf(s))], onClick: () => openSource(s.id) }));
|
| 357 |
if (rest.length) {
|
| 358 |
const v = sum(rest, (s) => s.readyUnits);
|
| 359 |
+
const other = plural('where.otherSources', rest.length);
|
| 360 |
+
items.push({ key: OTHER, label: other, value: v, text: fmtInt(v), tipLabel: other });
|
| 361 |
}
|
| 362 |
+
figureWithToggle($('where-src'), 'where-src', t('where.title'), {
|
| 363 |
+
chart: () => (items.length ? hbars(items) : el('p', { class: 'small muted' }, t('common.noneYet'))),
|
| 364 |
+
table: () => table([t('th.source'), t('th.form'), { label: t('th.readyUnits'), r: true }],
|
| 365 |
ranked.map((s) => [s.title, formName(formKeyOf(s)), fmtInt(s.readyUnits)])),
|
| 366 |
});
|
| 367 |
}
|
|
|
|
| 377 |
const inside = ready - answered;
|
| 378 |
const raw = sum(R, (s) => s.rawRows);
|
| 379 |
const parts = [
|
| 380 |
+
{ value: T.length, color: 'var(--ours)', label: t('kind.ready'), sub: `${plural('n.source', T.length)} · ${plural('n.unit', ready)}` },
|
| 381 |
+
{ value: R.length, color: 'var(--other-light)', label: t('ready.rawCorpora'), sub: `${plural('n.source', R.length)} · ${t('n.rowsCompact', { count: raw, n: fmtCompact(raw) })}` },
|
| 382 |
];
|
| 383 |
const inner = [
|
| 384 |
+
{ value: answered, color: 'var(--ours)', label: t('ready.answersApart'), sub: plural('n.unit', answered) },
|
| 385 |
+
{ value: inside, color: 'var(--fill-3)', label: t('ready.checksInside'), sub: plural('n.unit', inside) },
|
| 386 |
];
|
| 387 |
const forms = groupReady(T, formKeyOf);
|
| 388 |
const doms = groupReady(T, oneDomain);
|
| 389 |
+
const formParts = capParts(forms.map((e) => ({ ...e, label: formName(e.key) })), 'stack.otherForms');
|
| 390 |
+
const domParts = capParts(doms.map((e) => ({ ...e, label: domainName(e.key) })), 'stack.otherDomains');
|
| 391 |
const bar = (ps, twoRows = false) => el('div', { class: 'split' },
|
| 392 |
el('div', { class: 'stack', role: 'img', 'aria-label': ps.map((p) => `${p.label} ${p.sub}`).join(', ') },
|
| 393 |
ps.filter((p) => p.value > 0).map((p) => withTip(el('span', { tabindex: '0', style: { flex: `${p.value} 1 0`, background: p.color } }), p.sub, p.label))),
|
| 394 |
el('div', { class: `split-labels fixed${twoRows ? ' two-rows' : ''}` }, ps.map((p) => el('span', { title: `${p.label}: ${p.sub}` },
|
| 395 |
el('span', { class: 'sw', style: { background: p.color } }), el('b', {}, p.label), el('span', { class: 'small' }, p.sub)))));
|
| 396 |
const only = el('button', { type: 'button', class: 'btn small-btn', 'aria-pressed': String(F.status === 'ready'),
|
| 397 |
+
onclick: () => setFilter({ status: F.status === 'ready' ? '' : 'ready' }) }, icon('pick', { size: 16 }), F.status === 'ready' ? t('ready.showing') : t('ready.only'));
|
| 398 |
const mini = (ico, text, first) => el('p', { class: `mini-title${first ? ' first' : ''}` }, icon(ico, { size: 16 }), text);
|
| 399 |
const rows = (list, kind) => [...list].sort((x, y) => y.value - x.value).map((e) => [kind, e.label ?? '', fmtInt(e.sources), fmtInt(e.value)]);
|
| 400 |
+
figureWithToggle($('ready-card'), 'ready', t('ready.title'), {
|
| 401 |
+
chart: () => el('div', {}, mini('sources', t('metric.sources'), true), bar(parts), mini('check', t('th.readyUnits'), false), bar(inner),
|
| 402 |
+
mini('stack', t('ready.byForm'), false), bar(formParts, true), mini('browse', t('ready.byDomain'), false), bar(domParts, true)),
|
| 403 |
+
table: () => table([t('ready.split'), t('ready.part'), { label: t('metric.sources'), r: true }, { label: t('th.readyUnits'), r: true }], [
|
| 404 |
+
[t('ready.kind'), t('kind.ready'), fmtInt(T.length), fmtInt(ready)],
|
| 405 |
+
[t('ready.kind'), t('ready.rawRow', { count: raw, n: fmtInt(raw) }), fmtInt(R.length), ''],
|
| 406 |
+
[t('ready.units'), t('ready.answersKept'), fmtInt(T.filter((s) => s.answered).length), fmtInt(answered)],
|
| 407 |
+
[t('ready.units'), t('ready.resultsInside'), fmtInt(T.filter((s) => !s.answered).length), fmtInt(inside)],
|
| 408 |
+
...rows(forms.map((e) => ({ ...e, label: formName(e.key) })), t('th.form')),
|
| 409 |
+
...rows(doms.map((e) => ({ ...e, label: domainName(e.key) })), t('th.domain')),
|
| 410 |
]),
|
| 411 |
}, () => el('div', { class: 'card-foot' }, only));
|
| 412 |
}
|
| 413 |
+
// ---- filter row: the options (again in a new language), and their listeners (once)
|
| 414 |
function option(value, text) { return el('option', { value }, text); }
|
| 415 |
+
function fillFilterOptions() {
|
| 416 |
+
const all = t('filter.all');
|
| 417 |
+
clear($('f-domain')).append(option('', all), domainRows().map((k) => option(k, domainName(k))));
|
| 418 |
+
clear($('f-form')).append(option('', all), formsByCount().map((k) => option(k, formName(k))));
|
| 419 |
+
const tiers = [...new Set(IX.sources.map((s) => s.tier))].sort((a, b) => tierName(a).localeCompare(tierName(b)));
|
| 420 |
+
clear($('f-tier')).append(option('', all), tiers.map((k) => option(k, tierName(k))));
|
| 421 |
+
}
|
| 422 |
function fillFilterRow() {
|
| 423 |
+
fillFilterOptions();
|
|
|
|
|
|
|
|
|
|
| 424 |
const on = (id, key) => $(id).addEventListener('change', (e) => {
|
| 425 |
const patch = { [key]: e.target.value };
|
| 426 |
if (key === 'domain') patch.leaf = '';
|
|
|
|
| 428 |
});
|
| 429 |
on('f-domain', 'domain'); on('f-leaf', 'leaf'); on('f-form', 'form'); on('f-train', 'train'); on('f-redist', 'redist');
|
| 430 |
on('f-tier', 'tier'); on('f-size', 'size'); on('f-status', 'status');
|
| 431 |
+
let timer = null;
|
| 432 |
+
$('f-q').addEventListener('input', (e) => { clearTimeout(timer); timer = setTimeout(() => setFilter({ q: e.target.value.trim() }), 200); });
|
| 433 |
$('f-clear').addEventListener('click', () => setFilter(Object.fromEntries(FILTER_KEYS.map((k) => [k, '']))));
|
| 434 |
$('filter-toggle').addEventListener('click', () => {
|
| 435 |
const open = !$('filters').classList.contains('open');
|
|
|
|
| 444 |
if (document.activeElement !== $('f-q')) $('f-q').value = F.q;
|
| 445 |
const leaf = $('f-leaf');
|
| 446 |
const leaves = F.domain && F.domain !== NONE && F.domain !== MULTI ? leafKeys(F.domain) : [];
|
| 447 |
+
clear(leaf).append(option('', t('filter.all')), leaves.map((k) => option(k, domainName(k))));
|
| 448 |
leaf.disabled = !leaves.length;
|
| 449 |
leaf.value = F.leaf;
|
| 450 |
const n = activeFilters().length;
|
| 451 |
$('f-clear').disabled = !n;
|
| 452 |
+
$('filter-state').textContent = n ? `${t('filter.on', { count: n, n: fmtInt(n) })} · ${plural('n.source', sel.length)}` : t('filter.noneOn');
|
| 453 |
+
$('f-count').textContent = plural('n.source', sel.length);
|
| 454 |
}
|
| 455 |
|
| 456 |
// ---- axes of the map
|
|
|
|
| 513 |
// gray row and column of the map, never the headline)
|
| 514 |
const total = sum(base, (s) => metricOf(s, m));
|
| 515 |
const covered = new Set(base.flatMap((s) => s.tops)).size;
|
| 516 |
+
const amount = m === 'sources' ? plural('n.source', total) : m === 'rows' ? t('map.rows', { count: total, n: fmtCompact(total) }) : fmtBytes(total);
|
| 517 |
+
$('map-h').textContent = !base.length ? t('map.nothing') : covered ? t('map.head', { amount, domains: plural('n.domain', covered) }) : amount;
|
| 518 |
for (const b of $('metric-seg').querySelectorAll('button')) b.setAttribute('aria-pressed', String(b.dataset.metric === m));
|
| 519 |
|
| 520 |
const current = F.domain && F.form ? { row: F.domain, col: F.form } : null;
|
| 521 |
const rowObjs = rows.map((k) => ({ key: k, label: domainName(k), muted: k === NONE }));
|
| 522 |
const colObjs = cols.map((k) => ({ key: k, label: formName(k), head: formIcon(k, { size: 20 }), muted: k === NONE }));
|
| 523 |
+
figureWithToggle($('heat-fig'), 'heat', t('map.title', { metric: metricWord(m) }), {
|
| 524 |
chart: () => {
|
| 525 |
const h = heatmap({
|
| 526 |
rows: rowObjs, cols: colObjs, value: (r, c) => cell.get(`${r.key}|${c.key}`) || 0, fmt, current,
|
| 527 |
+
metricName: metricWord(m),
|
| 528 |
onPick: (r, c) => {
|
| 529 |
const form = c.key === OTHER ? '' : c.key;
|
| 530 |
if (F.domain === r.key && F.form === form) setFilter({ domain: '', leaf: '', form: '' });
|
| 531 |
else setFilter({ domain: r.key, leaf: '', form });
|
| 532 |
},
|
| 533 |
});
|
| 534 |
+
const key = el('ul', { class: 'icon-key', 'aria-label': t('map.columns') }, colObjs.map((c) => el('li', {}, formIcon(c.key, { size: 16 }), c.label)));
|
| 535 |
return el('div', { class: 'heat-body' }, h.grid, key, h.legend);
|
| 536 |
},
|
| 537 |
+
table: () => table([t('th.domain'), t('th.form'), { label: metricWord(m), r: true }],
|
| 538 |
rowObjs.flatMap((r) => colObjs.map((c) => [r, c, cell.get(`${r.key}|${c.key}`) || 0])).filter((x) => x[2] > 0)
|
| 539 |
.map(([r, c, v]) => [r.label, c.label, fmt(v)])),
|
| 540 |
});
|
|
|
|
| 561 |
}
|
| 562 |
}
|
| 563 |
const domMax = Math.max(0, ...rows.map((r) => domVal.get(r)));
|
| 564 |
+
figureWithToggle($('dom-fig'), 'dom', t('ready.byDomain'), {
|
| 565 |
chart: () => hbars(items, { max: domMax }),
|
| 566 |
+
table: () => table([t('th.domain'), { label: metricWord(m), r: true }], items.map((d) => [d.leaf ? ` ${d.label}` : d.label, d.text])),
|
| 567 |
});
|
| 568 |
|
| 569 |
const formBase = IX.sources.filter((s) => passes(s, { form: true }));
|
|
|
|
| 573 |
const fShown = UI.allForms ? fAll : fAll.slice(0, FORM_BARS);
|
| 574 |
const fItems = fShown.map((c) => ({ key: c, label: formName(c), icon: formIcon(c, { size: 16 }), value: fv.get(c) || 0, text: fmt(fv.get(c) || 0),
|
| 575 |
current: F.form === c, muted: c === NONE, tipLabel: formName(c), onClick: () => setFilter({ form: F.form === c ? '' : c }) }));
|
| 576 |
+
figureWithToggle($('form-fig'), 'form', t('ready.byForm'), {
|
| 577 |
chart: () => hbars(fItems),
|
| 578 |
+
table: () => table([t('th.form'), { label: metricWord(m), r: true }], fAll.map((c) => [formName(c), fmt(fv.get(c) || 0)])),
|
| 579 |
}, () => (fAll.length > FORM_BARS ? el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.allForms),
|
| 580 |
+
onclick: () => { UI.allForms = !UI.allForms; renderMap(); } }, UI.allForms ? t('common.fewer') : t('common.all', { n: fmtInt(fAll.length) })) : null));
|
| 581 |
}
|
| 582 |
|
| 583 |
+
// ---- 4. sources; the column labels are col.<key> in the locale files
|
| 584 |
const COLS = [
|
| 585 |
+
{ key: 'title' },
|
| 586 |
+
{ key: 'domain' },
|
| 587 |
+
{ key: 'form' },
|
| 588 |
+
{ key: 'units', r: true },
|
| 589 |
+
{ key: 'distinct', r: true },
|
| 590 |
+
{ key: 'tier' },
|
| 591 |
+
{ key: 'fit' },
|
| 592 |
];
|
| 593 |
function fitScore(s) { const f = fitOf(s); return sum(FIT, (x) => ({ yes: 2, part: 1, unknown: 0.5, no: 0 })[f[x.k]]); }
|
| 594 |
function sortValue(s, k) {
|
|
|
|
| 599 |
// task-ready units above raw rows (never compared as one count): a raw corpus has fewer than 1e12 rows
|
| 600 |
case 'units': return s.isTask ? 1e12 + s.readyUnits : s.rawRows;
|
| 601 |
case 'distinct': return s.distinct ?? -1;
|
| 602 |
+
case 'tier': return tierName(s.tier).toLowerCase();
|
| 603 |
case 'fit': return fitScore(s);
|
| 604 |
default: return 0;
|
| 605 |
}
|
| 606 |
}
|
| 607 |
+
const distinctText = (s) => (s.distinct == null ? t('common.unknown') : fmtBytes(s.distinct));
|
| 608 |
+
// #364: the same words as the index README, section Size (artpool/projection_templates/text.json, size_text), in the
|
| 609 |
+
// locale files as size.method
|
|
|
|
|
|
|
|
|
|
|
|
|
| 610 |
/** Distinct content and rows without a size of the sources S: the rule of index_projection.size_totals. */
|
| 611 |
function sizeTotals(S) {
|
| 612 |
const known = S.filter((s) => s.distinct != null);
|
| 613 |
return { distinct: known.length ? sum(known, (s) => s.distinct) : null, unsized: sum(S, (s) => s.rowsNoSize) };
|
| 614 |
}
|
| 615 |
function sizeNote(S) {
|
| 616 |
+
const s = sizeTotals(S);
|
| 617 |
+
if (s.distinct == null) return t('size.unknown');
|
| 618 |
+
const values = { size: fmtBytes(s.distinct), method: t('size.method') };
|
| 619 |
+
return s.unsized ? t('size.noteUnsized', { ...values, unsized: t('size.unsized', { count: s.unsized, n: fmtInt(s.unsized) }) }) : t('size.note', values);
|
| 620 |
}
|
| 621 |
|
| 622 |
function renderSources(sel) {
|
|
|
|
| 627 |
return (x < y ? -1 : x > y ? 1 : 0) * dir || a.title.localeCompare(b.title);
|
| 628 |
});
|
| 629 |
const nReady = sel.filter((s) => s.isTask).length;
|
| 630 |
+
$('sources-h').textContent = sel.length ? t('sources.head', { sources: plural('n.source', sel.length), ready: fmtInt(nReady) }) : t('sources.none');
|
| 631 |
const body = clear($('sources-body'));
|
| 632 |
if (!sel.length) return;
|
| 633 |
const shown = UI.showAll ? list : list.slice(0, FIRST_ROWS);
|
|
|
|
| 639 |
el('button', { type: 'button', onclick: () => {
|
| 640 |
const d = on ? (F.dir === 'asc' ? 'desc' : 'asc') : (c.r || c.key === 'fit' ? 'desc' : 'asc');
|
| 641 |
F.sort = c.key; F.dir = d; writeUrl(); renderSources(selected());
|
| 642 |
+
} }, t(`col.${c.key}`), on ? (F.dir === 'asc' ? ' ↑' : ' ↓') : ''));
|
| 643 |
}));
|
| 644 |
const tbody = el('tbody');
|
| 645 |
for (const s of shown) {
|
|
|
|
| 649 |
el('td', { class: 'title' },
|
| 650 |
el('button', { type: 'button', 'aria-expanded': String(open), 'aria-controls': `d-${s.id}`, onclick: toggle },
|
| 651 |
el('span', { class: 'caret', 'aria-hidden': 'true' }, open ? '▾' : '▸'), s.title),
|
| 652 |
+
s.isTask ? el('span', { class: 'ready-tag' }, icon('check', { size: 14 }), t('sources.ready')) : null),
|
| 653 |
+
el('td', { 'data-label': t('col.domain'), class: 'cat-cell' }, sourceDomainText(s)),
|
| 654 |
+
el('td', { 'data-label': t('col.form'), class: 'form-cell' }, formIcon(formKeyOf(s), { size: 16 }), formName(formKeyOf(s))),
|
| 655 |
+
el('td', { class: 'r num', 'data-label': s.isTask ? t('sources.unitsLabel') : t('sources.rowsLabel') }, s.isTask
|
| 656 |
+
? [fmtInt(s.readyUnits), el('span', { class: 'unit-word' }, t('sources.unitWord', { count: s.readyUnits }))]
|
| 657 |
+
: el('span', { class: 'muted' }, fmtCompact(s.rawRows), el('span', { class: 'unit-word' }, t('sources.rawWord', { count: s.rawRows })))),
|
| 658 |
+
el('td', { class: 'r num', 'data-label': t('col.distinct') }, distinctText(s)),
|
| 659 |
+
el('td', { 'data-label': t('col.tier') }, tierName(s.tier)),
|
| 660 |
el('td', { class: 'fit-cell' }, fitStrip(s))));
|
| 661 |
if (open) tbody.append(el('tr', { class: 'detail-row' }, el('td', { colspan: String(COLS.length), id: `d-${s.id}` }, detail(s))));
|
| 662 |
}
|
| 663 |
+
body.append(el('div', { class: 'list-key' }, el('span', { class: 'small' }, t('sources.fitKey')), FIT.map((x) => el('span', { class: 'fit-name' }, icon(x.icon, { size: 14 }), fitLabel(x.k))), fitKey()));
|
| 664 |
+
body.append(el('table', { class: 'src-table' }, el('caption', { class: 'visually-hidden' }, t('metric.sources')), el('thead', {}, head), tbody));
|
| 665 |
body.append(el('p', { class: 'small size-note' }, sizeNote(sel)));
|
| 666 |
if (list.length > FIRST_ROWS) {
|
| 667 |
body.append(el('p', { class: 'more' }, el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.showAll),
|
| 668 |
+
onclick: () => { UI.showAll = !UI.showAll; renderSources(selected()); } }, UI.showAll ? t('common.first', { n: fmtInt(FIRST_ROWS) }) : t('common.all', { n: fmtInt(list.length) }))));
|
| 669 |
}
|
| 670 |
}
|
| 671 |
|
|
|
|
| 691 |
const lo = Math.log10(Math.min(1e3, v('p10')));
|
| 692 |
const hi = Math.log10(Math.max(v('p99'), v('p90') * 10, MAX_FILE * 2));
|
| 693 |
const at = (x) => `${(100 * (Math.log10(x) - lo)) / (hi - lo)}%`;
|
| 694 |
+
return el('div', { class: 'strip', role: 'img', 'aria-label': t('detail.stripAria', { median: fmtBytes(v('p50')), p99: fmtBytes(v('p99')) }) },
|
| 695 |
el('span', { class: 'strip-axis' }),
|
| 696 |
el('span', { class: 'strip-band', style: { left: at(v('p10')), width: `calc(${at(v('p90'))} - ${at(v('p10'))})` } }),
|
| 697 |
el('span', { class: 'strip-limit', style: { left: at(MAX_FILE) } }),
|
| 698 |
el('span', { class: 'strip-dot', style: { left: at(v('p50')) } }),
|
| 699 |
+
el('span', { class: 'strip-lab', style: { left: at(v('p50')) } }, t('detail.median', { size: fmtBytes(v('p50')) })),
|
| 700 |
el('span', { class: 'strip-lab limit', style: { left: at(MAX_FILE) } }, '20 MB'));
|
| 701 |
}
|
| 702 |
function item(iconName, label, ...value) {
|
|
|
|
| 727 |
|
| 728 |
// fit at a glance
|
| 729 |
box.append(el('ul', { class: 'fit-row' }, FIT.map((x) => el('li', { class: `fit-item fit-${fit[x.k]}-item` },
|
| 730 |
+
icon(x.icon, { size: 18 }), el('span', {}, fitLabel(x.k)), fitDot(fit[x.k], fitLabel(x.k))))));
|
| 731 |
|
| 732 |
const dl = el('dl', { class: 'd-list' });
|
| 733 |
// 1 what it is
|
| 734 |
const text = s.summary || s.description;
|
| 735 |
+
dl.append(item('doc', t('detail.what'), text ? (text.length > 160 ? el('details', {}, el('summary', {}, firstSentence(text)), el('p', {}, text)) : text) : t('detail.notDescribed')));
|
| 736 |
// 2 what one unit is
|
| 737 |
+
const kindCount = (k) => {
|
| 738 |
+
const n = num(s.info.unit[k]);
|
| 739 |
+
return t(UNIT_KIND[k] ? `unitCount.${UNIT_KIND[k]}` : 'unitCount.code', { count: n, n: fmtCompact(n), kind: k });
|
| 740 |
+
};
|
| 741 |
const unitWhat = s.isTask
|
| 742 |
+
? [el('span', {}, s.answered ? t('detail.taskUnit') : t('detail.unitWithChecks')), el('span', { class: 'chips' },
|
| 743 |
+
el('span', { class: 'chip' }, plural('n.unitFile', s.roles.unit)), s.roles.companion ? el('span', { class: 'chip' }, plural('n.companion', s.roles.companion)) : null,
|
| 744 |
+
s.roles.answer ? el('span', { class: 'chip' }, plural('n.answer', s.roles.answer)) : null)]
|
| 745 |
+
: [el('span', {}, sentence(fmtOr(s.kinds.map((k) => (UNIT_KIND[k] ? t(`unitKind.${UNIT_KIND[k]}`) : k))))),
|
| 746 |
+
el('span', { class: 'chips' }, s.kinds.map((k) => el('span', { class: 'chip' }, kindCount(k))))];
|
| 747 |
+
dl.append(item('rows', t('detail.oneUnit'), ...unitWhat));
|
| 748 |
// 3 where it comes from
|
| 749 |
const origins = [...new Set(metas.map((m) => m.pool_source?.origin).filter(Boolean))];
|
| 750 |
const hosts = [...new Set(rows.map((r) => hostOf(r.uri)).filter(Boolean))];
|
| 751 |
const from = origins.length ? origins : hosts;
|
| 752 |
+
dl.append(item('pin', t('detail.from'), st.sample === 'loading' ? '…' : from.length ? el('span', { class: 'chips' }, from.slice(0, 3).map((h) => el('span', { class: 'chip mono-chip' }, h))) : t('detail.notRecorded')));
|
| 753 |
// 4 self-contained or companions
|
| 754 |
const comps = [...new Set(rows.filter((r, i) => metas[i].role === 'companion').map((r) => r.uri.split('/').pop()))];
|
| 755 |
+
dl.append(item('layers', t('detail.alone'), s.isTask
|
| 756 |
+
? (s.roles.companion ? [el('span', {}, t('detail.withCompanions')), el('span', { class: 'chips' }, (comps.length ? comps : ['LICENSE', 'README.md']).map((c) => el('span', { class: 'chip mono-chip' }, c)))] : t('detail.yes'))
|
| 757 |
+
: fit.alone === 'yes' ? t('detail.eachFile') : t('detail.container')));
|
| 758 |
// 5 where the answers are
|
| 759 |
+
dl.append(item('check', t('detail.answers'), s.answered ? el('span', { class: 'mono-chip chip' }, `answer/${s.id}/ · ${plural('n.file', s.answered)}`)
|
| 760 |
+
: s.isTask ? t('detail.answersInside') : el('span', { class: 'muted' }, t('detail.answersNone'))));
|
| 761 |
// 6 tools to open it
|
| 762 |
const exts = Object.entries(info.by_ext || {}).map(([k, v]) => [k, num(v.rows) || 0]).sort((a, b) => b[1] - a[1]).map(([k]) => k);
|
| 763 |
const tools = [];
|
| 764 |
+
for (const e of exts) { const tool = toolOf(e); if (tool && !tools.some((x) => x[1] === tool)) tools.push([e, tool]); if (tools.length >= 3) break; }
|
| 765 |
+
dl.append(item('tool', t('detail.opensWith'), tools.length ? el('span', { class: 'chips' }, tools.map(([e, tool]) => el('span', { class: 'chip' }, el('span', { class: 'mono' }, e === '(none)' ? t('detail.noExt') : `.${e}`), ` ${tool}`))) : t('detail.anyEditor')));
|
| 766 |
// 7 licence
|
| 767 |
const lr = info.license_rows || {};
|
| 768 |
+
const permit = (field) => PERMIT_ORDER.map((k) => ({ k, value: num(lr[field]?.[k]?.rows) || 0, color: PERMIT_COLOR[k], label: t(`permit.${k}`) }));
|
| 769 |
+
dl.append(item('training', t('detail.licence'), el('span', { class: 'lic' }, el('b', {}, tierName(s.tier)), el('span', { class: 'small' }, lic.name || s.spdx || '')),
|
| 770 |
el('span', { class: 'lic-bars' },
|
| 771 |
+
el('span', { class: 'mini-title' }, t('detail.training')), stack(permit('training_ok'), s.rows),
|
| 772 |
+
el('span', { class: 'mini-title' }, t('detail.shipping')), stack(permit('redistribution_ok'), s.rows))));
|
| 773 |
// 8 size and units
|
| 774 |
const q = info.size_quantiles_upper_bytes || {};
|
| 775 |
+
const amounts = s.isTask ? `${plural('n.unit', s.readyUnits)} · ${plural('n.file', s.rows)}` : plural('n.rawRows', s.rawRows);
|
| 776 |
+
const unsized = s.rowsNoSize ? ` · ${t('detail.unsized', { n: fmtCompact(s.rowsNoSize) })}` : '';
|
| 777 |
+
dl.append(item('size', t('detail.size'), el('span', { class: 'lic' }, el('b', {}, t('detail.distinct', { size: distinctText(s) })), el('span', { class: 'small' },
|
| 778 |
+
`${amounts}${unsized}`)),
|
| 779 |
q.p50 ? sizeStrip(q) : null));
|
| 780 |
// 9 one example file
|
| 781 |
let ex = null;
|
|
|
|
| 787 |
ex = el('span', { class: 'example' }, href ? el('a', { href, rel: 'noopener', class: 'mono' }, name) : el('span', { class: 'mono' }, name),
|
| 788 |
r.size_bytes != null ? el('span', { class: 'small' }, ` · ${fmtBytes(num(r.size_bytes))}`) : null);
|
| 789 |
}
|
| 790 |
+
dl.append(item('link', t('detail.example'), st.sample === 'loading' ? '…' : st.sample?.error ? el('span', { class: 'problem-text' }, st.sample.error) : ex || t('detail.noFile')));
|
| 791 |
// 10 the command that takes this source
|
| 792 |
const cmd = command(commandArgs(null, { sources: [s.id] }));
|
| 793 |
+
dl.append(item('run', t('detail.take'), el('code', { class: 'cmd-inline' }, cmd), copyButton(t('copy.button'), cmd)));
|
| 794 |
box.append(dl);
|
| 795 |
|
| 796 |
// batches and the rows read, on request
|
| 797 |
const b = st.batches;
|
| 798 |
+
const more = el('details', { class: 'd-more' }, el('summary', {}, t('detail.more')));
|
| 799 |
if (Array.isArray(b)) {
|
| 800 |
+
more.append(table([t('th.batch'), { label: t('th.rows'), r: true }, t('th.changed'), t('th.state')],
|
| 801 |
+
b.map((x) => [x.batch, x.rows == null ? t('detail.notCounted') : fmtInt(x.rows), fmtDate(x.date),
|
| 802 |
+
x.status ? t(`batch.${x.status}`, { defaultValue: human(x.status) }) : ''])));
|
| 803 |
}
|
| 804 |
if (rows.length) {
|
| 805 |
+
more.append(el('div', { class: 'table-wrap sample' }, table([t('th.address'), { label: t('th.size'), r: true }, 'sha256'],
|
| 806 |
rows.map((r) => [r.member_path ? `${r.uri} ! ${r.member_path}` : r.range_offset != null ? `${r.uri} @ ${num(r.range_offset)}+${num(r.range_length)}` : r.uri,
|
| 807 |
r.size_bytes == null ? '' : fmtInt(num(r.size_bytes)), hex(r.sha256).slice(0, 16) + (r.sha256 ? '…' : '')]), st.file)));
|
| 808 |
}
|
|
|
|
| 836 |
const paths = all ? [`${root}/data/*/*/*.parquet`] : sel.map((s) => `${root}/data/${s.id}/*/*.parquet`);
|
| 837 |
const chips = filterChips();
|
| 838 |
return [
|
| 839 |
+
`-- ${t('sql.head', { sha: IX.sha.slice(0, 7), sources: plural('n.source', sel.length), rows: plural('n.row', sum(sel, (s) => s.rows)) })}`,
|
| 840 |
+
`-- ${chips.length ? chips.map(([k, v]) => t('sql.chip', { k, v })).join('; ') : t('sql.all')}`,
|
| 841 |
`SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
|
| 842 |
'FROM read_parquet([',
|
| 843 |
paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
|
|
|
|
| 876 |
const src = first ? first.id : '<source>';
|
| 877 |
const line = (depth, iconName, name, note) => el('li', { style: { paddingLeft: `${depth * 20}px` } }, icon(iconName, { size: 16 }),
|
| 878 |
el('span', { class: 'tree-name' }, el('span', { class: 'mono' }, name), note ? el('span', { class: 'small' }, note) : null));
|
| 879 |
+
return el('figure', { class: 'card tree-fig' }, el('p', { class: 'chart-title' }, t('tree.title')),
|
| 880 |
el('ul', { class: 'tree' },
|
| 881 |
line(0, 'folder', 'data/'),
|
| 882 |
+
line(1, 'folder', `<domain>/${src}/`, t('tree.units')),
|
| 883 |
+
line(1, 'folder', `answer/${src}/`, t('tree.answers')),
|
| 884 |
line(1, 'folder', '_meta/'),
|
| 885 |
+
line(2, 'rows', 'manifest.jsonl', t('tree.manifest')),
|
| 886 |
+
line(2, 'rows', 'sizes.tsv', t('tree.sizes'))),
|
| 887 |
+
el('p', { class: 'small tree-note' }, el('span', { class: 'mono' }, '<domain>'), t('tree.domainNote')),
|
| 888 |
+
el('p', { class: 'chart-title profile-title' }, t('tree.leftOut')),
|
| 889 |
+
el('ul', { class: 'chips profile' }, ['duplicates', 'benchmark', 'licence', 'english', 'personal', 'large']
|
| 890 |
+
.map((k) => el('li', { class: 'chip' }, t(`profile.${k}`)))),
|
| 891 |
el('p', { class: 'small agent-links' }, icon('robot', { size: 16 }),
|
| 892 |
+
el('a', { href: `${DATASET_URL}#for-agents` }, t('agents.readme')), ' · ', el('a', { href: 'llms.txt' }, 'llms.txt')));
|
| 893 |
}
|
| 894 |
function renderGet(sel) {
|
| 895 |
const ready = sum(sel, (s) => s.readyUnits);
|
| 896 |
const raw = sum(sel, (s) => s.rawRows);
|
| 897 |
+
$('get-h').textContent = !sel.length ? t('get.nothing') : sel.length === 1 ? t('get.takeOne', { count: SAMPLE, title: sel[0].title })
|
| 898 |
+
: t('get.takeEach', { count: SAMPLE, sources: plural('n.source', sel.length) });
|
| 899 |
const body = clear($('get-body'));
|
| 900 |
if (!sel.length) return;
|
| 901 |
const chips = filterChips();
|
| 902 |
+
const tabs = segmented('', [['cmd', t('get.command')], ['sql', 'SQL'], ['csv', 'CSV']], UI.tab, (v) => { UI.tab = v; vt(() => renderGet(selected())); });
|
| 903 |
const pickRow = el('div', { class: 'pick-row' },
|
| 904 |
+
el('span', { class: 'fact-line' }, icon('sources', { size: 18 }), el('b', {}, fmtInt(sel.length)), t('get.sourceWord', { count: sel.length })),
|
| 905 |
+
el('span', { class: 'fact-line' }, icon('check', { size: 18 }), el('b', {}, fmtInt(ready)), t('cat.lead.ready', { count: ready })),
|
| 906 |
+
raw ? el('span', { class: 'fact-line muted' }, icon('rows', { size: 18 }), el('b', {}, fmtCompact(raw)), t('cat.lead.raw', { count: raw })) : null,
|
| 907 |
+
el('span', { class: 'chips' }, chips.length ? chips.map(([k, v]) => el('span', { class: 'chip', title: k }, v)) : el('span', { class: 'chip' }, t('get.wholeIndex'))),
|
| 908 |
+
el('a', { class: 'small', href: '#filters' }, t('get.change')), tabs);
|
| 909 |
body.append(pickRow);
|
| 910 |
const codeCard = (iconName, title, code, n) => el('div', { class: 'code-card' },
|
| 911 |
el('div', { class: 'code-head' }, el('span', { class: 'code-ico' }, icon(iconName, { size: 20 })),
|
| 912 |
n ? el('span', { class: 'step-n' }, String(n)) : null, el('span', { class: 'code-title' }, title),
|
| 913 |
+
copyButton(t('copy.button'), code, { dark: true })),
|
| 914 |
el('pre', { class: 'code' }, el('code', {}, code)));
|
| 915 |
if (UI.tab === 'cmd') {
|
| 916 |
const sa = selectionArgs(sel);
|
| 917 |
const dry = command(sa.args, true);
|
| 918 |
const take = command(sa.args, false);
|
| 919 |
body.append(el('div', { class: 'get-grid' },
|
| 920 |
+
el('div', { class: 'cmd-steps' }, codeCard('search', t('get.look'), dry, 1), codeCard('run', t('get.take'), take, 2)),
|
| 921 |
folderTree(sel)));
|
| 922 |
} else if (UI.tab === 'sql') {
|
| 923 |
+
body.append(codeCard('database', t('get.sql'), selectionSql(sel)));
|
| 924 |
} else {
|
| 925 |
const rows = sum(sel, (s) => s.rows);
|
| 926 |
const key = sel.map((s) => s.id).join(',');
|
|
|
|
| 928 |
const tooMany = rows > CSV_MAX_ROWS;
|
| 929 |
const btn = el('button', { type: 'button', class: 'btn solid', disabled: tooMany, onclick: async () => {
|
| 930 |
btn.disabled = true;
|
| 931 |
+
const say = (text) => { csvState.textContent = text; UI.csv = { key, text }; };
|
| 932 |
try {
|
| 933 |
+
const csv = await buildCsv(sel, (d, n) => say(t('csv.progress', { count: n, done: fmtInt(d), n: fmtInt(n) })));
|
| 934 |
const blob = new Blob([csv], { type: 'text/csv' });
|
| 935 |
const a = el('a', { href: URL.createObjectURL(blob), download: `agentic-env-index-${IX.sha.slice(0, 7)}-${sel.length === 1 ? sel[0].id : `${sel.length}-sources`}.csv` });
|
| 936 |
document.body.append(a); a.click(); a.remove();
|
| 937 |
setTimeout(() => URL.revokeObjectURL(a.href), 10000);
|
| 938 |
+
say(plural('n.row', csv.split('\n').length - 2));
|
| 939 |
} catch (e) { say(e.message); }
|
| 940 |
btn.disabled = false;
|
| 941 |
+
} }, icon('csv', { size: 16 }), t('csv.button', { rows: plural('n.row', rows) }));
|
| 942 |
+
body.append(el('div', { class: 'card csv-card copy-row' }, icon('csv', { size: 28 }), btn, tooMany ? el('span', { class: 'small' }, t('csv.tooMany', { n: fmtShort(CSV_MAX_ROWS) })) : csvState));
|
| 943 |
}
|
| 944 |
}
|
| 945 |
|
|
|
|
| 949 |
const by = new Map();
|
| 950 |
for (const r of rows) by.set(r.category, (by.get(r.category) || 0) + 1);
|
| 951 |
const ranked = [...by].sort((a, b) => b[1] - a[1]);
|
| 952 |
+
const name = reasonName;
|
| 953 |
+
$('excluded-h').textContent = rows.length ? t('excluded.head', { count: rows.length, n: fmtInt(rows.length) }) : t('excluded.none');
|
| 954 |
const fig = clear($('excluded-fig'));
|
| 955 |
if (!rows.length) return;
|
| 956 |
const items = ranked.map(([k, v]) => ({ key: k, label: name(k), value: v, text: fmtInt(v), tipLabel: IX.requirements.get(k) || name(k) }));
|
| 957 |
+
figureWithToggle(fig, 'excluded', t('excluded.reason'), {
|
| 958 |
chart: () => hbars(items),
|
| 959 |
+
table: () => table([t('excluded.reason'), t('excluded.requirement'), { label: t('excluded.items'), r: true }], ranked.map(([k, v]) => [name(k), IX.requirements.get(k) || '', fmtInt(v)])),
|
| 960 |
}, () => {
|
| 961 |
const list = [...rows].sort((a, b) => (a.category || '').localeCompare(b.category || '') || (a.source || '').localeCompare(b.source || ''));
|
| 962 |
+
return el('details', { class: 'long' }, el('summary', {}, t('common.all', { n: fmtInt(rows.length) })),
|
| 963 |
+
el('div', { class: 'table-wrap' }, table([t('th.source'), t('excluded.unit'), t('excluded.notMet'), t('excluded.detail'), t('excluded.reviewed')],
|
| 964 |
+
list.map((r) => [r.source, r.level === 'source' ? t('excluded.wholeSource') : r.unit, IX.requirements.get(r.category) || name(r.category), r.detail || '', r.reviewed || '']))));
|
| 965 |
});
|
| 966 |
}
|
| 967 |
function snapshots() {
|
|
|
|
| 978 |
}
|
| 979 |
function renderHistory() {
|
| 980 |
const snaps = snapshots();
|
| 981 |
+
figureWithToggle($('history-fig'), 'history', t('history.title'), {
|
| 982 |
+
chart: () => columns(snaps.map((x) => ({ key: x.date, label: fmtDate(x.date, { month: 'numeric', day: 'numeric' }), value: x.sources || 0, top: fmtInt(x.sources),
|
| 983 |
+
sub: x.start ? t('history.start') : `+${fmtInt(x.added)}` }))),
|
| 984 |
+
table: () => table([t('history.date'), t('history.entry'), { label: t('history.sourcesAdded'), r: true }, { label: t('history.rowsAdded'), r: true }],
|
| 985 |
+
[...IX.changelog].reverse().map((e) => [fmtDate(e.date), e.kind === 'start' ? t('history.started') : sentence(e.title),
|
| 986 |
e.kind === 'update' ? fmtInt(e.sourcesAdded) : e.kind === 'start' ? fmtInt(e.sources) : '',
|
| 987 |
e.kind === 'update' ? fmtInt(e.rowsAdded) : e.kind === 'start' ? fmtInt(e.rows) : ''])),
|
| 988 |
});
|
|
|
|
| 1001 |
});
|
| 1002 |
if (IX.overrides.catalog || IX.overrides.vocabulary) {
|
| 1003 |
const chip = $('preview-chip');
|
| 1004 |
+
clear(chip).append(el('span', { class: 'dot other', 'aria-hidden': 'true' }), t('hero.preview'));
|
| 1005 |
chip.title = [IX.overrides.catalog, IX.overrides.vocabulary].filter(Boolean).join(' ');
|
| 1006 |
chip.hidden = false;
|
| 1007 |
}
|
| 1008 |
}
|
| 1009 |
+
// the issue a request link opens is written in English, the language of the hub
|
| 1010 |
function updateRequestLinks() {
|
| 1011 |
+
const chips = filterChips(tEn);
|
| 1012 |
const u = new URL(`${GITHUB}/issues/new`);
|
| 1013 |
u.searchParams.set('template', 'data_request.yml');
|
| 1014 |
+
if (chips.length) u.searchParams.set('what', tEn('request.like', { chips: chips.map(([k, v]) => `${k}: ${v}`).join('; ') }));
|
| 1015 |
for (const a of document.querySelectorAll('.request-link')) a.href = u.toString();
|
| 1016 |
}
|
| 1017 |
function progress() {
|
|
|
|
| 1052 |
updateRequestLinks();
|
| 1053 |
}
|
| 1054 |
|
| 1055 |
+
// the note above the page (a part of the index not read, or the index not read at all), kept so a new language
|
| 1056 |
+
// writes it again: {key, values, problem}
|
| 1057 |
+
let notice = null;
|
| 1058 |
+
function drawNotice() {
|
| 1059 |
+
if (!notice) return;
|
| 1060 |
+
const n = $('notice');
|
| 1061 |
+
clear(n).append(notice.problem ? el('span', { class: 'dot problem', 'aria-hidden': 'true' }) : null, t(notice.key, notice.values));
|
| 1062 |
+
n.hidden = false;
|
| 1063 |
+
}
|
| 1064 |
+
|
| 1065 |
+
/** The page in the language just chosen: the words of index.html, then every part drawn from the index. */
|
| 1066 |
+
function renderLanguage() {
|
| 1067 |
+
translatePage();
|
| 1068 |
+
drawNotice();
|
| 1069 |
+
if (!IX) return;
|
| 1070 |
+
hideTip();
|
| 1071 |
+
renderHeader();
|
| 1072 |
+
renderCategories();
|
| 1073 |
+
renderReady();
|
| 1074 |
+
fillFilterOptions();
|
| 1075 |
+
renderScoped();
|
| 1076 |
+
renderExcluded();
|
| 1077 |
+
renderHistory();
|
| 1078 |
+
}
|
| 1079 |
+
|
| 1080 |
async function main() {
|
| 1081 |
readUrl();
|
| 1082 |
+
translatePage();
|
| 1083 |
+
setUpSwitch();
|
| 1084 |
+
onLanguage(renderLanguage);
|
| 1085 |
decorate();
|
| 1086 |
reveal();
|
| 1087 |
let ticking = false;
|
|
|
|
| 1091 |
addEventListener('resize', onScroll);
|
| 1092 |
progress();
|
| 1093 |
try {
|
| 1094 |
+
IX = await loadIndex(new URLSearchParams(location.search), (key, values) => { notice = { key, values }; drawNotice(); });
|
| 1095 |
} catch (e) {
|
| 1096 |
+
notice = { key: 'notice.failed', values: { error: e.message }, problem: true };
|
| 1097 |
+
drawNotice();
|
|
|
|
| 1098 |
console.error(e);
|
| 1099 |
return;
|
| 1100 |
}
|
charts.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
| 1 |
// Small chart kit: elements, number formats, tooltip, chart/table toggle, thin bars, heatmap.
|
| 2 |
// Every string from the data goes in with textContent (through el), never innerHTML.
|
|
|
|
| 3 |
|
| 4 |
export function el(tag, attrs = {}, ...kids) {
|
| 5 |
const n = document.createElement(tag);
|
|
@@ -20,41 +21,52 @@ export function el(tag, attrs = {}, ...kids) {
|
|
| 20 |
|
| 21 |
export const clear = (n) => { while (n.firstChild) n.firstChild.remove(); return n; };
|
| 22 |
|
| 23 |
-
// ---- numbers
|
| 24 |
-
const
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
export function fmtCompact(v) {
|
| 27 |
-
if (v == null) return 'unknown';
|
| 28 |
-
|
| 29 |
-
const three = (x) => (x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2));
|
| 30 |
-
if (a >= 1e9) return `${three(v / 1e9)} billion`;
|
| 31 |
-
if (a >= 1e6) return `${three(v / 1e6)} million`;
|
| 32 |
-
if (a >= 1e4) return `${three(v / 1e3)} thousand`;
|
| 33 |
-
return intFmt.format(v);
|
| 34 |
}
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
return String(Math.round(v));
|
| 43 |
}
|
|
|
|
| 44 |
export function fmtBytes(v) {
|
| 45 |
-
if (v == null) return 'unknown';
|
| 46 |
const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB'];
|
| 47 |
let i = 0;
|
| 48 |
let x = v;
|
| 49 |
while (x >= 1000 && i < units.length - 1) { x /= 1000; i++; }
|
| 50 |
-
const s = i === 0 ?
|
| 51 |
return `${s} ${units[i]}`;
|
| 52 |
}
|
| 53 |
export const fmtPct = (part, whole) => {
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
| 57 |
};
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
// ---- tooltip: the value leads, the label follows; also shown on keyboard focus
|
| 60 |
const tip = () => document.getElementById('tip');
|
|
@@ -106,7 +118,7 @@ export function figureWithToggle(fig, key, title, views, extra) {
|
|
| 106 |
clear(fig).append(
|
| 107 |
el('div', { class: 'chart-head' },
|
| 108 |
el('p', { class: 'chart-title' }, title),
|
| 109 |
-
segmented('
|
| 110 |
mode === 'chart' ? views.chart() : el('div', { class: 'table-wrap' }, views.table()),
|
| 111 |
);
|
| 112 |
const more = extra ? extra() : null;
|
|
@@ -177,7 +189,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
|
|
| 177 |
for (const r of rows) for (const c of cols) vals.push(value(r, c));
|
| 178 |
const scale = logBins(vals);
|
| 179 |
const maxV = Math.max(0, ...vals);
|
| 180 |
-
const grid = el('div', { class: 'heat', role: 'grid', 'aria-label':
|
| 181 |
style: { gridTemplateColumns: `minmax(min-content, 10rem) repeat(${cols.length}, minmax(22px, 3rem))`,
|
| 182 |
gridTemplateRows: `auto repeat(${rows.length}, minmax(34px, 1fr))` } });
|
| 183 |
grid.append(el('div', { class: 'corner', role: 'presentation' }));
|
|
@@ -196,7 +208,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
|
|
| 196 |
const isCur = current && current.row === r.key && current.col === c.key;
|
| 197 |
const cell = el('button', {
|
| 198 |
type: 'button', class: `cell${fill ? '' : ' empty'}${isCur ? ' current' : ''}`, role: 'gridcell', tabindex: '-1',
|
| 199 |
-
'aria-label':
|
| 200 |
style: fill ? { background: fill } : null,
|
| 201 |
onclick: () => onPick(r, c),
|
| 202 |
});
|
|
@@ -234,7 +246,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
|
|
| 234 |
const ticks = el('span', { class: 'ticks', 'aria-hidden': 'true' },
|
| 235 |
el('span', {}, '0'), scale.edges.slice(0, RAMP.length).map((e) => el('span', {}, fmtShort(e))));
|
| 236 |
const legend = el('div', { class: 'scale' },
|
| 237 |
-
el('span', {},
|
| 238 |
el('span', {}, steps, ticks));
|
| 239 |
return { grid, legend };
|
| 240 |
}
|
|
|
|
| 1 |
// Small chart kit: elements, number formats, tooltip, chart/table toggle, thin bars, heatmap.
|
| 2 |
// Every string from the data goes in with textContent (through el), never innerHTML.
|
| 3 |
+
import { locale, t } from './i18n.js';
|
| 4 |
|
| 5 |
export function el(tag, attrs = {}, ...kids) {
|
| 6 |
const n = document.createElement(tag);
|
|
|
|
| 21 |
|
| 22 |
export const clear = (n) => { while (n.firstChild) n.firstChild.remove(); return n; };
|
| 23 |
|
| 24 |
+
// ---- numbers and dates, by Intl in the current language
|
| 25 |
+
const formats = new Map();
|
| 26 |
+
const nf = (opts = {}) => {
|
| 27 |
+
const k = `${locale()} ${JSON.stringify(opts)}`;
|
| 28 |
+
if (!formats.has(k)) formats.set(k, new Intl.NumberFormat(locale(), opts));
|
| 29 |
+
return formats.get(k);
|
| 30 |
+
};
|
| 31 |
+
// three significant digits and a scale word: 8.97 million in English, 897 and the sign for ten thousand in Chinese
|
| 32 |
+
const COMPACT = { notation: 'compact', compactDisplay: 'long', minimumSignificantDigits: 3, maximumSignificantDigits: 3 };
|
| 33 |
+
export const fmtInt = (v) => (v == null ? t('num.unknown') : nf().format(Math.round(v)));
|
| 34 |
export function fmtCompact(v) {
|
| 35 |
+
if (v == null) return t('num.unknown');
|
| 36 |
+
return Math.abs(v) >= 1e4 ? nf(COMPACT).format(v) : nf().format(v);
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
}
|
| 38 |
+
/** fmtCompact split into its digits and its scale word (8970000 -> ['8.97', 'million'] in English); small numbers
|
| 39 |
+
* whole, with no scale word. */
|
| 40 |
+
export function compactParts(v) {
|
| 41 |
+
if (Math.abs(v) < 1e4) return [fmtInt(v), ''];
|
| 42 |
+
const parts = nf(COMPACT).formatToParts(v);
|
| 43 |
+
const join = (keep) => parts.filter((p) => keep(p.type)).map((p) => p.value).join('');
|
| 44 |
+
return [join((k) => k !== 'compact' && k !== 'literal'), join((k) => k === 'compact')];
|
|
|
|
| 45 |
}
|
| 46 |
+
export const fmtShort = (v) => (v == null ? '' : nf({ notation: 'compact' }).format(v));
|
| 47 |
export function fmtBytes(v) {
|
| 48 |
+
if (v == null) return t('num.unknown');
|
| 49 |
const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB'];
|
| 50 |
let i = 0;
|
| 51 |
let x = v;
|
| 52 |
while (x >= 1000 && i < units.length - 1) { x /= 1000; i++; }
|
| 53 |
+
const s = i === 0 ? nf().format(x) : nf({ minimumSignificantDigits: 3, maximumSignificantDigits: 3 }).format(x);
|
| 54 |
return `${s} ${units[i]}`;
|
| 55 |
}
|
| 56 |
export const fmtPct = (part, whole) => {
|
| 57 |
+
const pct = (p) => nf({ style: 'percent', maximumFractionDigits: 0 }).format(p);
|
| 58 |
+
if (!whole) return pct(0);
|
| 59 |
+
const p = part / whole;
|
| 60 |
+
return p > 0 && p < 0.01 ? `<${pct(0.01)}` : pct(p);
|
| 61 |
};
|
| 62 |
+
/** A date of the index (YYYY-MM-DD at the start of an ISO time), in UTC; opts: Intl.DateTimeFormat options. */
|
| 63 |
+
export function fmtDate(iso, opts = { day: 'numeric', month: 'short', year: 'numeric' }) {
|
| 64 |
+
if (!iso) return '';
|
| 65 |
+
const d = new Date(`${iso.slice(0, 10)}T00:00:00Z`);
|
| 66 |
+
return Number.isNaN(d.getTime()) ? iso.slice(0, 10) : new Intl.DateTimeFormat(locale(), { ...opts, timeZone: 'UTC' }).format(d);
|
| 67 |
+
}
|
| 68 |
+
/** Alternatives as one phrase: a, b or c. */
|
| 69 |
+
export const fmtOr = (list) => new Intl.ListFormat(locale(), { type: 'disjunction' }).format(list);
|
| 70 |
|
| 71 |
// ---- tooltip: the value leads, the label follows; also shown on keyboard focus
|
| 72 |
const tip = () => document.getElementById('tip');
|
|
|
|
| 118 |
clear(fig).append(
|
| 119 |
el('div', { class: 'chart-head' },
|
| 120 |
el('p', { class: 'chart-title' }, title),
|
| 121 |
+
segmented(t('chart.view'), [['chart', t('chart.chart')], ['table', t('chart.table')]], mode, (v) => { viewState.set(key, v); vt(draw); })),
|
| 122 |
mode === 'chart' ? views.chart() : el('div', { class: 'table-wrap' }, views.table()),
|
| 123 |
);
|
| 124 |
const more = extra ? extra() : null;
|
|
|
|
| 189 |
for (const r of rows) for (const c of cols) vals.push(value(r, c));
|
| 190 |
const scale = logBins(vals);
|
| 191 |
const maxV = Math.max(0, ...vals);
|
| 192 |
+
const grid = el('div', { class: 'heat', role: 'grid', 'aria-label': t('heat.aria', { metric: metricName }),
|
| 193 |
style: { gridTemplateColumns: `minmax(min-content, 10rem) repeat(${cols.length}, minmax(22px, 3rem))`,
|
| 194 |
gridTemplateRows: `auto repeat(${rows.length}, minmax(34px, 1fr))` } });
|
| 195 |
grid.append(el('div', { class: 'corner', role: 'presentation' }));
|
|
|
|
| 208 |
const isCur = current && current.row === r.key && current.col === c.key;
|
| 209 |
const cell = el('button', {
|
| 210 |
type: 'button', class: `cell${fill ? '' : ' empty'}${isCur ? ' current' : ''}`, role: 'gridcell', tabindex: '-1',
|
| 211 |
+
'aria-label': t('heat.cell', { row: r.label, col: c.label, value: fmt(v) }), 'aria-pressed': isCur ? 'true' : 'false',
|
| 212 |
style: fill ? { background: fill } : null,
|
| 213 |
onclick: () => onPick(r, c),
|
| 214 |
});
|
|
|
|
| 246 |
const ticks = el('span', { class: 'ticks', 'aria-hidden': 'true' },
|
| 247 |
el('span', {}, '0'), scale.edges.slice(0, RAMP.length).map((e) => el('span', {}, fmtShort(e))));
|
| 248 |
const legend = el('div', { class: 'scale' },
|
| 249 |
+
el('span', {}, t('heat.scale', { metric: metricName })),
|
| 250 |
el('span', {}, steps, ticks));
|
| 251 |
return { grid, legend };
|
| 252 |
}
|
data.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
// Reading the index in the browser. Every file comes from one commit of the dataset, so every number agrees.
|
| 2 |
import { asyncBufferFromUrl, parquetMetadataAsync, parquetReadObjects } from 'https://cdn.jsdelivr.net/npm/hyparquet@1.31.5/+esm';
|
| 3 |
import { compressors } from 'https://cdn.jsdelivr.net/npm/hyparquet-compressors@1.1.2/+esm';
|
|
|
|
| 4 |
|
| 5 |
export const REPO = 'Battam/agentic-env-index';
|
| 6 |
export const HF = 'https://huggingface.co';
|
|
@@ -47,7 +48,7 @@ export async function readColumns(url, cols, range = {}) {
|
|
| 47 |
/** The commit to read: ?rev=<sha or branch>, else main. Returns {sha, date, files}. */
|
| 48 |
export async function loadRevision(rev) {
|
| 49 |
const j = await getJson(`${HF}/api/datasets/${REPO}/revision/${encodeURIComponent(rev || 'main')}`);
|
| 50 |
-
if (!/^[0-9a-f]{40}$/.test(j.sha || '')) throw new Error('
|
| 51 |
return { sha: j.sha, date: j.lastModified || null, files: (j.siblings || []).map((s) => s.rfilename) };
|
| 52 |
}
|
| 53 |
|
|
@@ -194,7 +195,8 @@ async function readCatalog(url) {
|
|
| 194 |
return readParquet(url);
|
| 195 |
}
|
| 196 |
|
| 197 |
-
/** Everything the page shows, from one commit. params: URLSearchParams (rev, catalog, vocabulary)
|
|
|
|
| 198 |
export async function loadIndex(params, onNote) {
|
| 199 |
const rev = await loadRevision(params.get('rev'));
|
| 200 |
const at = (p) => fileUrl(rev.sha, p);
|
|
@@ -210,7 +212,7 @@ export async function loadIndex(params, onNote) {
|
|
| 210 |
readParquet(at('meta/not_included.parquet')).catch(() => []),
|
| 211 |
getText(at('CHANGELOG.md')).catch(() => ''),
|
| 212 |
getText(at('README.md')).catch(() => ''),
|
| 213 |
-
vocabUrl ? getJson(vocabUrl).catch((e) => { onNote?.(
|
| 214 |
]);
|
| 215 |
|
| 216 |
const vocab = parseVocabulary(vocabJson || {});
|
|
|
|
| 1 |
// Reading the index in the browser. Every file comes from one commit of the dataset, so every number agrees.
|
| 2 |
import { asyncBufferFromUrl, parquetMetadataAsync, parquetReadObjects } from 'https://cdn.jsdelivr.net/npm/hyparquet@1.31.5/+esm';
|
| 3 |
import { compressors } from 'https://cdn.jsdelivr.net/npm/hyparquet-compressors@1.1.2/+esm';
|
| 4 |
+
import { t } from './i18n.js';
|
| 5 |
|
| 6 |
export const REPO = 'Battam/agentic-env-index';
|
| 7 |
export const HF = 'https://huggingface.co';
|
|
|
|
| 48 |
/** The commit to read: ?rev=<sha or branch>, else main. Returns {sha, date, files}. */
|
| 49 |
export async function loadRevision(rev) {
|
| 50 |
const j = await getJson(`${HF}/api/datasets/${REPO}/revision/${encodeURIComponent(rev || 'main')}`);
|
| 51 |
+
if (!/^[0-9a-f]{40}$/.test(j.sha || '')) throw new Error(t('error.noCommit'));
|
| 52 |
return { sha: j.sha, date: j.lastModified || null, files: (j.siblings || []).map((s) => s.rfilename) };
|
| 53 |
}
|
| 54 |
|
|
|
|
| 195 |
return readParquet(url);
|
| 196 |
}
|
| 197 |
|
| 198 |
+
/** Everything the page shows, from one commit. params: URLSearchParams (rev, catalog, vocabulary); onNote(key, values):
|
| 199 |
+
* a note for the reader, as a key of the locale files. */
|
| 200 |
export async function loadIndex(params, onNote) {
|
| 201 |
const rev = await loadRevision(params.get('rev'));
|
| 202 |
const at = (p) => fileUrl(rev.sha, p);
|
|
|
|
| 212 |
readParquet(at('meta/not_included.parquet')).catch(() => []),
|
| 213 |
getText(at('CHANGELOG.md')).catch(() => ''),
|
| 214 |
getText(at('README.md')).catch(() => ''),
|
| 215 |
+
vocabUrl ? getJson(vocabUrl).catch((e) => { onNote?.('notice.vocabulary', { error: e.message }); return null; }) : null,
|
| 216 |
]);
|
| 217 |
|
| 218 |
const vocab = parseVocabulary(vocabJson || {});
|
first.js
CHANGED
|
@@ -1,7 +1,8 @@
|
|
| 1 |
// The first screen, drawn before the index is read: the pictograms, and the hero from first.json (written by the deploy
|
| 2 |
// step from the index commit of the day). app.js redraws the hero from the index itself when it has read it; the numbers
|
| 3 |
// are the same when both name the same index commit (portal.py first_screen and data.js count alike).
|
| 4 |
-
import { fmtBytes, fmtCompact, fmtInt } from './charts.js';
|
|
|
|
| 5 |
import { icon } from './icons.js';
|
| 6 |
|
| 7 |
const reduced = () => matchMedia('(prefers-reduced-motion: reduce)').matches;
|
|
@@ -33,27 +34,23 @@ export function setNumber(node, key, value, fmt) {
|
|
| 33 |
node.addEventListener('animationend', () => node.classList.remove('updated'), { once: true });
|
| 34 |
}
|
| 35 |
|
| 36 |
-
function dateText(iso) {
|
| 37 |
-
if (!iso) return '';
|
| 38 |
-
const d = new Date(`${iso.slice(0, 10)}T00:00:00Z`);
|
| 39 |
-
return Number.isNaN(d.getTime()) ? iso.slice(0, 10) : d.toLocaleDateString('en-GB', { day: 'numeric', month: 'short', year: 'numeric', timeZone: 'UTC' });
|
| 40 |
-
}
|
| 41 |
-
|
| 42 |
/** The size line (#364): distinct content and the rows without a size, not in it; the words of the index README,
|
| 43 |
* section Size. */
|
| 44 |
function drawSize(v) {
|
| 45 |
const node = document.getElementById('hero-size');
|
| 46 |
if (!node || !('distinct_bytes' in v)) return;
|
| 47 |
-
if (v.distinct_bytes == null) { node.replaceChildren('
|
| 48 |
const big = document.createElement('b');
|
| 49 |
-
const unsized = v.unsized_rows ? ` · ${
|
| 50 |
-
node.replaceChildren(big, ` distinct
|
| 51 |
setNumber(big, 'distinct_bytes', v.distinct_bytes, fmtBytes);
|
| 52 |
}
|
| 53 |
|
|
|
|
| 54 |
/** v: {ready_units, ready_sources, answers, raw_rows, sources, distinct_bytes, unsized_rows, index, date}. */
|
| 55 |
export function drawHero(v) {
|
| 56 |
if (!v) return;
|
|
|
|
| 57 |
setNumber(document.getElementById('hero-big'), 'ready_units', v.ready_units, fmtInt);
|
| 58 |
setNumber(document.getElementById('material-n'), 'material_units', v.ready_units, fmtInt);
|
| 59 |
drawSize(v);
|
|
@@ -63,15 +60,17 @@ export function drawHero(v) {
|
|
| 63 |
}
|
| 64 |
const pin = document.getElementById('pin-link');
|
| 65 |
if (v.index && pin) {
|
| 66 |
-
pin.textContent =
|
| 67 |
pin.href = `https://huggingface.co/datasets/Battam/agentic-env-index/tree/${v.index}`;
|
| 68 |
-
pin.title =
|
| 69 |
}
|
| 70 |
const foot = document.getElementById('foot-pin');
|
| 71 |
-
if (v.index && foot) foot.textContent =
|
| 72 |
}
|
| 73 |
|
| 74 |
decorate();
|
|
|
|
|
|
|
| 75 |
let drawnFromIndex = false;
|
| 76 |
export const markFromIndex = () => { drawnFromIndex = true; };
|
| 77 |
// first.json describes main of the published index; with ?rev=, ?catalog= or ?vocabulary= the page shows other data, so
|
|
|
|
| 1 |
// The first screen, drawn before the index is read: the pictograms, and the hero from first.json (written by the deploy
|
| 2 |
// step from the index commit of the day). app.js redraws the hero from the index itself when it has read it; the numbers
|
| 3 |
// are the same when both name the same index commit (portal.py first_screen and data.js count alike).
|
| 4 |
+
import { fmtBytes, fmtCompact, fmtDate, fmtInt } from './charts.js';
|
| 5 |
+
import { onLanguage, t } from './i18n.js';
|
| 6 |
import { icon } from './icons.js';
|
| 7 |
|
| 8 |
const reduced = () => matchMedia('(prefers-reduced-motion: reduce)').matches;
|
|
|
|
| 34 |
node.addEventListener('animationend', () => node.classList.remove('updated'), { once: true });
|
| 35 |
}
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
/** The size line (#364): distinct content and the rows without a size, not in it; the words of the index README,
|
| 38 |
* section Size. */
|
| 39 |
function drawSize(v) {
|
| 40 |
const node = document.getElementById('hero-size');
|
| 41 |
if (!node || !('distinct_bytes' in v)) return;
|
| 42 |
+
if (v.distinct_bytes == null) { node.replaceChildren(t('hero.sizeUnknown')); return; }
|
| 43 |
const big = document.createElement('b');
|
| 44 |
+
const unsized = v.unsized_rows ? ` · ${t('hero.unsized', { count: v.unsized_rows, n: fmtInt(v.unsized_rows) })}` : '';
|
| 45 |
+
node.replaceChildren(big, ` ${t('hero.distinct')}${unsized}`);
|
| 46 |
setNumber(big, 'distinct_bytes', v.distinct_bytes, fmtBytes);
|
| 47 |
}
|
| 48 |
|
| 49 |
+
let last = null;
|
| 50 |
/** v: {ready_units, ready_sources, answers, raw_rows, sources, distinct_bytes, unsized_rows, index, date}. */
|
| 51 |
export function drawHero(v) {
|
| 52 |
if (!v) return;
|
| 53 |
+
last = v;
|
| 54 |
setNumber(document.getElementById('hero-big'), 'ready_units', v.ready_units, fmtInt);
|
| 55 |
setNumber(document.getElementById('material-n'), 'material_units', v.ready_units, fmtInt);
|
| 56 |
drawSize(v);
|
|
|
|
| 60 |
}
|
| 61 |
const pin = document.getElementById('pin-link');
|
| 62 |
if (v.index && pin) {
|
| 63 |
+
pin.textContent = fmtDate(v.date);
|
| 64 |
pin.href = `https://huggingface.co/datasets/Battam/agentic-env-index/tree/${v.index}`;
|
| 65 |
+
pin.title = t('hero.commit', { sha: v.index });
|
| 66 |
}
|
| 67 |
const foot = document.getElementById('foot-pin');
|
| 68 |
+
if (v.index && foot) foot.textContent = t('foot.index', { sha: v.index.slice(0, 7), date: fmtDate(v.date) });
|
| 69 |
}
|
| 70 |
|
| 71 |
decorate();
|
| 72 |
+
// the hero in the new language: the same numbers, written again
|
| 73 |
+
onLanguage(() => drawHero(last));
|
| 74 |
let drawnFromIndex = false;
|
| 75 |
export const markFromIndex = () => { drawnFromIndex = true; };
|
| 76 |
// first.json describes main of the published index; with ?rev=, ?catalog= or ?vocabulary= the page shows other data, so
|
first.json
CHANGED
|
@@ -1,13 +1,13 @@
|
|
| 1 |
{
|
| 2 |
"answers": 590,
|
| 3 |
"date": "2026-10-29",
|
| 4 |
-
"distinct_bytes":
|
| 5 |
"format": "portal-first/1",
|
| 6 |
-
"index": "
|
| 7 |
-
"raw_rows":
|
| 8 |
"raw_sources": 89,
|
| 9 |
-
"ready_sources":
|
| 10 |
-
"ready_units":
|
| 11 |
-
"sources":
|
| 12 |
-
"unsized_rows":
|
| 13 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"answers": 590,
|
| 3 |
"date": "2026-10-29",
|
| 4 |
+
"distinct_bytes": 3786223653585058,
|
| 5 |
"format": "portal-first/1",
|
| 6 |
+
"index": "02b52f1fb8ec44ec91c6593f759faff0be42b200",
|
| 7 |
+
"raw_rows": 1150137524,
|
| 8 |
"raw_sources": 89,
|
| 9 |
+
"ready_sources": 1783,
|
| 10 |
+
"ready_units": 3962,
|
| 11 |
+
"sources": 1872,
|
| 12 |
+
"unsized_rows": 281582670
|
| 13 |
}
|
i18n.js
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// The page's languages, English and Simplified Chinese, by i18next; the reader's choice by its browser language
|
| 2 |
+
// detector: the query string first (?lng=zh), then localStorage, and kept in localStorage. Every word the page writes
|
| 3 |
+
// comes from t by key (locales/en.json, locales/zh.json); text read from the dataset stays as it is.
|
| 4 |
+
// Standard tool: the page's languages by i18next, with i18next-browser-languagedetector for the choice and its memory.
|
| 5 |
+
import i18next from 'https://cdn.jsdelivr.net/npm/i18next@26.4.5/+esm';
|
| 6 |
+
import LanguageDetector from 'https://cdn.jsdelivr.net/npm/i18next-browser-languagedetector@8.2.1/+esm';
|
| 7 |
+
|
| 8 |
+
const LANGS = ['en', 'zh'];
|
| 9 |
+
const HTML_LANG = { en: 'en', zh: 'zh-CN' };
|
| 10 |
+
|
| 11 |
+
const resources = Object.fromEntries(await Promise.all(LANGS.map(async (lng) => {
|
| 12 |
+
const r = await fetch(new URL(`locales/${lng}.json`, import.meta.url));
|
| 13 |
+
return [lng, { translation: await r.json() }];
|
| 14 |
+
})));
|
| 15 |
+
|
| 16 |
+
await i18next.use(LanguageDetector).init({
|
| 17 |
+
resources,
|
| 18 |
+
fallbackLng: 'en',
|
| 19 |
+
supportedLngs: LANGS,
|
| 20 |
+
// flat keys: a key may hold a code with dots (domain.D10.2) or colons
|
| 21 |
+
keySeparator: false,
|
| 22 |
+
nsSeparator: false,
|
| 23 |
+
// every text goes into the page with textContent, never innerHTML
|
| 24 |
+
interpolation: { escapeValue: false },
|
| 25 |
+
detection: { order: ['querystring', 'localStorage'], lookupQuerystring: 'lng', caches: ['localStorage'] },
|
| 26 |
+
});
|
| 27 |
+
|
| 28 |
+
export const t = (...args) => i18next.t(...args);
|
| 29 |
+
/** The words of the issue texts the page prefills: the hub writes English. */
|
| 30 |
+
export const tEn = i18next.getFixedT('en');
|
| 31 |
+
export const lang = () => i18next.resolvedLanguage;
|
| 32 |
+
/** The current language as a BCP 47 tag, for <html lang> and Intl. */
|
| 33 |
+
export const locale = () => HTML_LANG[lang()];
|
| 34 |
+
export const onLanguage = (fn) => i18next.on('languageChanged', fn);
|
| 35 |
+
|
| 36 |
+
/** The page's own words in index.html (its <title> too): [data-i18n] sets the text, [data-i18n-attr] attributes
|
| 37 |
+
* ("name:key;name:key"); <html lang> and the language switch. */
|
| 38 |
+
export function translatePage() {
|
| 39 |
+
document.documentElement.lang = locale();
|
| 40 |
+
for (const n of document.querySelectorAll('[data-i18n]')) n.textContent = t(n.dataset.i18n);
|
| 41 |
+
for (const n of document.querySelectorAll('[data-i18n-attr]')) {
|
| 42 |
+
for (const pair of n.dataset.i18nAttr.split(';')) {
|
| 43 |
+
const [name, key] = pair.split(':');
|
| 44 |
+
n.setAttribute(name, t(key));
|
| 45 |
+
}
|
| 46 |
+
}
|
| 47 |
+
const other = LANGS.find((l) => l !== lang());
|
| 48 |
+
const button = document.getElementById('lang-switch');
|
| 49 |
+
button.textContent = i18next.getFixedT(other)('lang.name');
|
| 50 |
+
button.lang = HTML_LANG[other];
|
| 51 |
+
button.title = t('lang.switch');
|
| 52 |
+
button.hidden = false;
|
| 53 |
+
}
|
| 54 |
+
|
| 55 |
+
/** The switch: the other language, without a reload. A ?lng= in the address goes, so the choice kept in localStorage
|
| 56 |
+
* holds on the next visit. */
|
| 57 |
+
export function setUpSwitch() {
|
| 58 |
+
document.getElementById('lang-switch').addEventListener('click', () => {
|
| 59 |
+
const p = new URLSearchParams(location.search);
|
| 60 |
+
if (p.has('lng')) {
|
| 61 |
+
p.delete('lng');
|
| 62 |
+
const s = p.toString();
|
| 63 |
+
history.replaceState(null, '', s ? `?${s}` : location.pathname);
|
| 64 |
+
}
|
| 65 |
+
i18next.changeLanguage(LANGS.find((l) => l !== lang()));
|
| 66 |
+
});
|
| 67 |
+
}
|
index.html
CHANGED
|
@@ -3,33 +3,37 @@
|
|
| 3 |
<head>
|
| 4 |
<meta charset="utf-8">
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
-
<title>Agentic Env Catalog</title>
|
| 7 |
<link rel="icon" href="data:,">
|
| 8 |
-
<link rel="alternate" type="text/plain" href="llms.txt" title="For agents">
|
| 9 |
-
<meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it.">
|
| 10 |
<meta name="theme-color" content="#112532">
|
| 11 |
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 12 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 13 |
<link rel="preconnect" href="https://huggingface.co">
|
| 14 |
<link rel="preload" href="first.json" as="fetch" crossorigin>
|
|
|
|
|
|
|
| 15 |
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=IBM+Plex+Mono:wght@500&family=Noto+Sans+SC:wght@400;500;700&display=swap">
|
| 16 |
<link rel="stylesheet" href="style.css">
|
| 17 |
<script type="module" src="first.js" async></script>
|
| 18 |
<script type="module" src="app.js"></script>
|
| 19 |
</head>
|
| 20 |
<body>
|
| 21 |
-
<nav class="topnav" aria-label="Page sections">
|
| 22 |
<div class="wrap nav-row">
|
| 23 |
-
<a class="brand" href="#top" data-icon="layers" data-size="22"><span>Agentic Env Catalog</span></a>
|
| 24 |
<ol class="progress">
|
| 25 |
-
<li><a href="#material" data-part="overview" data-icon="overview"><span class="pl">Overview</span><span class="track"><span class="fill"></span></span></a></li>
|
| 26 |
-
<li><a href="#map" data-part="browse" data-icon="browse"><span class="pl">Browse</span><span class="track"><span class="fill"></span></span></a></li>
|
| 27 |
-
<li><a href="#get" data-part="get" data-icon="get"><span class="pl">Get the data</span><span class="track"><span class="fill"></span></span></a></li>
|
| 28 |
-
<li><a href="#excluded" data-part="excluded" data-icon="excluded"><span class="pl">Not included</span><span class="track"><span class="fill"></span></span></a></li>
|
| 29 |
</ol>
|
| 30 |
<div class="nav-links">
|
| 31 |
-
<a href="llms.txt" data-icon="robot" data-size="16">For agents</a>
|
| 32 |
-
<a class="btn solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
|
|
|
|
|
|
|
| 33 |
</div>
|
| 34 |
</div>
|
| 35 |
</nav>
|
|
@@ -38,30 +42,30 @@
|
|
| 38 |
<header class="hero" id="top" data-part="overview">
|
| 39 |
<div class="wrap hero-grid">
|
| 40 |
<div class="hero-main">
|
| 41 |
-
<p class="eyebrow" data-icon="sources" data-size="18"><span>The Agentic Env Index, by category</span></p>
|
| 42 |
-
<h1>Real material for agent tasks</h1>
|
| 43 |
<div class="hero-number">
|
| 44 |
<span class="display" id="hero-big"><span class="sk sk-display"></span></span>
|
| 45 |
-
<span class="display-label">task-ready units</span>
|
| 46 |
</div>
|
| 47 |
<p class="hero-size" id="hero-size"><span class="sk sk-line"></span></p>
|
| 48 |
<div class="hero-cta">
|
| 49 |
-
<a class="btn hero-btn" href="#material" data-icon="browse" data-size="18">Browse categories</a>
|
| 50 |
-
<a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18">Get the data</a>
|
| 51 |
</div>
|
| 52 |
</div>
|
| 53 |
-
<ul class="hero-facts" id="hero-facts" aria-label="Facts">
|
| 54 |
-
<li class="fact-tile" data-k="ready_sources"><span class="fact-ico" data-icon="check" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">sources ready</span></li>
|
| 55 |
-
<li class="fact-tile" data-k="answers"><span class="fact-ico" data-icon="target" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">units with an answer</span></li>
|
| 56 |
-
<li class="fact-tile" data-k="raw_rows"><span class="fact-ico" data-icon="rows" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">raw rows</span></li>
|
| 57 |
-
<li class="fact-tile" data-k="sources"><span class="fact-ico" data-icon="database" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">sources indexed</span></li>
|
| 58 |
</ul>
|
| 59 |
</div>
|
| 60 |
<div class="wrap hero-foot">
|
| 61 |
-
<span class="hero-pin">Index of <a id="pin-link" href="https://huggingface.co/datasets/Battam/agentic-env-index"><span class="sk sk-word"></span></a></span>
|
| 62 |
<span class="chip chip-dark" id="preview-chip" hidden></span>
|
| 63 |
-
<a href="https://huggingface.co/datasets/Battam/agentic-env-index">Dataset</a>
|
| 64 |
-
<a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md">Catalog README</a>
|
| 65 |
<span class="notice" id="notice" hidden></span>
|
| 66 |
</div>
|
| 67 |
</header>
|
|
@@ -69,14 +73,14 @@
|
|
| 69 |
<section class="band band-canvas" id="material" data-part="overview" aria-labelledby="material-h">
|
| 70 |
<div class="wrap">
|
| 71 |
<div class="band-head reveal">
|
| 72 |
-
<h2 id="material-h">Where the <span id="material-n"><span class="sk sk-num"></span></span> task-ready units are</h2>
|
| 73 |
</div>
|
| 74 |
<!-- two cards: the task-ready units stacked four ways, and the largest task-ready sources; the skeletons have the final cards' sizes, so nothing moves when the index arrives -->
|
| 75 |
<div class="where-grid">
|
| 76 |
<figure class="card chart-card where-card" id="ready-card"><div class="sk sk-block"></div></figure>
|
| 77 |
<figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
|
| 78 |
</div>
|
| 79 |
-
<div class="band-head sub-head reveal"><h3 id="cats-h">Most requested categories</h3></div>
|
| 80 |
<!-- one card per most requested category (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
|
| 81 |
<ul class="cat-grid" id="req-cards">
|
| 82 |
<li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
|
|
@@ -86,41 +90,41 @@
|
|
| 86 |
|
| 87 |
<div class="band band-white" id="browse" data-part="browse">
|
| 88 |
<div class="wrap browse-grid">
|
| 89 |
-
<aside class="filters" id="filters" aria-label="Filters">
|
| 90 |
<button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row" data-icon="pick" data-size="18">
|
| 91 |
-
<span>Filters</span><span class="state" id="filter-state">none on</span>
|
| 92 |
</button>
|
| 93 |
<div class="filter-row" id="filter-row">
|
| 94 |
-
<p class="filter-title" data-icon="pick" data-size="18"><span>Filters</span></p>
|
| 95 |
-
<label>Domain<select id="f-domain"><option value="">All</option></select></label>
|
| 96 |
-
<label>Subdomain<select id="f-leaf" disabled><option value="">All</option></select></label>
|
| 97 |
-
<label>Form<select id="f-form"><option value="">All</option></select></label>
|
| 98 |
-
<label>Kind<select id="f-status">
|
| 99 |
-
<option value="">Any</option>
|
| 100 |
-
<option value="ready">Task-ready</option>
|
| 101 |
-
<option value="raw">Raw corpus</option>
|
| 102 |
-
<option value="flagged">Flagged</option>
|
| 103 |
</select></label>
|
| 104 |
-
<label>Training<select id="f-train">
|
| 105 |
-
<option value="">Any</option>
|
| 106 |
-
<option value="yes">Allowed, all rows</option>
|
| 107 |
-
<option value="cond">Allowed or conditional</option>
|
| 108 |
</select></label>
|
| 109 |
-
<label>Shipping<select id="f-redist">
|
| 110 |
-
<option value="">Any</option>
|
| 111 |
-
<option value="yes">Allowed, all rows</option>
|
| 112 |
-
<option value="cond">Allowed or conditional</option>
|
| 113 |
</select></label>
|
| 114 |
-
<label>Licence tier<select id="f-tier"><option value="">All</option></select></label>
|
| 115 |
-
<label>Source size<select id="f-size">
|
| 116 |
-
<option value="">Any</option>
|
| 117 |
-
<option value="lt1g">Under 1 GB</option>
|
| 118 |
-
<option value="1g1t">1 GB to 1 TB</option>
|
| 119 |
-
<option value="ge1t">1 TB or more</option>
|
| 120 |
-
<option value="unknown">Not published</option>
|
| 121 |
</select></label>
|
| 122 |
-
<label class="wide">Search<input id="f-q" type="search" placeholder="Title or text" autocomplete="off"></label>
|
| 123 |
-
<div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled>Clear</button></div>
|
| 124 |
</div>
|
| 125 |
</aside>
|
| 126 |
|
|
@@ -128,11 +132,11 @@
|
|
| 128 |
<section id="map" aria-labelledby="map-h">
|
| 129 |
<div class="band-head row reveal">
|
| 130 |
<h2 id="map-h"><span class="sk sk-line"></span></h2>
|
| 131 |
-
<span class="seg-wrap"><span class="seg-label" id="metric-label">Count</span>
|
| 132 |
<span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
|
| 133 |
-
<button type="button" data-metric="sources" aria-pressed="true">Sources</button>
|
| 134 |
-
<button type="button" data-metric="rows" aria-pressed="false">Pointer rows</button>
|
| 135 |
-
<button type="button" data-metric="bytes" aria-pressed="false">Size</button>
|
| 136 |
</span>
|
| 137 |
</span>
|
| 138 |
</div>
|
|
@@ -160,12 +164,12 @@
|
|
| 160 |
<div class="card agents-card lift reveal" id="agents">
|
| 161 |
<span class="agents-ico" data-icon="robot" data-size="36"></span>
|
| 162 |
<div class="agents-text">
|
| 163 |
-
<h3 id="agents-h">For agents</h3>
|
| 164 |
-
<p class="small">Catalog columns · the one command · the default profile · where answers land</p>
|
| 165 |
</div>
|
| 166 |
<div class="agents-actions">
|
| 167 |
<a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a>
|
| 168 |
-
<a class="btn" href="https://huggingface.co/datasets/Battam/agentic-env-index#for-agents" data-icon="doc" data-size="16">README: For agents</a>
|
| 169 |
</div>
|
| 170 |
</div>
|
| 171 |
</div>
|
|
@@ -187,16 +191,16 @@
|
|
| 187 |
<div class="foot-cta">
|
| 188 |
<span class="foot-ico" data-icon="request" data-size="32"></span>
|
| 189 |
<div>
|
| 190 |
-
<h2>Missing material?</h2>
|
| 191 |
-
<p>Ask for a category, a form or a source.</p>
|
| 192 |
</div>
|
| 193 |
-
<a class="btn on-dark-solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
|
| 194 |
</div>
|
| 195 |
<ul class="foot-links">
|
| 196 |
-
<li><a href="https://huggingface.co/datasets/Battam/agentic-env-index">Dataset</a></li>
|
| 197 |
-
<li><a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md">Catalog README</a></li>
|
| 198 |
-
<li><a href="llms.txt">For agents</a></li>
|
| 199 |
-
<li><a href="#top">Back to top</a></li>
|
| 200 |
</ul>
|
| 201 |
<p class="foot-pin mono" id="foot-pin"></p>
|
| 202 |
</div>
|
|
|
|
| 3 |
<head>
|
| 4 |
<meta charset="utf-8">
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
+
<title data-i18n="page.title">Agentic Env Catalog</title>
|
| 7 |
<link rel="icon" href="data:,">
|
| 8 |
+
<link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents">
|
| 9 |
+
<meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it." data-i18n-attr="content:page.description">
|
| 10 |
<meta name="theme-color" content="#112532">
|
| 11 |
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 12 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 13 |
<link rel="preconnect" href="https://huggingface.co">
|
| 14 |
<link rel="preload" href="first.json" as="fetch" crossorigin>
|
| 15 |
+
<link rel="preload" href="locales/en.json" as="fetch" crossorigin>
|
| 16 |
+
<link rel="preload" href="locales/zh.json" as="fetch" crossorigin>
|
| 17 |
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=IBM+Plex+Mono:wght@500&family=Noto+Sans+SC:wght@400;500;700&display=swap">
|
| 18 |
<link rel="stylesheet" href="style.css">
|
| 19 |
<script type="module" src="first.js" async></script>
|
| 20 |
<script type="module" src="app.js"></script>
|
| 21 |
</head>
|
| 22 |
<body>
|
| 23 |
+
<nav class="topnav" aria-label="Page sections" data-i18n-attr="aria-label:nav.sections">
|
| 24 |
<div class="wrap nav-row">
|
| 25 |
+
<a class="brand" href="#top" data-icon="layers" data-size="22"><span data-i18n="page.title">Agentic Env Catalog</span></a>
|
| 26 |
<ol class="progress">
|
| 27 |
+
<li><a href="#material" data-part="overview" data-icon="overview"><span class="pl" data-i18n="nav.overview">Overview</span><span class="track"><span class="fill"></span></span></a></li>
|
| 28 |
+
<li><a href="#map" data-part="browse" data-icon="browse"><span class="pl" data-i18n="nav.browse">Browse</span><span class="track"><span class="fill"></span></span></a></li>
|
| 29 |
+
<li><a href="#get" data-part="get" data-icon="get"><span class="pl" data-i18n="nav.get">Get the data</span><span class="track"><span class="fill"></span></span></a></li>
|
| 30 |
+
<li><a href="#excluded" data-part="excluded" data-icon="excluded"><span class="pl" data-i18n="nav.excluded">Not included</span><span class="track"><span class="fill"></span></span></a></li>
|
| 31 |
</ol>
|
| 32 |
<div class="nav-links">
|
| 33 |
+
<a href="llms.txt" data-icon="robot" data-size="16"><span data-i18n="nav.agents">For agents</span></a>
|
| 34 |
+
<a class="btn solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml" data-i18n="nav.request">Request data</a>
|
| 35 |
+
<!-- the language switch: the other language's name, in that language (i18n.js) -->
|
| 36 |
+
<button class="btn lang-btn" id="lang-switch" type="button" hidden></button>
|
| 37 |
</div>
|
| 38 |
</div>
|
| 39 |
</nav>
|
|
|
|
| 42 |
<header class="hero" id="top" data-part="overview">
|
| 43 |
<div class="wrap hero-grid">
|
| 44 |
<div class="hero-main">
|
| 45 |
+
<p class="eyebrow" data-icon="sources" data-size="18"><span data-i18n="hero.eyebrow">The Agentic Env Index, by category</span></p>
|
| 46 |
+
<h1 data-i18n="hero.title">Real material for agent tasks</h1>
|
| 47 |
<div class="hero-number">
|
| 48 |
<span class="display" id="hero-big"><span class="sk sk-display"></span></span>
|
| 49 |
+
<span class="display-label" data-i18n="hero.units">task-ready units</span>
|
| 50 |
</div>
|
| 51 |
<p class="hero-size" id="hero-size"><span class="sk sk-line"></span></p>
|
| 52 |
<div class="hero-cta">
|
| 53 |
+
<a class="btn hero-btn" href="#material" data-icon="browse" data-size="18"><span data-i18n="hero.browse">Browse categories</span></a>
|
| 54 |
+
<a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18"><span data-i18n="nav.get">Get the data</span></a>
|
| 55 |
</div>
|
| 56 |
</div>
|
| 57 |
+
<ul class="hero-facts" id="hero-facts" aria-label="Facts" data-i18n-attr="aria-label:hero.facts">
|
| 58 |
+
<li class="fact-tile" data-k="ready_sources"><span class="fact-ico" data-icon="check" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.ready_sources">sources ready</span></li>
|
| 59 |
+
<li class="fact-tile" data-k="answers"><span class="fact-ico" data-icon="target" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.answers">units with an answer</span></li>
|
| 60 |
+
<li class="fact-tile" data-k="raw_rows"><span class="fact-ico" data-icon="rows" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.raw_rows">raw rows</span></li>
|
| 61 |
+
<li class="fact-tile" data-k="sources"><span class="fact-ico" data-icon="database" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.sources">sources indexed</span></li>
|
| 62 |
</ul>
|
| 63 |
</div>
|
| 64 |
<div class="wrap hero-foot">
|
| 65 |
+
<span class="hero-pin"><span data-i18n="hero.indexOf">Index of</span> <a id="pin-link" href="https://huggingface.co/datasets/Battam/agentic-env-index"><span class="sk sk-word"></span></a></span>
|
| 66 |
<span class="chip chip-dark" id="preview-chip" hidden></span>
|
| 67 |
+
<a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a>
|
| 68 |
+
<a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md" data-i18n="link.readme">Catalog README</a>
|
| 69 |
<span class="notice" id="notice" hidden></span>
|
| 70 |
</div>
|
| 71 |
</header>
|
|
|
|
| 73 |
<section class="band band-canvas" id="material" data-part="overview" aria-labelledby="material-h">
|
| 74 |
<div class="wrap">
|
| 75 |
<div class="band-head reveal">
|
| 76 |
+
<h2 id="material-h"><span data-i18n="material.before">Where the</span> <span id="material-n"><span class="sk sk-num"></span></span> <span data-i18n="material.after">task-ready units are</span></h2>
|
| 77 |
</div>
|
| 78 |
<!-- two cards: the task-ready units stacked four ways, and the largest task-ready sources; the skeletons have the final cards' sizes, so nothing moves when the index arrives -->
|
| 79 |
<div class="where-grid">
|
| 80 |
<figure class="card chart-card where-card" id="ready-card"><div class="sk sk-block"></div></figure>
|
| 81 |
<figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
|
| 82 |
</div>
|
| 83 |
+
<div class="band-head sub-head reveal"><h3 id="cats-h" data-i18n="material.cats">Most requested categories</h3></div>
|
| 84 |
<!-- one card per most requested category (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
|
| 85 |
<ul class="cat-grid" id="req-cards">
|
| 86 |
<li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
|
|
|
|
| 90 |
|
| 91 |
<div class="band band-white" id="browse" data-part="browse">
|
| 92 |
<div class="wrap browse-grid">
|
| 93 |
+
<aside class="filters" id="filters" aria-label="Filters" data-i18n-attr="aria-label:filter.title">
|
| 94 |
<button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row" data-icon="pick" data-size="18">
|
| 95 |
+
<span data-i18n="filter.title">Filters</span><span class="state" id="filter-state" data-i18n="filter.noneOn">none on</span>
|
| 96 |
</button>
|
| 97 |
<div class="filter-row" id="filter-row">
|
| 98 |
+
<p class="filter-title" data-icon="pick" data-size="18"><span data-i18n="filter.title">Filters</span></p>
|
| 99 |
+
<label><span data-i18n="filter.domain">Domain</span><select id="f-domain"><option value="" data-i18n="filter.all">All</option></select></label>
|
| 100 |
+
<label><span data-i18n="filter.leaf">Subdomain</span><select id="f-leaf" disabled><option value="" data-i18n="filter.all">All</option></select></label>
|
| 101 |
+
<label><span data-i18n="filter.form">Form</span><select id="f-form"><option value="" data-i18n="filter.all">All</option></select></label>
|
| 102 |
+
<label><span data-i18n="filter.kind">Kind</span><select id="f-status">
|
| 103 |
+
<option value="" data-i18n="filter.any">Any</option>
|
| 104 |
+
<option value="ready" data-i18n="filter.status.ready">Task-ready</option>
|
| 105 |
+
<option value="raw" data-i18n="filter.status.raw">Raw corpus</option>
|
| 106 |
+
<option value="flagged" data-i18n="filter.status.flagged">Flagged</option>
|
| 107 |
</select></label>
|
| 108 |
+
<label><span data-i18n="filter.training">Training</span><select id="f-train">
|
| 109 |
+
<option value="" data-i18n="filter.any">Any</option>
|
| 110 |
+
<option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
|
| 111 |
+
<option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
|
| 112 |
</select></label>
|
| 113 |
+
<label><span data-i18n="filter.shipping">Shipping</span><select id="f-redist">
|
| 114 |
+
<option value="" data-i18n="filter.any">Any</option>
|
| 115 |
+
<option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
|
| 116 |
+
<option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
|
| 117 |
</select></label>
|
| 118 |
+
<label><span data-i18n="filter.tier">Licence tier</span><select id="f-tier"><option value="" data-i18n="filter.all">All</option></select></label>
|
| 119 |
+
<label><span data-i18n="filter.size">Source size</span><select id="f-size">
|
| 120 |
+
<option value="" data-i18n="filter.any">Any</option>
|
| 121 |
+
<option value="lt1g" data-i18n="filter.size.lt1g">Under 1 GB</option>
|
| 122 |
+
<option value="1g1t" data-i18n="filter.size.1g1t">1 GB to 1 TB</option>
|
| 123 |
+
<option value="ge1t" data-i18n="filter.size.ge1t">1 TB or more</option>
|
| 124 |
+
<option value="unknown" data-i18n="filter.size.unknown">Not published</option>
|
| 125 |
</select></label>
|
| 126 |
+
<label class="wide"><span data-i18n="filter.search">Search</span><input id="f-q" type="search" placeholder="Title or text" data-i18n-attr="placeholder:filter.searchHint" autocomplete="off"></label>
|
| 127 |
+
<div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled data-i18n="filter.clear">Clear</button></div>
|
| 128 |
</div>
|
| 129 |
</aside>
|
| 130 |
|
|
|
|
| 132 |
<section id="map" aria-labelledby="map-h">
|
| 133 |
<div class="band-head row reveal">
|
| 134 |
<h2 id="map-h"><span class="sk sk-line"></span></h2>
|
| 135 |
+
<span class="seg-wrap"><span class="seg-label" id="metric-label" data-i18n="metric.label">Count</span>
|
| 136 |
<span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
|
| 137 |
+
<button type="button" data-metric="sources" aria-pressed="true" data-i18n="metric.sources">Sources</button>
|
| 138 |
+
<button type="button" data-metric="rows" aria-pressed="false" data-i18n="metric.rows">Pointer rows</button>
|
| 139 |
+
<button type="button" data-metric="bytes" aria-pressed="false" data-i18n="metric.bytes">Size</button>
|
| 140 |
</span>
|
| 141 |
</span>
|
| 142 |
</div>
|
|
|
|
| 164 |
<div class="card agents-card lift reveal" id="agents">
|
| 165 |
<span class="agents-ico" data-icon="robot" data-size="36"></span>
|
| 166 |
<div class="agents-text">
|
| 167 |
+
<h3 id="agents-h" data-i18n="nav.agents">For agents</h3>
|
| 168 |
+
<p class="small" data-i18n="agents.text">Catalog columns · the one command · the default profile · where answers land</p>
|
| 169 |
</div>
|
| 170 |
<div class="agents-actions">
|
| 171 |
<a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a>
|
| 172 |
+
<a class="btn" href="https://huggingface.co/datasets/Battam/agentic-env-index#for-agents" data-icon="doc" data-size="16"><span data-i18n="agents.readme">README: For agents</span></a>
|
| 173 |
</div>
|
| 174 |
</div>
|
| 175 |
</div>
|
|
|
|
| 191 |
<div class="foot-cta">
|
| 192 |
<span class="foot-ico" data-icon="request" data-size="32"></span>
|
| 193 |
<div>
|
| 194 |
+
<h2 data-i18n="foot.missing">Missing material?</h2>
|
| 195 |
+
<p data-i18n="foot.ask">Ask for a category, a form or a source.</p>
|
| 196 |
</div>
|
| 197 |
+
<a class="btn on-dark-solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml" data-i18n="nav.request">Request data</a>
|
| 198 |
</div>
|
| 199 |
<ul class="foot-links">
|
| 200 |
+
<li><a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a></li>
|
| 201 |
+
<li><a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md" data-i18n="link.readme">Catalog README</a></li>
|
| 202 |
+
<li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li>
|
| 203 |
+
<li><a href="#top" data-i18n="foot.top">Back to top</a></li>
|
| 204 |
</ul>
|
| 205 |
<p class="foot-pin mono" id="foot-pin"></p>
|
| 206 |
</div>
|
llms.txt
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
# Agentic Env Index: for agents
|
| 2 |
|
| 3 |
- Pick sources from `meta/catalog.parquet` (one row per source, about 50 KB; do not download `meta/` whole). Columns: `source`; `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a raw corpus has one unit per row), `answers` (units that have an answer, the expected results kept apart) and `files` (files of every role); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (`not scanned` when no comparison is recorded for the source); `license.ship`; `pii.level`; `distinct_bytes` (distinct content) and `bytes` (referenced, the sum over rows). The vocabulary is `meta/vocabulary.json`: every domain code with its name, what it covers and its parent, and every form with what it means.
|
| 4 |
-
- Size of this release: Distinct content: 3.79 PB. 281,
|
| 5 |
- Task-ready sources are collections whose rows carry the roles unit, companion and answer (`answers` above 0, or `units` below `files`): each unit comes with its expected results. Every other source is a raw corpus to mine, one unit per row.
|
| 6 |
- Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--domain`, `--form`, `--tag`, `--ext`; `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
|
| 7 |
- It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
|
| 8 |
-
- The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, collections awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; `--all` turns the profile off, and each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet). Compared with the benchmark list (keys of 2026-10-24): 1,
|
| 9 |
- Output: `./data/<domain>/<source>/...` holds the units and their companion files (README, LICENSE); `./data/answer/<source>/...` holds the answers and expected results, apart from the material; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, answer, origin_kind, leaf, role, unit_group, answer_for); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
|
| 10 |
- Never put `answer/` into the environment the agent works in: the answers are there to check its result, not to be read by it. Each kind of answer, and what else to keep out:
|
| 11 |
- Answer files (role `answer`; an answer in JSON names its kind in the field `kind` or `answer_kind`): the expected result of one unit, such as an output, a value or a verdict. Keep out of the environment also any copy of it, and any file that holds the same result.
|
|
|
|
| 1 |
# Agentic Env Index: for agents
|
| 2 |
|
| 3 |
- Pick sources from `meta/catalog.parquet` (one row per source, about 50 KB; do not download `meta/` whole). Columns: `source`; `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a raw corpus has one unit per row), `answers` (units that have an answer, the expected results kept apart) and `files` (files of every role); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (`not scanned` when no comparison is recorded for the source); `license.ship`; `pii.level`; `distinct_bytes` (distinct content) and `bytes` (referenced, the sum over rows). The vocabulary is `meta/vocabulary.json`: every domain code with its name, what it covers and its parent, and every form with what it means.
|
| 4 |
+
- Size of this release: Distinct content: 3.79 PB. 281,582,670 rows have no size and are not in that number. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it.
|
| 5 |
- Task-ready sources are collections whose rows carry the roles unit, companion and answer (`answers` above 0, or `units` below `files`): each unit comes with its expected results. Every other source is a raw corpus to mine, one unit per row.
|
| 6 |
- Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--domain`, `--form`, `--tag`, `--ext`; `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
|
| 7 |
- It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
|
| 8 |
+
- The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, collections awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; `--all` turns the profile off, and each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet). Compared with the benchmark list (keys of 2026-10-24): 1,872 of 1,872 sources, 1,783 of 1,783 task-ready sources. The profile also removes every row whose data file was not compared (`--allow-benchmark` keeps them, unchecked).
|
| 9 |
- Output: `./data/<domain>/<source>/...` holds the units and their companion files (README, LICENSE); `./data/answer/<source>/...` holds the answers and expected results, apart from the material; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, answer, origin_kind, leaf, role, unit_group, answer_for); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
|
| 10 |
- Never put `answer/` into the environment the agent works in: the answers are there to check its result, not to be read by it. Each kind of answer, and what else to keep out:
|
| 11 |
- Answer files (role `answer`; an answer in JSON names its kind in the field `kind` or `answer_kind`): the expected result of one unit, such as an output, a value or a verdict. Keep out of the environment also any copy of it, and any file that holds the same result.
|
locales/en.json
ADDED
|
@@ -0,0 +1,379 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"lang.name": "English",
|
| 3 |
+
"lang.switch": "Show this page in Chinese",
|
| 4 |
+
|
| 5 |
+
"page.title": "Agentic Env Catalog",
|
| 6 |
+
"page.description": "Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it.",
|
| 7 |
+
|
| 8 |
+
"nav.sections": "Page sections",
|
| 9 |
+
"nav.overview": "Overview",
|
| 10 |
+
"nav.browse": "Browse",
|
| 11 |
+
"nav.get": "Get the data",
|
| 12 |
+
"nav.excluded": "Not included",
|
| 13 |
+
"nav.agents": "For agents",
|
| 14 |
+
"nav.request": "Request data",
|
| 15 |
+
|
| 16 |
+
"hero.eyebrow": "The Agentic Env Index, by category",
|
| 17 |
+
"hero.title": "Real material for agent tasks",
|
| 18 |
+
"hero.units": "task-ready units",
|
| 19 |
+
"hero.browse": "Browse categories",
|
| 20 |
+
"hero.facts": "Facts",
|
| 21 |
+
"hero.indexOf": "Index of",
|
| 22 |
+
"hero.sizeUnknown": "Distinct content not known yet",
|
| 23 |
+
"hero.distinct": "distinct content",
|
| 24 |
+
"hero.unsized_one": "{{n}} row has no size and is not in that number",
|
| 25 |
+
"hero.unsized_other": "{{n}} rows have no size and are not in that number",
|
| 26 |
+
"hero.commit": "Commit {{sha}}",
|
| 27 |
+
"hero.preview": "Preview data",
|
| 28 |
+
"fact.ready_sources": "sources ready",
|
| 29 |
+
"fact.answers": "units with an answer",
|
| 30 |
+
"fact.raw_rows": "raw rows",
|
| 31 |
+
"fact.sources": "sources indexed",
|
| 32 |
+
"link.dataset": "Dataset",
|
| 33 |
+
"link.readme": "Catalog README",
|
| 34 |
+
|
| 35 |
+
"foot.missing": "Missing material?",
|
| 36 |
+
"foot.ask": "Ask for a category, a form or a source.",
|
| 37 |
+
"foot.top": "Back to top",
|
| 38 |
+
"foot.index": "Index {{sha}} · {{date}}",
|
| 39 |
+
|
| 40 |
+
"notice.vocabulary": "The vocabulary could not be read ({{error}}); codes are shown as they are.",
|
| 41 |
+
"notice.failed": "Could not read the index: {{error}}",
|
| 42 |
+
"error.noCommit": "the Hub gave no commit id",
|
| 43 |
+
|
| 44 |
+
"num.unknown": "unknown",
|
| 45 |
+
"common.unknown": "Unknown",
|
| 46 |
+
"common.noneYet": "none yet",
|
| 47 |
+
"common.all": "All {{n}}",
|
| 48 |
+
"common.first": "First {{n}}",
|
| 49 |
+
"common.fewer": "Fewer",
|
| 50 |
+
"copy.button": "Copy",
|
| 51 |
+
"copy.done": "Copied",
|
| 52 |
+
"copy.select": "Select the text",
|
| 53 |
+
"chart.view": "View",
|
| 54 |
+
"chart.chart": "Chart",
|
| 55 |
+
"chart.table": "Table",
|
| 56 |
+
"heat.aria": "{{metric}} by domain and form",
|
| 57 |
+
"heat.scale": "{{metric}}, log scale",
|
| 58 |
+
"heat.cell": "{{row}}, {{col}}: {{value}}",
|
| 59 |
+
|
| 60 |
+
"n.source_one": "{{n}} source",
|
| 61 |
+
"n.source_other": "{{n}} sources",
|
| 62 |
+
"n.domain_one": "{{n}} domain",
|
| 63 |
+
"n.domain_other": "{{n}} domains",
|
| 64 |
+
"n.unit_one": "{{n}} unit",
|
| 65 |
+
"n.unit_other": "{{n}} units",
|
| 66 |
+
"n.unitFile_one": "{{n}} unit file",
|
| 67 |
+
"n.unitFile_other": "{{n}} unit files",
|
| 68 |
+
"n.companion_one": "{{n}} companion",
|
| 69 |
+
"n.companion_other": "{{n}} companions",
|
| 70 |
+
"n.answer_one": "{{n}} answer",
|
| 71 |
+
"n.answer_other": "{{n}} answers",
|
| 72 |
+
"n.file_one": "{{n}} file",
|
| 73 |
+
"n.file_other": "{{n}} files",
|
| 74 |
+
"n.row_one": "{{n}} row",
|
| 75 |
+
"n.row_other": "{{n}} rows",
|
| 76 |
+
"n.rawRows_one": "{{n}} raw row",
|
| 77 |
+
"n.rawRows_other": "{{n}} raw rows",
|
| 78 |
+
"n.rowsCompact_one": "{{n}} row",
|
| 79 |
+
"n.rowsCompact_other": "{{n}} rows",
|
| 80 |
+
|
| 81 |
+
"name.notClassified": "Not classified",
|
| 82 |
+
"name.severalDomains": "Several domains",
|
| 83 |
+
"name.otherForms": "Other forms",
|
| 84 |
+
|
| 85 |
+
"domain.D1": "Software engineering",
|
| 86 |
+
"domain.D2": "System administration",
|
| 87 |
+
"domain.D3": "Data science",
|
| 88 |
+
"domain.D4": "Machine learning",
|
| 89 |
+
"domain.D5": "Science",
|
| 90 |
+
"domain.D6": "Operations",
|
| 91 |
+
"domain.D7": "Hardware",
|
| 92 |
+
"domain.D8": "Security",
|
| 93 |
+
"domain.D9": "Media",
|
| 94 |
+
"domain.D10": "Misc",
|
| 95 |
+
"domain.D10.2": "Constraint scheduling",
|
| 96 |
+
"domain.D4.1": "Training and debugging",
|
| 97 |
+
"domain.D4.5": "Paper reproductions",
|
| 98 |
+
"domain.D3.1": "ETL and multi-source merging",
|
| 99 |
+
"domain.D3.6": "Visualisation and reports",
|
| 100 |
+
"domain.D2.5": "File systems and recovery",
|
| 101 |
+
"domain.D2.6": "Crash and performance analysis",
|
| 102 |
+
"domain.D2.7": "Virtualisation and emulators",
|
| 103 |
+
"domain.D2.8": "Large text and log editing",
|
| 104 |
+
"domain.D8.2": "Reverse engineering and binaries",
|
| 105 |
+
|
| 106 |
+
"tier.permissive": "Permissive",
|
| 107 |
+
"tier.share-alike": "Share-alike",
|
| 108 |
+
"tier.nc-research": "Non-commercial",
|
| 109 |
+
"tier.no-training": "No training",
|
| 110 |
+
"tier.internal-only": "Internal only",
|
| 111 |
+
"tier.mixed": "Mixed",
|
| 112 |
+
"tier.unknown": "Not stated",
|
| 113 |
+
"permit.yes": "Allowed",
|
| 114 |
+
"permit.no": "Not allowed",
|
| 115 |
+
"permit.conditional": "Conditional",
|
| 116 |
+
"permit.unknown": "Not stated",
|
| 117 |
+
"reason.not_english": "Not English",
|
| 118 |
+
"reason.licence_restricted": "Licence too tight",
|
| 119 |
+
"reason.access_gated": "Behind a login",
|
| 120 |
+
"reason.personal_data": "Personal data",
|
| 121 |
+
"reason.answer_not_verifiable": "Answer uncheckable",
|
| 122 |
+
"reason.answer_cells_insufficient": "Few answer cells",
|
| 123 |
+
"reason.not_in_scope": "Out of scope",
|
| 124 |
+
"reason.below_threshold": "Too little activity",
|
| 125 |
+
"reason.series_cap": "Series cap",
|
| 126 |
+
"reason.duplicate": "Duplicate",
|
| 127 |
+
"reason.other": "Other",
|
| 128 |
+
"unitKind.file": "a file",
|
| 129 |
+
"unitKind.apiResponse": "an API response",
|
| 130 |
+
"unitKind.warcRecord": "a record in a web archive",
|
| 131 |
+
"unitKind.archiveMember": "a file inside an archive",
|
| 132 |
+
"unitKind.byteRange": "a byte range of a bigger file",
|
| 133 |
+
"unitCount.file_one": "{{n}} file",
|
| 134 |
+
"unitCount.file_other": "{{n}} files",
|
| 135 |
+
"unitCount.apiResponse_one": "{{n}} API response",
|
| 136 |
+
"unitCount.apiResponse_other": "{{n}} API responses",
|
| 137 |
+
"unitCount.warcRecord_one": "{{n}} WARC record",
|
| 138 |
+
"unitCount.warcRecord_other": "{{n}} WARC records",
|
| 139 |
+
"unitCount.archiveMember_one": "{{n}} archive member",
|
| 140 |
+
"unitCount.archiveMember_other": "{{n}} archive members",
|
| 141 |
+
"unitCount.byteRange_one": "{{n}} byte range",
|
| 142 |
+
"unitCount.byteRange_other": "{{n}} byte ranges",
|
| 143 |
+
"unitCount.code_one": "{{n}} {{kind}}",
|
| 144 |
+
"unitCount.code_other": "{{n}} {{kind}}s",
|
| 145 |
+
"tool.editor": "any editor",
|
| 146 |
+
"tool.browser": "a browser",
|
| 147 |
+
"tool.suiteRunner": "the suite runner",
|
| 148 |
+
"batch.collecting": "Collecting",
|
| 149 |
+
|
| 150 |
+
"fit.ship": "Shippable licence",
|
| 151 |
+
"fit.bench": "No benchmark overlap",
|
| 152 |
+
"fit.pii": "No personal data",
|
| 153 |
+
"fit.size": "Files up to 20 MB",
|
| 154 |
+
"fit.alone": "Unit stands alone",
|
| 155 |
+
"fit.answers": "Answers or expected results",
|
| 156 |
+
"fit.login": "No login",
|
| 157 |
+
"fit.dot": "{{label}}: {{state}}",
|
| 158 |
+
"fitWord.yes": "Fits",
|
| 159 |
+
"fitWord.part": "Partly",
|
| 160 |
+
"fitWord.no": "Does not fit",
|
| 161 |
+
"fitWord.unknown": "Not known",
|
| 162 |
+
|
| 163 |
+
"material.before": "Where the",
|
| 164 |
+
"material.after": "task-ready units are",
|
| 165 |
+
"material.cats": "Most requested categories",
|
| 166 |
+
"kind.ready": "Task-ready",
|
| 167 |
+
"cat.lead.ready_one": "task-ready unit",
|
| 168 |
+
"cat.lead.ready_other": "task-ready units",
|
| 169 |
+
"cat.lead.raw_one": "raw row",
|
| 170 |
+
"cat.lead.raw_other": "raw rows",
|
| 171 |
+
"cat.lead.sources_one": "source",
|
| 172 |
+
"cat.lead.sources_other": "sources",
|
| 173 |
+
"cat.scaled": "{{scale}} {{word}}",
|
| 174 |
+
"cat.wanted": "Wanted",
|
| 175 |
+
"cat.request": "Request",
|
| 176 |
+
"cat.plusRaw_one": "+ {{n}} raw row",
|
| 177 |
+
"cat.plusRaw_other": "+ {{n}} raw rows",
|
| 178 |
+
"cat.readyTip_one": "{{ready}} of {{n}} source task-ready",
|
| 179 |
+
"cat.readyTip_other": "{{ready}} of {{n}} sources task-ready",
|
| 180 |
+
"cat.shippable": "Shippable",
|
| 181 |
+
"cat.shippableRaw": "Shippable, raw",
|
| 182 |
+
"cat.shipTipReady": "of the task-ready units may be shipped",
|
| 183 |
+
"cat.shipTipRaw": "of the raw rows may be shipped",
|
| 184 |
+
"cat.sample": "Take a sample",
|
| 185 |
+
"request.more": "More material for: {{name}}",
|
| 186 |
+
"request.like": "Like this catalog selection ({{chips}}), but: ",
|
| 187 |
+
|
| 188 |
+
"stack.otherForms_one": "Other {{n}} form",
|
| 189 |
+
"stack.otherForms_other": "Other {{n}} forms",
|
| 190 |
+
"stack.otherDomains_one": "Other {{n}} domain",
|
| 191 |
+
"stack.otherDomains_other": "Other {{n}} domains",
|
| 192 |
+
"stack.notClassified": "Not classified yet",
|
| 193 |
+
"where.title": "Largest task-ready sources",
|
| 194 |
+
"where.otherSources_one": "Other {{n}} source",
|
| 195 |
+
"where.otherSources_other": "Other {{n}} sources",
|
| 196 |
+
"ready.title": "Task-ready or raw",
|
| 197 |
+
"ready.rawCorpora": "Raw corpora",
|
| 198 |
+
"ready.answersApart": "Answers apart",
|
| 199 |
+
"ready.checksInside": "Checks inside",
|
| 200 |
+
"ready.showing": "Showing task-ready",
|
| 201 |
+
"ready.only": "Only task-ready",
|
| 202 |
+
"ready.byForm": "By form",
|
| 203 |
+
"ready.byDomain": "By domain",
|
| 204 |
+
"ready.split": "Split",
|
| 205 |
+
"ready.part": "Part",
|
| 206 |
+
"ready.kind": "Kind",
|
| 207 |
+
"ready.units": "Units",
|
| 208 |
+
"ready.rawRow_one": "Raw corpora ({{n}} row)",
|
| 209 |
+
"ready.rawRow_other": "Raw corpora ({{n}} rows)",
|
| 210 |
+
"ready.answersKept": "Answers kept apart",
|
| 211 |
+
"ready.resultsInside": "Expected results inside",
|
| 212 |
+
"th.source": "Source",
|
| 213 |
+
"th.form": "Form",
|
| 214 |
+
"th.domain": "Domain",
|
| 215 |
+
"th.readyUnits": "Task-ready units",
|
| 216 |
+
"th.batch": "Batch",
|
| 217 |
+
"th.rows": "Rows",
|
| 218 |
+
"th.changed": "Changed",
|
| 219 |
+
"th.state": "State",
|
| 220 |
+
"th.address": "Address",
|
| 221 |
+
"th.size": "Size",
|
| 222 |
+
|
| 223 |
+
"filter.title": "Filters",
|
| 224 |
+
"filter.noneOn": "none on",
|
| 225 |
+
"filter.on_one": "{{n}} on",
|
| 226 |
+
"filter.on_other": "{{n}} on",
|
| 227 |
+
"filter.all": "All",
|
| 228 |
+
"filter.any": "Any",
|
| 229 |
+
"filter.domain": "Domain",
|
| 230 |
+
"filter.leaf": "Subdomain",
|
| 231 |
+
"filter.form": "Form",
|
| 232 |
+
"filter.kind": "Kind",
|
| 233 |
+
"filter.status.ready": "Task-ready",
|
| 234 |
+
"filter.status.raw": "Raw corpus",
|
| 235 |
+
"filter.status.flagged": "Flagged",
|
| 236 |
+
"filter.training": "Training",
|
| 237 |
+
"filter.shipping": "Shipping",
|
| 238 |
+
"filter.permit.yes": "Allowed, all rows",
|
| 239 |
+
"filter.permit.cond": "Allowed or conditional",
|
| 240 |
+
"filter.tier": "Licence tier",
|
| 241 |
+
"filter.size": "Source size",
|
| 242 |
+
"filter.size.lt1g": "Under 1 GB",
|
| 243 |
+
"filter.size.1g1t": "1 GB to 1 TB",
|
| 244 |
+
"filter.size.ge1t": "1 TB or more",
|
| 245 |
+
"filter.size.unknown": "Not published",
|
| 246 |
+
"filter.search": "Search",
|
| 247 |
+
"filter.searchHint": "Title or text",
|
| 248 |
+
"filter.clear": "Clear",
|
| 249 |
+
"chip.domain": "Domain",
|
| 250 |
+
"chip.form": "Form",
|
| 251 |
+
"chip.training": "Training",
|
| 252 |
+
"chip.redistribution": "Redistribution",
|
| 253 |
+
"chip.tier": "Tier",
|
| 254 |
+
"chip.size": "Size",
|
| 255 |
+
"chip.kind": "Kind",
|
| 256 |
+
"chip.search": "Search",
|
| 257 |
+
|
| 258 |
+
"metric.label": "Count",
|
| 259 |
+
"metric.sources": "Sources",
|
| 260 |
+
"metric.rows": "Pointer rows",
|
| 261 |
+
"metric.bytes": "Size",
|
| 262 |
+
"map.rows_one": "{{n}} pointer row",
|
| 263 |
+
"map.rows_other": "{{n}} pointer rows",
|
| 264 |
+
"map.nothing": "Nothing matches",
|
| 265 |
+
"map.head": "{{amount}}, {{domains}}",
|
| 266 |
+
"map.title": "{{metric}}: domain × form",
|
| 267 |
+
"map.columns": "Columns",
|
| 268 |
+
|
| 269 |
+
"col.title": "Source",
|
| 270 |
+
"col.domain": "Category",
|
| 271 |
+
"col.form": "Form",
|
| 272 |
+
"col.units": "Units or rows",
|
| 273 |
+
"col.distinct": "Distinct size",
|
| 274 |
+
"col.tier": "Licence",
|
| 275 |
+
"col.fit": "Fit",
|
| 276 |
+
"sources.head": "{{sources}}, {{ready}} task-ready",
|
| 277 |
+
"sources.none": "No source matches",
|
| 278 |
+
"sources.ready": "Ready",
|
| 279 |
+
"sources.unitsLabel": "Units",
|
| 280 |
+
"sources.rowsLabel": "Rows",
|
| 281 |
+
"sources.unitWord_one": " unit",
|
| 282 |
+
"sources.unitWord_other": " units",
|
| 283 |
+
"sources.rawWord_one": " raw row",
|
| 284 |
+
"sources.rawWord_other": " raw rows",
|
| 285 |
+
"sources.fitKey": "Fit:",
|
| 286 |
+
"size.method": "Distinct content is the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column distinct_bytes; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. tools/fetch.py removes duplicates over your own selection when it reads it.",
|
| 287 |
+
"size.unknown": "Distinct content (the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content) is not known for this release yet.",
|
| 288 |
+
"size.note": "Distinct content: {{size}}. {{method}}",
|
| 289 |
+
"size.noteUnsized": "Distinct content: {{size}}. {{unsized}} {{method}}",
|
| 290 |
+
"size.unsized_one": "{{n}} row has no size and is not in that number.",
|
| 291 |
+
"size.unsized_other": "{{n}} rows have no size and are not in that number.",
|
| 292 |
+
|
| 293 |
+
"detail.what": "What it is",
|
| 294 |
+
"detail.notDescribed": "Not described",
|
| 295 |
+
"detail.oneUnit": "One unit",
|
| 296 |
+
"detail.taskUnit": "One task unit",
|
| 297 |
+
"detail.unitWithChecks": "One unit with its checks",
|
| 298 |
+
"detail.from": "Comes from",
|
| 299 |
+
"detail.notRecorded": "Not recorded",
|
| 300 |
+
"detail.alone": "Stands alone",
|
| 301 |
+
"detail.withCompanions": "With companions",
|
| 302 |
+
"detail.yes": "Yes",
|
| 303 |
+
"detail.eachFile": "Yes, each file",
|
| 304 |
+
"detail.container": "Needs its container file",
|
| 305 |
+
"detail.answers": "Answers",
|
| 306 |
+
"detail.answersInside": "Inside each unit (tests, expected output)",
|
| 307 |
+
"detail.answersNone": "None: raw material",
|
| 308 |
+
"detail.opensWith": "Opens with",
|
| 309 |
+
"detail.noExt": "no ext",
|
| 310 |
+
"detail.anyEditor": "Any editor",
|
| 311 |
+
"detail.licence": "Licence",
|
| 312 |
+
"detail.training": "Training",
|
| 313 |
+
"detail.shipping": "Shipping",
|
| 314 |
+
"detail.size": "Size",
|
| 315 |
+
"detail.distinct": "{{size}} distinct",
|
| 316 |
+
"detail.unsized": "{{n}} unsized",
|
| 317 |
+
"detail.median": "median {{size}}",
|
| 318 |
+
"detail.stripAria": "median {{median}}, 99% under {{p99}}",
|
| 319 |
+
"detail.example": "Example",
|
| 320 |
+
"detail.noFile": "No data file",
|
| 321 |
+
"detail.take": "Take it",
|
| 322 |
+
"detail.more": "Batches and first rows",
|
| 323 |
+
"detail.notCounted": "not counted",
|
| 324 |
+
|
| 325 |
+
"get.nothing": "Nothing to take",
|
| 326 |
+
"get.takeOne_one": "Take {{count}} unit from {{title}}",
|
| 327 |
+
"get.takeOne_other": "Take {{count}} units from {{title}}",
|
| 328 |
+
"get.takeEach_one": "Take {{count}} unit from each of {{sources}}",
|
| 329 |
+
"get.takeEach_other": "Take {{count}} units from each of {{sources}}",
|
| 330 |
+
"get.command": "Command",
|
| 331 |
+
"get.sourceWord_one": "source",
|
| 332 |
+
"get.sourceWord_other": "sources",
|
| 333 |
+
"get.wholeIndex": "Whole index",
|
| 334 |
+
"get.change": "Change",
|
| 335 |
+
"get.look": "Look first",
|
| 336 |
+
"get.take": "Take",
|
| 337 |
+
"get.sql": "DuckDB SQL, pinned to the commit",
|
| 338 |
+
"sql.head": "Agentic Env Index @ {{sha}}: {{sources}}, {{rows}}",
|
| 339 |
+
"sql.all": "all sources",
|
| 340 |
+
"sql.chip": "{{k}}: {{v}}",
|
| 341 |
+
"csv.progress_one": "{{done}}/{{n}} file",
|
| 342 |
+
"csv.progress_other": "{{done}}/{{n}} files",
|
| 343 |
+
"csv.button": "CSV · {{rows}}",
|
| 344 |
+
"csv.tooMany": "Over {{n}} rows: use the command or SQL",
|
| 345 |
+
"tree.title": "What it writes",
|
| 346 |
+
"tree.units": "units, README, LICENSE",
|
| 347 |
+
"tree.answers": "answers, kept apart",
|
| 348 |
+
"tree.manifest": "one line per file, sha256 checked",
|
| 349 |
+
"tree.sizes": "one line per source",
|
| 350 |
+
"tree.domainNote": ": such as D3_data or D3",
|
| 351 |
+
"tree.leftOut": "Left out by default",
|
| 352 |
+
"profile.duplicates": "Duplicates",
|
| 353 |
+
"profile.benchmark": "Benchmark copies",
|
| 354 |
+
"profile.licence": "No shipping licence",
|
| 355 |
+
"profile.english": "Not English",
|
| 356 |
+
"profile.personal": "Personal data",
|
| 357 |
+
"profile.large": "Over 20 MB",
|
| 358 |
+
"agents.text": "Catalog columns · the one command · the default profile · where answers land",
|
| 359 |
+
"agents.readme": "README: For agents",
|
| 360 |
+
|
| 361 |
+
"excluded.head_one": "{{n}} reviewed and left out",
|
| 362 |
+
"excluded.head_other": "{{n}} reviewed and left out",
|
| 363 |
+
"excluded.none": "Nothing left out",
|
| 364 |
+
"excluded.reason": "Reason",
|
| 365 |
+
"excluded.requirement": "Requirement",
|
| 366 |
+
"excluded.items": "Items",
|
| 367 |
+
"excluded.unit": "Unit",
|
| 368 |
+
"excluded.notMet": "Requirement not met",
|
| 369 |
+
"excluded.detail": "Detail",
|
| 370 |
+
"excluded.reviewed": "Reviewed",
|
| 371 |
+
"excluded.wholeSource": "whole source",
|
| 372 |
+
"history.title": "History: sources after each snapshot",
|
| 373 |
+
"history.start": "start",
|
| 374 |
+
"history.date": "Date",
|
| 375 |
+
"history.entry": "Entry",
|
| 376 |
+
"history.sourcesAdded": "Sources added",
|
| 377 |
+
"history.rowsAdded": "Rows added",
|
| 378 |
+
"history.started": "Log started"
|
| 379 |
+
}
|
locales/zh.json
ADDED
|
@@ -0,0 +1,343 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"lang.name": "中文",
|
| 3 |
+
"lang.switch": "以英文显示本页",
|
| 4 |
+
|
| 5 |
+
"page.title": "Agentic Env 数据目录",
|
| 6 |
+
"page.description": "来自公开指针索引 Battam/agentic-env-index 的终端智能体任务真实素材:按类别浏览,一眼看出是否适配,一条命令即可获取。",
|
| 7 |
+
|
| 8 |
+
"nav.sections": "页面分区",
|
| 9 |
+
"nav.overview": "概览",
|
| 10 |
+
"nav.browse": "浏览",
|
| 11 |
+
"nav.get": "获取数据",
|
| 12 |
+
"nav.excluded": "未收录",
|
| 13 |
+
"nav.agents": "给智能体",
|
| 14 |
+
"nav.request": "申请数据",
|
| 15 |
+
|
| 16 |
+
"hero.eyebrow": "Agentic Env 索引,按类别",
|
| 17 |
+
"hero.title": "面向智能体任务的真实素材",
|
| 18 |
+
"hero.units": "个任务就绪单元",
|
| 19 |
+
"hero.browse": "浏览类别",
|
| 20 |
+
"hero.facts": "概况",
|
| 21 |
+
"hero.indexOf": "索引版本",
|
| 22 |
+
"hero.sizeUnknown": "去重后内容大小尚未统计",
|
| 23 |
+
"hero.distinct": "去重后内容",
|
| 24 |
+
"hero.unsized_other": "{{n}} 行没有大小,未计入该数字",
|
| 25 |
+
"hero.commit": "提交 {{sha}}",
|
| 26 |
+
"hero.preview": "预览数据",
|
| 27 |
+
"fact.ready_sources": "个就绪数据源",
|
| 28 |
+
"fact.answers": "个带答案的单元",
|
| 29 |
+
"fact.raw_rows": "行原始数据",
|
| 30 |
+
"fact.sources": "个已索引数据源",
|
| 31 |
+
"link.dataset": "数据集",
|
| 32 |
+
"link.readme": "目录 README",
|
| 33 |
+
|
| 34 |
+
"foot.missing": "缺少素材?",
|
| 35 |
+
"foot.ask": "可以申请一个类别、一种形态或一个数据源。",
|
| 36 |
+
"foot.top": "回到顶部",
|
| 37 |
+
"foot.index": "索引 {{sha}} · {{date}}",
|
| 38 |
+
|
| 39 |
+
"notice.vocabulary": "无法读取词表({{error}});代码按原样显示。",
|
| 40 |
+
"notice.failed": "无法读取索引:{{error}}",
|
| 41 |
+
"error.noCommit": "Hub 未返回提交 ID",
|
| 42 |
+
|
| 43 |
+
"num.unknown": "未知",
|
| 44 |
+
"common.unknown": "未知",
|
| 45 |
+
"common.noneYet": "暂无",
|
| 46 |
+
"common.all": "全部 {{n}} 项",
|
| 47 |
+
"common.first": "前 {{n}} 项",
|
| 48 |
+
"common.fewer": "收起",
|
| 49 |
+
"copy.button": "复制",
|
| 50 |
+
"copy.done": "已复制",
|
| 51 |
+
"copy.select": "请手动选中文本",
|
| 52 |
+
"chart.view": "视图",
|
| 53 |
+
"chart.chart": "图表",
|
| 54 |
+
"chart.table": "表格",
|
| 55 |
+
"heat.aria": "按领域和形态统计的{{metric}}",
|
| 56 |
+
"heat.scale": "{{metric}},对数刻度",
|
| 57 |
+
"heat.cell": "{{row}},{{col}}:{{value}}",
|
| 58 |
+
|
| 59 |
+
"n.source_other": "{{n}} 个数据源",
|
| 60 |
+
"n.domain_other": "{{n}} 个领域",
|
| 61 |
+
"n.unit_other": "{{n}} 个单元",
|
| 62 |
+
"n.unitFile_other": "{{n}} 个单元文件",
|
| 63 |
+
"n.companion_other": "{{n}} 个配套文件",
|
| 64 |
+
"n.answer_other": "{{n}} 个答案",
|
| 65 |
+
"n.file_other": "{{n}} 个文件",
|
| 66 |
+
"n.row_other": "{{n}} 行",
|
| 67 |
+
"n.rawRows_other": "{{n}} 行原始数据",
|
| 68 |
+
"n.rowsCompact_other": "{{n}} 行",
|
| 69 |
+
|
| 70 |
+
"name.notClassified": "未分类",
|
| 71 |
+
"name.severalDomains": "多个领域",
|
| 72 |
+
"name.otherForms": "其他形态",
|
| 73 |
+
|
| 74 |
+
"domain.D1": "软件工程",
|
| 75 |
+
"domain.D2": "系统管理",
|
| 76 |
+
"domain.D3": "数据科学",
|
| 77 |
+
"domain.D4": "机器学习",
|
| 78 |
+
"domain.D5": "科学",
|
| 79 |
+
"domain.D6": "运维",
|
| 80 |
+
"domain.D7": "硬件",
|
| 81 |
+
"domain.D8": "安全",
|
| 82 |
+
"domain.D9": "媒体",
|
| 83 |
+
"domain.D10": "其他",
|
| 84 |
+
"domain.D10.2": "约束调度",
|
| 85 |
+
"domain.D4.1": "训练与调试",
|
| 86 |
+
"domain.D4.5": "论文复现",
|
| 87 |
+
"domain.D3.1": "ETL 与多源合并",
|
| 88 |
+
"domain.D3.6": "可视化与报告",
|
| 89 |
+
"domain.D2.5": "文件系统与数据恢复",
|
| 90 |
+
"domain.D2.6": "崩溃与性能分析",
|
| 91 |
+
"domain.D2.7": "虚拟化与模拟器",
|
| 92 |
+
"domain.D2.8": "大文本与日志编辑",
|
| 93 |
+
"domain.D8.2": "逆向工程与二进制",
|
| 94 |
+
|
| 95 |
+
"tier.permissive": "宽松",
|
| 96 |
+
"tier.share-alike": "相同方式共享",
|
| 97 |
+
"tier.nc-research": "非商业",
|
| 98 |
+
"tier.no-training": "禁止训练",
|
| 99 |
+
"tier.internal-only": "仅限内部",
|
| 100 |
+
"tier.mixed": "混合",
|
| 101 |
+
"tier.unknown": "未注明",
|
| 102 |
+
"permit.yes": "允许",
|
| 103 |
+
"permit.no": "不允许",
|
| 104 |
+
"permit.conditional": "有条件",
|
| 105 |
+
"permit.unknown": "未注明",
|
| 106 |
+
"reason.not_english": "非英文",
|
| 107 |
+
"reason.licence_restricted": "许可证限制过严",
|
| 108 |
+
"reason.access_gated": "需要登录",
|
| 109 |
+
"reason.personal_data": "含个人数据",
|
| 110 |
+
"reason.answer_not_verifiable": "答案无法核验",
|
| 111 |
+
"reason.answer_cells_insufficient": "答案单元格过少",
|
| 112 |
+
"reason.not_in_scope": "不在范围内",
|
| 113 |
+
"reason.below_threshold": "活跃度过低",
|
| 114 |
+
"reason.series_cap": "系列数量上限",
|
| 115 |
+
"reason.duplicate": "重复",
|
| 116 |
+
"reason.other": "其他",
|
| 117 |
+
"unitKind.file": "一个文件",
|
| 118 |
+
"unitKind.apiResponse": "一条 API 响应",
|
| 119 |
+
"unitKind.warcRecord": "Web 归档中的一条记录",
|
| 120 |
+
"unitKind.archiveMember": "压缩包内的一个文件",
|
| 121 |
+
"unitKind.byteRange": "大文件中的一段字节范围",
|
| 122 |
+
"unitCount.file_other": "{{n}} 个文件",
|
| 123 |
+
"unitCount.apiResponse_other": "{{n}} 条 API 响应",
|
| 124 |
+
"unitCount.warcRecord_other": "{{n}} 条 WARC 记录",
|
| 125 |
+
"unitCount.archiveMember_other": "{{n}} 个压缩包成员",
|
| 126 |
+
"unitCount.byteRange_other": "{{n}} 段字节范围",
|
| 127 |
+
"unitCount.code_other": "{{n}} 个 {{kind}}",
|
| 128 |
+
"tool.editor": "任意编辑器",
|
| 129 |
+
"tool.browser": "浏览器",
|
| 130 |
+
"tool.suiteRunner": "测试套件运行器",
|
| 131 |
+
"batch.collecting": "收集中",
|
| 132 |
+
|
| 133 |
+
"fit.ship": "许可证允许分发",
|
| 134 |
+
"fit.bench": "与基准无重合",
|
| 135 |
+
"fit.pii": "无个人数据",
|
| 136 |
+
"fit.size": "文件不超过 20 MB",
|
| 137 |
+
"fit.alone": "单元可独立使用",
|
| 138 |
+
"fit.answers": "有答案或预期结果",
|
| 139 |
+
"fit.login": "无需登录",
|
| 140 |
+
"fit.dot": "{{label}}:{{state}}",
|
| 141 |
+
"fitWord.yes": "适配",
|
| 142 |
+
"fitWord.part": "部分适配",
|
| 143 |
+
"fitWord.no": "不适配",
|
| 144 |
+
"fitWord.unknown": "未知",
|
| 145 |
+
|
| 146 |
+
"material.before": "这",
|
| 147 |
+
"material.after": "个任务就绪单元的分布",
|
| 148 |
+
"material.cats": "需求最多的类别",
|
| 149 |
+
"kind.ready": "任务就绪",
|
| 150 |
+
"cat.lead.ready_other": "个任务就绪单元",
|
| 151 |
+
"cat.lead.raw_other": "行原始数据",
|
| 152 |
+
"cat.lead.sources_other": "个数据源",
|
| 153 |
+
"cat.scaled": "{{scale}}{{word}}",
|
| 154 |
+
"cat.wanted": "征集中",
|
| 155 |
+
"cat.request": "申请",
|
| 156 |
+
"cat.plusRaw_other": "另有 {{n}} 行原始数据",
|
| 157 |
+
"cat.readyTip_other": "{{n}} 个数据源中有 {{ready}} 个任务就绪",
|
| 158 |
+
"cat.shippable": "可分发",
|
| 159 |
+
"cat.shippableRaw": "可分发(原始)",
|
| 160 |
+
"cat.shipTipReady": "的任务就绪单元可以分发",
|
| 161 |
+
"cat.shipTipRaw": "的原始数据行可以分发",
|
| 162 |
+
"cat.sample": "取样",
|
| 163 |
+
"request.more": "更多素材:{{name}}",
|
| 164 |
+
"request.like": "类似当前目录选择({{chips}}),但:",
|
| 165 |
+
|
| 166 |
+
"stack.otherForms_other": "其他 {{n}} 种形态",
|
| 167 |
+
"stack.otherDomains_other": "其他 {{n}} 个领域",
|
| 168 |
+
"stack.notClassified": "尚未分类",
|
| 169 |
+
"where.title": "最大的任务就绪数据源",
|
| 170 |
+
"where.otherSources_other": "其他 {{n}} 个数据源",
|
| 171 |
+
"ready.title": "任务就绪还是原始",
|
| 172 |
+
"ready.rawCorpora": "原始语料",
|
| 173 |
+
"ready.answersApart": "答案单独存放",
|
| 174 |
+
"ready.checksInside": "检查在单元内",
|
| 175 |
+
"ready.showing": "正在只看任务就绪",
|
| 176 |
+
"ready.only": "只看任务就绪",
|
| 177 |
+
"ready.byForm": "按形态",
|
| 178 |
+
"ready.byDomain": "按领域",
|
| 179 |
+
"ready.split": "划分",
|
| 180 |
+
"ready.part": "部分",
|
| 181 |
+
"ready.kind": "类型",
|
| 182 |
+
"ready.units": "单元",
|
| 183 |
+
"ready.rawRow_other": "原始语料({{n}} 行)",
|
| 184 |
+
"ready.answersKept": "答案单独存放",
|
| 185 |
+
"ready.resultsInside": "预期结果在单元内",
|
| 186 |
+
"th.source": "数据源",
|
| 187 |
+
"th.form": "形态",
|
| 188 |
+
"th.domain": "领域",
|
| 189 |
+
"th.readyUnits": "任务就绪单元",
|
| 190 |
+
"th.batch": "批次",
|
| 191 |
+
"th.rows": "行数",
|
| 192 |
+
"th.changed": "更新于",
|
| 193 |
+
"th.state": "状态",
|
| 194 |
+
"th.address": "地址",
|
| 195 |
+
"th.size": "大小",
|
| 196 |
+
|
| 197 |
+
"filter.title": "筛选",
|
| 198 |
+
"filter.noneOn": "未启用",
|
| 199 |
+
"filter.on_other": "已启用 {{n}} 项",
|
| 200 |
+
"filter.all": "全部",
|
| 201 |
+
"filter.any": "不限",
|
| 202 |
+
"filter.domain": "领域",
|
| 203 |
+
"filter.leaf": "子领域",
|
| 204 |
+
"filter.form": "形态",
|
| 205 |
+
"filter.kind": "类型",
|
| 206 |
+
"filter.status.ready": "任务就绪",
|
| 207 |
+
"filter.status.raw": "原始语料",
|
| 208 |
+
"filter.status.flagged": "已标记",
|
| 209 |
+
"filter.training": "训练",
|
| 210 |
+
"filter.shipping": "分发",
|
| 211 |
+
"filter.permit.yes": "允许,所有行",
|
| 212 |
+
"filter.permit.cond": "允许或有条件",
|
| 213 |
+
"filter.tier": "许可证等级",
|
| 214 |
+
"filter.size": "数据源大小",
|
| 215 |
+
"filter.size.lt1g": "小于 1 GB",
|
| 216 |
+
"filter.size.1g1t": "1 GB 到 1 TB",
|
| 217 |
+
"filter.size.ge1t": "1 TB 及以上",
|
| 218 |
+
"filter.size.unknown": "未公布",
|
| 219 |
+
"filter.search": "搜索",
|
| 220 |
+
"filter.searchHint": "标题或正文",
|
| 221 |
+
"filter.clear": "清除",
|
| 222 |
+
"chip.domain": "领域",
|
| 223 |
+
"chip.form": "形态",
|
| 224 |
+
"chip.training": "训练",
|
| 225 |
+
"chip.redistribution": "再分发",
|
| 226 |
+
"chip.tier": "等级",
|
| 227 |
+
"chip.size": "大小",
|
| 228 |
+
"chip.kind": "类型",
|
| 229 |
+
"chip.search": "搜索",
|
| 230 |
+
|
| 231 |
+
"metric.label": "统计",
|
| 232 |
+
"metric.sources": "数据源",
|
| 233 |
+
"metric.rows": "指针行",
|
| 234 |
+
"metric.bytes": "大小",
|
| 235 |
+
"map.rows_other": "{{n}} 条指针行",
|
| 236 |
+
"map.nothing": "没有匹配项",
|
| 237 |
+
"map.head": "{{amount}},{{domains}}",
|
| 238 |
+
"map.title": "{{metric}}:领域 × 形态",
|
| 239 |
+
"map.columns": "列",
|
| 240 |
+
|
| 241 |
+
"col.title": "数据源",
|
| 242 |
+
"col.domain": "类别",
|
| 243 |
+
"col.form": "形态",
|
| 244 |
+
"col.units": "单元或行数",
|
| 245 |
+
"col.distinct": "去重后大小",
|
| 246 |
+
"col.tier": "许可证",
|
| 247 |
+
"col.fit": "适配",
|
| 248 |
+
"sources.head": "{{sources}},其中 {{ready}} 个任务就绪",
|
| 249 |
+
"sources.none": "没有匹配的数据源",
|
| 250 |
+
"sources.ready": "就绪",
|
| 251 |
+
"sources.unitsLabel": "单元",
|
| 252 |
+
"sources.rowsLabel": "行数",
|
| 253 |
+
"sources.unitWord_other": " 个单元",
|
| 254 |
+
"sources.rawWord_other": " 行原始数据",
|
| 255 |
+
"sources.fitKey": "适配:",
|
| 256 |
+
"size.method": "去重后内容是各行 size_bytes 之和,减去与其他行内容重复的行的大小(在同一哈希范围内依次比较 sha256、sha1、md5 加大小;每份内容只保留一份,归入注册顺序中的第一个数据源),每次发布时统计,按数据源记在目录列 distinct_bytes 中;若同一数据源还整体列出了某个压缩包,其成员的哈希与压缩包不同,因此不会被减去。tools/fetch.py 读取时会在你自己的选择范围内去重。",
|
| 257 |
+
"size.unknown": "本次发布的去重后内容(各行 size_bytes 之和减去与其他行内容重复的行的大小)尚未统计。",
|
| 258 |
+
"size.note": "去重后内容:{{size}}。{{method}}",
|
| 259 |
+
"size.noteUnsized": "去重后内容:{{size}}。{{unsized}}{{method}}",
|
| 260 |
+
"size.unsized_other": "{{n}} 行没有大小,未计入该数字。",
|
| 261 |
+
|
| 262 |
+
"detail.what": "简介",
|
| 263 |
+
"detail.notDescribed": "暂无描述",
|
| 264 |
+
"detail.oneUnit": "一个单元",
|
| 265 |
+
"detail.taskUnit": "一个任务单元",
|
| 266 |
+
"detail.unitWithChecks": "一个带检查的单元",
|
| 267 |
+
"detail.from": "来源",
|
| 268 |
+
"detail.notRecorded": "未记录",
|
| 269 |
+
"detail.alone": "可独立使用",
|
| 270 |
+
"detail.withCompanions": "需配套文件",
|
| 271 |
+
"detail.yes": "是",
|
| 272 |
+
"detail.eachFile": "是,每个文件都可以",
|
| 273 |
+
"detail.container": "需要其所在的容器文件",
|
| 274 |
+
"detail.answers": "答案",
|
| 275 |
+
"detail.answersInside": "在每个单元内(测试、预期输出)",
|
| 276 |
+
"detail.answersNone": "无:原始素材",
|
| 277 |
+
"detail.opensWith": "打开方式",
|
| 278 |
+
"detail.noExt": "无扩展名",
|
| 279 |
+
"detail.anyEditor": "任意编辑器",
|
| 280 |
+
"detail.licence": "许可证",
|
| 281 |
+
"detail.training": "训练",
|
| 282 |
+
"detail.shipping": "分发",
|
| 283 |
+
"detail.size": "大小",
|
| 284 |
+
"detail.distinct": "去重后 {{size}}",
|
| 285 |
+
"detail.unsized": "{{n}} 行无大小",
|
| 286 |
+
"detail.median": "中位数 {{size}}",
|
| 287 |
+
"detail.stripAria": "中位数 {{median}},99% 小于 {{p99}}",
|
| 288 |
+
"detail.example": "示例",
|
| 289 |
+
"detail.noFile": "无数据文件",
|
| 290 |
+
"detail.take": "获取",
|
| 291 |
+
"detail.more": "批次与前几行",
|
| 292 |
+
"detail.notCounted": "未统计",
|
| 293 |
+
|
| 294 |
+
"get.nothing": "没有可获取的内容",
|
| 295 |
+
"get.takeOne_other": "从 {{title}} 获取 {{count}} 个单元",
|
| 296 |
+
"get.takeEach_other": "从 {{sources}}中各获取 {{count}} 个单元",
|
| 297 |
+
"get.command": "命令",
|
| 298 |
+
"get.sourceWord_other": "个数据源",
|
| 299 |
+
"get.wholeIndex": "整个索引",
|
| 300 |
+
"get.change": "修改",
|
| 301 |
+
"get.look": "先预览",
|
| 302 |
+
"get.take": "获取",
|
| 303 |
+
"get.sql": "DuckDB SQL,固定到该提交",
|
| 304 |
+
"sql.head": "Agentic Env 索引 @ {{sha}}:{{sources}},{{rows}}",
|
| 305 |
+
"sql.all": "全部数据源",
|
| 306 |
+
"sql.chip": "{{k}}:{{v}}",
|
| 307 |
+
"csv.progress_other": "{{done}}/{{n}} 个文件",
|
| 308 |
+
"csv.button": "CSV · {{rows}}",
|
| 309 |
+
"csv.tooMany": "超过 {{n}} 行:请使用命令或 SQL",
|
| 310 |
+
"tree.title": "写入内容",
|
| 311 |
+
"tree.units": "单元、README、LICENSE",
|
| 312 |
+
"tree.answers": "答案,单独存放",
|
| 313 |
+
"tree.manifest": "每个文件一行,已校验 sha256",
|
| 314 |
+
"tree.sizes": "每个数据源一行",
|
| 315 |
+
"tree.domainNote": ":例如 D3_data 或 D3",
|
| 316 |
+
"tree.leftOut": "默认排除",
|
| 317 |
+
"profile.duplicates": "重复内容",
|
| 318 |
+
"profile.benchmark": "基准副本",
|
| 319 |
+
"profile.licence": "许可证不允许分发",
|
| 320 |
+
"profile.english": "非英文",
|
| 321 |
+
"profile.personal": "个人数据",
|
| 322 |
+
"profile.large": "超过 20 MB",
|
| 323 |
+
"agents.text": "目录字段 · 一条命令 · 默认配置 · 答案存放位置",
|
| 324 |
+
"agents.readme": "README:给智能体",
|
| 325 |
+
|
| 326 |
+
"excluded.head_other": "已审核并排除 {{n}} 项",
|
| 327 |
+
"excluded.none": "没有排除项",
|
| 328 |
+
"excluded.reason": "原因",
|
| 329 |
+
"excluded.requirement": "要求",
|
| 330 |
+
"excluded.items": "项数",
|
| 331 |
+
"excluded.unit": "单元",
|
| 332 |
+
"excluded.notMet": "未满足的要求",
|
| 333 |
+
"excluded.detail": "详情",
|
| 334 |
+
"excluded.reviewed": "审核日期",
|
| 335 |
+
"excluded.wholeSource": "整个数据源",
|
| 336 |
+
"history.title": "历史:每次快照后的数据源数",
|
| 337 |
+
"history.start": "起点",
|
| 338 |
+
"history.date": "日期",
|
| 339 |
+
"history.entry": "条目",
|
| 340 |
+
"history.sourcesAdded": "新增数据源",
|
| 341 |
+
"history.rowsAdded": "新增行数",
|
| 342 |
+
"history.started": "日志开始"
|
| 343 |
+
}
|
portal.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
-
"commit": "
|
| 3 |
-
"digest": "
|
| 4 |
-
"files":
|
| 5 |
"format": "portal-deploy/1",
|
| 6 |
-
"index": "
|
| 7 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"commit": "2da961963825eb361a239d6c04fb0a0b48af73b1",
|
| 3 |
+
"digest": "4e64fccdd34a52a2ad12f75da2950a09c332c6acc32c3fe2706eb47f22ff476e",
|
| 4 |
+
"files": 12,
|
| 5 |
"format": "portal-deploy/1",
|
| 6 |
+
"index": "02b52f1fb8ec44ec91c6593f759faff0be42b200"
|
| 7 |
}
|
style.css
CHANGED
|
@@ -36,6 +36,8 @@
|
|
| 36 |
--shadow: 0 1px 2px rgba(17, 37, 50, 0.06), 0 8px 24px -12px rgba(17, 37, 50, 0.18);
|
| 37 |
--shadow-lift: 0 2px 4px rgba(17, 37, 50, 0.08), 0 18px 40px -16px rgba(17, 37, 50, 0.28);
|
| 38 |
--sans: "Inter", "Noto Sans SC", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
|
|
|
|
|
|
|
| 39 |
--mono: "IBM Plex Mono", ui-monospace, "Cascadia Mono", Consolas, monospace;
|
| 40 |
/* type */
|
| 41 |
--fs-small: 13px;
|
|
@@ -53,6 +55,7 @@
|
|
| 53 |
--t: 160ms;
|
| 54 |
}
|
| 55 |
@media (prefers-reduced-motion: reduce) { :root { --t: 0ms; } }
|
|
|
|
| 56 |
|
| 57 |
* { box-sizing: border-box; }
|
| 58 |
[hidden] { display: none !important; }
|
|
@@ -121,6 +124,7 @@ button.link { border: 0; background: none; color: var(--link); text-decoration:
|
|
| 121 |
.nav-links { display: flex; align-items: center; gap: var(--s2); font-size: 14px; }
|
| 122 |
.nav-links a:not(.btn) { display: inline-flex; align-items: center; gap: 6px; color: var(--ink-2); text-decoration: none; }
|
| 123 |
.nav-links a:not(.btn):hover { color: var(--link); text-decoration: underline; }
|
|
|
|
| 124 |
|
| 125 |
/* ---- skeletons: shapes, never words ---- */
|
| 126 |
.sk { display: block; border-radius: 8px; background: linear-gradient(90deg, #e3eaef 0%, #f3f7f9 50%, #e3eaef 100%); background-size: 200% 100%; }
|
|
@@ -531,8 +535,9 @@ details.long[open] > summary { margin-bottom: var(--s1); }
|
|
| 531 |
@media (max-width: 720px) {
|
| 532 |
:root { --fs-small: 12px; --fs-body: 15px; --fs-sub: 18px; --fs-h2: 26px; --fs-h1: 28px; --nav-h: 56px; --r: 14px; }
|
| 533 |
.nav-row { grid-template-columns: auto minmax(0, 1fr); gap: var(--s2); }
|
| 534 |
-
|
| 535 |
-
.nav-
|
|
|
|
| 536 |
.progress { gap: 6px; }
|
| 537 |
.progress a { font-size: 11px; column-gap: 3px; }
|
| 538 |
.hero-grid { padding-top: var(--s5); gap: var(--s4); }
|
|
|
|
| 36 |
--shadow: 0 1px 2px rgba(17, 37, 50, 0.06), 0 8px 24px -12px rgba(17, 37, 50, 0.18);
|
| 37 |
--shadow-lift: 0 2px 4px rgba(17, 37, 50, 0.08), 0 18px 40px -16px rgba(17, 37, 50, 0.28);
|
| 38 |
--sans: "Inter", "Noto Sans SC", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
|
| 39 |
+
/* Chinese text (the page in Chinese, or the language switch while it names Chinese): Noto Sans SC first */
|
| 40 |
+
--sans-zh: "Noto Sans SC", "Inter", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
|
| 41 |
--mono: "IBM Plex Mono", ui-monospace, "Cascadia Mono", Consolas, monospace;
|
| 42 |
/* type */
|
| 43 |
--fs-small: 13px;
|
|
|
|
| 55 |
--t: 160ms;
|
| 56 |
}
|
| 57 |
@media (prefers-reduced-motion: reduce) { :root { --t: 0ms; } }
|
| 58 |
+
html:lang(zh) { --sans: var(--sans-zh); }
|
| 59 |
|
| 60 |
* { box-sizing: border-box; }
|
| 61 |
[hidden] { display: none !important; }
|
|
|
|
| 124 |
.nav-links { display: flex; align-items: center; gap: var(--s2); font-size: 14px; }
|
| 125 |
.nav-links a:not(.btn) { display: inline-flex; align-items: center; gap: 6px; color: var(--ink-2); text-decoration: none; }
|
| 126 |
.nav-links a:not(.btn):hover { color: var(--link); text-decoration: underline; }
|
| 127 |
+
.lang-btn:lang(zh) { font-family: var(--sans-zh); }
|
| 128 |
|
| 129 |
/* ---- skeletons: shapes, never words ---- */
|
| 130 |
.sk { display: block; border-radius: 8px; background: linear-gradient(90deg, #e3eaef 0%, #f3f7f9 50%, #e3eaef 100%); background-size: 200% 100%; }
|
|
|
|
| 535 |
@media (max-width: 720px) {
|
| 536 |
:root { --fs-small: 12px; --fs-body: 15px; --fs-sub: 18px; --fs-h2: 26px; --fs-h1: 28px; --nav-h: 56px; --r: 14px; }
|
| 537 |
.nav-row { grid-template-columns: auto minmax(0, 1fr); gap: var(--s2); }
|
| 538 |
+
/* on a phone the nav keeps the progress bar and the language switch */
|
| 539 |
+
.brand, .nav-links > :not(.lang-btn) { display: none; }
|
| 540 |
+
.nav-row { grid-template-columns: minmax(0, 1fr) auto; }
|
| 541 |
.progress { gap: 6px; }
|
| 542 |
.progress a { font-size: 11px; column-gap: 3px; }
|
| 543 |
.hero-grid { padding-top: var(--s5); gap: var(--s4); }
|