data-line commited on
Commit
fcd1156
·
1 Parent(s): 442062d

portal: catalog page of repository commit 2da961963825

Browse files

Files of catalog/portal/ (12), digest 4e64fccdd34a52a2ad12f75da2950a09c332c6acc32c3fe2706eb47f22ff476e.

Files changed (13) hide show
  1. README.md +28 -1
  2. app.js +267 -245
  3. charts.js +39 -27
  4. data.js +5 -3
  5. first.js +12 -13
  6. first.json +7 -7
  7. i18n.js +67 -0
  8. index.html +72 -68
  9. llms.txt +2 -2
  10. locales/en.json +379 -0
  11. locales/zh.json +343 -0
  12. portal.json +4 -4
  13. style.css +7 -2
README.md CHANGED
@@ -84,12 +84,36 @@ the code. Maps from code to name, and nested `children`, are read too.
84
  | `vocabulary=<url>` | read the names of domains and forms from this URL instead of `meta/vocabulary.json` |
85
  | `domain`, `leaf`, `form`, `train`, `redist`, `tier`, `size`, `status`, `q` | the filters (`train` and `redist`: `yes` for every row allowed, `cond` for allowed or with conditions; `status` is the kind: `ready`, `raw` or `flagged`) |
86
  | `metric`, `sort`, `dir`, `open` | the map's count, the sort of the source list (default: fit), the open source |
 
87
 
88
  The filters scope the map, the source list and "Get the data"; the overview, the history and "Not included" describe the
89
  whole index. The licence filters keep a source only when every one of its rows has that permission
90
  (`meta/sources.json`, `license_rows`), so the counts, the SQL and the CSV agree without a per-row licence condition.
91
  "Source size" is the total known size of a source (referenced).
92
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
  ## Files
94
 
95
  - `index.html`, `style.css`: the frame, in layers: a tinted canvas, a deep slate hero band with the one big number,
@@ -104,6 +128,9 @@ whole index. The licence filters keep a source only when every one of its rows h
104
  the same index commit they are the same numbers; when main has moved on, the changed ones are briefly highlighted.
105
  With `?rev=`, `?catalog=` or `?vocabulary=` the hero waits for the index (`first.json` describes main).
106
  - `data.js`: reading the commit (hyparquet for the parquet files, byte ranges for data files).
 
 
 
107
  - `charts.js`: thin bars, columns, the stacked split, the heatmap, the tooltip, the chart and table toggle, and `vt`
108
  (views and tabs switch with the View Transitions API where there is one and motion is welcome).
109
  - `icons.js`: the pictograms (inline SVG, one stroke style) for the forms, the domains, the facts and the four parts.
@@ -129,7 +156,7 @@ Fonts: Inter (400 to 800) for text and numbers, Noto Sans SC for CJK, IBM Plex M
129
  Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
130
 
131
  Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
132
- are zstd-compressed). There is no build step. To look at it locally: `python -m http.server` in this folder (without a
133
  `first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
134
  writes one).
135
 
 
84
  | `vocabulary=<url>` | read the names of domains and forms from this URL instead of `meta/vocabulary.json` |
85
  | `domain`, `leaf`, `form`, `train`, `redist`, `tier`, `size`, `status`, `q` | the filters (`train` and `redist`: `yes` for every row allowed, `cond` for allowed or with conditions; `status` is the kind: `ready`, `raw` or `flagged`) |
86
  | `metric`, `sort`, `dir`, `open` | the map's count, the sort of the source list (default: fit), the open source |
87
+ | `lng=<en or zh>` | open the page in English or in Simplified Chinese (and keep that choice) |
88
 
89
  The filters scope the map, the source list and "Get the data"; the overview, the history and "Not included" describe the
90
  whole index. The licence filters keep a source only when every one of its rows has that permission
91
  (`meta/sources.json`, `license_rows`), so the counts, the SQL and the CSV agree without a per-row licence condition.
92
  "Source size" is the total known size of a source (referenced).
93
 
94
+ ## Languages
95
+
96
+ The page writes its own words in English or in Simplified Chinese, with i18next and its browser language detector.
97
+ Every word the page itself writes comes from i18next's `t` by key: headings, labels, buttons, notes, tooltips, `title`
98
+ and `aria-label` attributes, table headers, chart titles, axes and legends, the first screen, the empty, loading and
99
+ error states, the document title and the meta description, and the labels the page maps codes to (licence tiers,
100
+ reasons for leaving out, kinds of unit, the fallback names of domains).
101
+ The English words are in `locales/en.json`, the Chinese ones in `locales/zh.json`, with the same keys (flat keys, since
102
+ a key may hold a code with dots such as `domain.D10.2`).
103
+ Counts use i18next's plurals (`_one` and `_other` in English; Chinese has only `_other`);
104
+ numbers and dates are formatted with `Intl.NumberFormat` and `Intl.DateTimeFormat` in the current language.
105
+ What the page reads from the dataset or from `first.json` stays as it is in both languages: source names, titles,
106
+ summaries, descriptions, licence names, statements and notes, and the names of domains and forms from the vocabulary.
107
+ The issue a "Request" link opens is prefilled in English, the language of the hub.
108
+
109
+ The switch is one button in the header that names the other language in that language.
110
+ It changes the language without a reload and sets `<html lang>` (`en` or `zh-CN`; Chinese text then uses Noto Sans SC
111
+ first, `:lang(zh)` in `style.css`).
112
+ The detector reads the query string first (`?lng=zh`), then localStorage, and keeps the choice in localStorage
113
+ (`i18nextLng`), so the page opens in English unless the reader chose Chinese;
114
+ the switch drops a `lng` from the address, so the kept choice holds on the next visit.
115
+ `llms.txt` and this README stay in English.
116
+
117
  ## Files
118
 
119
  - `index.html`, `style.css`: the frame, in layers: a tinted canvas, a deep slate hero band with the one big number,
 
128
  the same index commit they are the same numbers; when main has moved on, the changed ones are briefly highlighted.
129
  With `?rev=`, `?catalog=` or `?vocabulary=` the hero waits for the index (`first.json` describes main).
130
  - `data.js`: reading the commit (hyparquet for the parquet files, byte ranges for data files).
131
+ - `i18n.js`: the page's languages (i18next with its browser language detector, the words from `locales/en.json` and
132
+ `locales/zh.json`), the words of `index.html` (`data-i18n` sets an element's text, `data-i18n-attr` its attributes)
133
+ and the language switch; the other modules draw their text again on i18next's `languageChanged`.
134
  - `charts.js`: thin bars, columns, the stacked split, the heatmap, the tooltip, the chart and table toggle, and `vt`
135
  (views and tabs switch with the View Transitions API where there is one and motion is welcome).
136
  - `icons.js`: the pictograms (inline SVG, one stroke style) for the forms, the domains, the facts and the four parts.
 
156
  Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
157
 
158
  Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
159
+ are zstd-compressed), `i18next@26.4.5` and `i18next-browser-languagedetector@8.2.1`. There is no build step. To look at it locally: `python -m http.server` in this folder (without a
160
  `first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
161
  writes one).
162
 
app.js CHANGED
@@ -2,24 +2,16 @@
2
  // at a glance, take it with one command. Graphics first; every chart keeps a short label and a table view.
3
  import { DATASET_URL, fileUrl, loadBatchDates, loadIndex, num, parquetRows, readColumns, REPO, topOf } from './data.js';
4
  import {
5
- clear, columns, el, figureWithToggle, fmtBytes, fmtCompact, fmtInt, fmtPct, fmtShort, heatmap, hbars, hideTip, meter,
6
- segmented, stack, table, vt, withTip,
7
  } from './charts.js';
8
  import { decorate, drawHero, markFromIndex, setNumber } from './first.js';
 
9
  import { domainIcon, formIcon, icon } from './icons.js';
10
 
11
- // The categories the group asks for most, shown first. Names are fallbacks for when the vocabulary has none.
12
- const MOST_REQUESTED = [
13
- ['D10.2', 'Constraint scheduling'], ['D4.1', 'Training and debugging'], ['D4.5', 'Paper reproductions'],
14
- ['D3.1', 'ETL and multi-source merging'], ['D3.6', 'Visualisation and reports'], ['D2.5', 'File systems and recovery'],
15
- ['D2.6', 'Crash and performance analysis'], ['D2.7', 'Virtualisation and emulators'],
16
- ['D2.8', 'Large text and log editing'], ['D8.2', 'Reverse engineering and binaries'],
17
- ];
18
- // names of the top-level domains until meta/vocabulary.json is published
19
- const TOP_NAMES = {
20
- D1: 'Software engineering', D2: 'System administration', D3: 'Data science', D4: 'Machine learning', D5: 'Science',
21
- D6: 'Operations', D7: 'Hardware', D8: 'Security', D9: 'Media', D10: 'Misc',
22
- };
23
  const GITHUB = 'https://github.com/msra-sg/agentic-env-pipeline';
24
  const CSV_MAX_ROWS = 200000;
25
  const ROW_COLUMNS = ['source', 'batch', 'uri', 'range_offset', 'range_length', 'member_path', 'size_bytes', 'sha256'];
@@ -31,45 +23,41 @@ const MAX_FILE = 20e6;
31
  const MAX_LISTED = 30;
32
  const $ = (id) => document.getElementById(id);
33
 
34
- // ---- plain names (codes only inside the SQL and the commands)
35
- const TIER = {
36
- permissive: 'Permissive', 'share-alike': 'Share-alike', 'nc-research': 'Non-commercial',
37
- 'no-training': 'No training', 'internal-only': 'Internal only', mixed: 'Mixed', unknown: 'Not stated',
38
- };
39
- const PERMIT = { yes: 'Allowed', no: 'Not allowed', conditional: 'Conditional', unknown: 'Not stated' };
40
  const PERMIT_ORDER = ['yes', 'no', 'conditional', 'unknown'];
41
  const PERMIT_COLOR = { yes: 'var(--ours)', no: 'var(--problem)', conditional: 'var(--fill-3)', unknown: 'var(--other)' };
42
- const CATEGORY = {
43
- not_english: 'Not English', licence_restricted: 'Licence too tight', access_gated: 'Behind a login',
44
- personal_data: 'Personal data', answer_not_verifiable: 'Answer uncheckable', answer_cells_insufficient: 'Few answer cells',
45
- not_in_scope: 'Out of scope', below_threshold: 'Too little activity', series_cap: 'Series cap', duplicate: 'Duplicate',
46
- other: 'Other',
47
- };
48
- const UNIT_KIND = { file: 'a file', 'API response': 'an API response', 'WARC record': 'a record in a web archive', 'archive member': 'a file inside an archive', 'byte range': 'a byte range of a bigger file' };
49
- // file type -> the tool that opens it
50
  const TOOLS = {
51
  xlsx: 'openpyxl', xlsm: 'openpyxl', xls: 'LibreOffice', ods: 'LibreOffice', csv: 'DuckDB', tsv: 'DuckDB', parquet: 'DuckDB',
52
  json: 'jq', jsonl: 'jq', ndjson: 'jq', pdf: 'pdftotext', doc: 'LibreOffice', docx: 'python-docx', ppt: 'LibreOffice',
53
  pptx: 'python-pptx', gz: 'gzip', tgz: 'tar', tar: 'tar', bz2: 'bzip2', xz: 'xz', zst: 'zstd', zip: 'unzip', whl: 'unzip',
54
  jar: 'unzip', '7z': '7-Zip', crate: 'tar', deb: 'dpkg-deb', rpm: 'rpm2cpio', warc: 'warcio', arc: 'warcio', pgn: 'python-chess',
55
- py: 'Python', md: 'any editor', txt: 'any editor', '(none)': 'file(1)', html: 'a browser', htm: 'a browser', xml: 'xmllint',
56
- yaml: 'yq', yml: 'yq', toml: 'any editor', evtx: 'python-evtx', log: 'grep', rego: 'OPA', sql: 'DuckDB', db: 'SQLite',
57
  safetensors: 'safetensors', final: 'TensorBoard', ipynb: 'Jupyter', eml: 'Python email', mbox: 'Python mailbox',
58
  ics: 'icalendar', e01: 'The Sleuth Kit', img: 'The Sleuth Kit', dd: 'The Sleuth Kit', raw: 'The Sleuth Kit',
59
  ys: 'Yosys', v: 'Icarus Verilog', sv: 'Verilator', go: 'Go', rs: 'Cargo', zig: 'Zig', lua: 'Lua', rb: 'Ruby', c: 'gcc',
60
- tcl: 'tclsh', test: 'the suite runner', smt2: 'Z3', mzn: 'MiniZinc', dzn: 'MiniZinc', lp: 'HiGHS', mps: 'HiGHS',
61
  };
 
 
62
  const NONE = '_none';
63
  const MULTI = '_multi';
64
  const OTHER = '_other';
65
  const human = (code) => String(code).replace(/[_-]+/g, ' ').replace(/^./, (c) => c.toUpperCase());
66
  function sentence(s) {
67
- let t = String(s).trim();
68
- if (!/\s/.test(t)) t = t.replace(/[-_]+/g, ' ');
69
- t = t.replace(/\b(ci|ctf|etl|ml|qa|sql|api)\b/gi, (w) => (w.toLowerCase() === 'qa' ? 'Q&A' : w.toUpperCase()));
70
- return t.charAt(0).toUpperCase() + t.slice(1);
71
  }
72
- const plural = (n, w) => `${fmtInt(n)} ${w}${n === 1 ? '' : 's'}`;
 
73
  const sum = (xs, f) => xs.reduce((a, x) => a + f(x), 0);
74
 
75
  let IX = null;
@@ -99,15 +87,14 @@ const vocabName = (sec, code) => {
99
  const n = IX.vocab[sec].names.get(code);
100
  return n ? sentence(n) : null;
101
  };
102
- function domainName(key) {
103
- if (key === NONE) return 'Not classified';
104
- if (key === MULTI) return vocabName('domain', 'multi') || 'Several domains';
105
- const fallback = TOP_NAMES[key] || MOST_REQUESTED.find(([c]) => c === key)?.[1];
106
- return vocabName('domain', key) || fallback || key;
107
  }
108
- function formName(key) {
109
- if (key === NONE || key == null) return 'Not classified';
110
- if (key === OTHER) return 'Other forms';
111
  return vocabName('form', key) || sentence(key);
112
  }
113
  const domainKeysOf = (s) => (s.domainKind === 'codes' ? s.tops : [s.domainKind === 'multi' ? MULTI : NONE]);
@@ -121,17 +108,18 @@ function sourceDomainText(s) {
121
  }
122
  const inCategory = (s, code) => (code.includes('.') ? s.codes.includes(code) : s.tops.includes(code));
123
 
124
- // ---- fit: what decides whether a source can feed a task (yes, part, no, unknown)
125
  const FIT = [
126
- { k: 'ship', label: 'Shippable licence', icon: 'share' },
127
- { k: 'bench', label: 'No benchmark overlap', icon: 'target' },
128
- { k: 'pii', label: 'No personal data', icon: 'person' },
129
- { k: 'size', label: 'Files up to 20 MB', icon: 'size' },
130
- { k: 'alone', label: 'Unit stands alone', icon: 'doc' },
131
- { k: 'answers', label: 'Answers or expected results', icon: 'check' },
132
- { k: 'login', label: 'No login', icon: 'link' },
133
  ];
134
- const FIT_WORD = { yes: 'Fits', part: 'Partly', no: 'Does not fit', unknown: 'Not known' };
 
135
  function fitOf(s) {
136
  const q = s.info.size_quantiles_upper_bytes || {};
137
  const c = s.contamination?.level;
@@ -149,14 +137,14 @@ function fitOf(s) {
149
  };
150
  }
151
  function fitDot(state, label) {
152
- return el('span', { class: `fit fit-${state}`, role: 'img', 'aria-label': `${label}: ${FIT_WORD[state]}` });
153
  }
154
  function fitStrip(s) {
155
  const f = fitOf(s);
156
- return el('span', { class: 'fit-strip' }, FIT.map((x) => withTip(fitDot(f[x.k], x.label), FIT_WORD[f[x.k]], x.label)));
157
  }
158
  function fitKey() {
159
- return el('span', { class: 'fit-key' }, ['yes', 'part', 'no', 'unknown'].map((k) => el('span', {}, el('span', { class: `fit fit-${k}`, 'aria-hidden': 'true' }), FIT_WORD[k])));
160
  }
161
 
162
  // ---- filters
@@ -193,22 +181,23 @@ function passes(s, skip = {}) {
193
  const selected = () => IX.sources.filter((s) => passes(s));
194
  const FILTER_KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q'];
195
  const activeFilters = () => FILTER_KEYS.filter((k) => F[k]);
196
- function filterChips() {
 
197
  const out = [];
198
- const opt = (id) => $(id).selectedOptions[0]?.textContent || '';
199
- if (F.domain) out.push(['Domain', F.leaf ? domainName(F.leaf) : domainName(F.domain)]);
200
- if (F.form) out.push(['Form', formName(F.form)]);
201
- if (F.train) out.push(['Training', opt('f-train')]);
202
- if (F.redist) out.push(['Redistribution', opt('f-redist')]);
203
- if (F.tier) out.push(['Tier', TIER[F.tier] || human(F.tier)]);
204
- if (F.size) out.push(['Size', opt('f-size')]);
205
- if (F.status) out.push(['Kind', opt('f-status')]);
206
- if (F.q) out.push(['Search', F.q]);
207
  return out;
208
  }
209
  function metricOf(s, m = F.metric) { return m === 'sources' ? 1 : m === 'rows' ? s.rows : s.distinct || 0; }
210
  const metricFmt = (m = F.metric) => (m === 'bytes' ? fmtBytes : (v) => fmtInt(v));
211
- const METRIC_WORD = { sources: 'Sources', rows: 'Pointer rows', bytes: 'Size' };
212
 
213
  function setFilter(patch) {
214
  Object.assign(F, patch);
@@ -241,7 +230,7 @@ function selectionArgs(sel) {
241
  return { args: commandArgs(sel, { domain, form: F.form }), listed: false };
242
  }
243
  async function copy(text, state) {
244
- try { await navigator.clipboard.writeText(text); state.textContent = 'Copied'; } catch { state.textContent = 'Select the text'; }
245
  setTimeout(() => { state.textContent = ''; }, 2400);
246
  }
247
  function copyButton(label, text, { dark = false, iconName = 'copy' } = {}) {
@@ -265,24 +254,18 @@ function catStats(code) {
265
  return { sources: S.length, readySources: S.filter((s) => s.isTask).length, ready, raw, onReady, ship, shipOf: onReady ? ready : raw };
266
  }
267
  const bySize = (a, b) => b.st.ready - a.st.ready || b.st.raw - a.st.raw || b.st.sources - a.st.sources;
268
- function requestUrl(name) {
269
  const req = new URL(`${GITHUB}/issues/new`);
270
  req.searchParams.set('template', 'data_request.yml');
271
- req.searchParams.set('what', `More material for: ${name}`);
272
  return req.toString();
273
  }
274
- /** 8970000 -> ['8.97', 'million']; small numbers whole. */
275
- function bigParts(v) {
276
- const t = fmtCompact(v);
277
- const m = t.match(/^([\d.,]+) (thousand|million|billion)$/);
278
- return m ? [m[1], m[2]] : [fmtInt(v), ''];
279
- }
280
  /** The largest true number of a category: task-ready units, raw rows or sources. */
281
  function leadOf(st) {
282
  const c = [
283
- { v: st.ready, k: 'ready', word: st.ready === 1 ? 'task-ready unit' : 'task-ready units' },
284
- { v: st.raw, k: 'raw', word: 'raw rows' },
285
- { v: st.sources, k: 'sources', word: st.sources === 1 ? 'source' : 'sources' },
286
  ].filter((x) => x.v > 0);
287
  return c.sort((x, y) => y.v - x.v)[0];
288
  }
@@ -297,35 +280,35 @@ function catCard({ code, st }) {
297
  el('div', { class: 'cat-main' },
298
  el('span', { class: 'cat-name', title: name }, name),
299
  el('span', { class: 'wanted-art', 'aria-hidden': 'true' }, domainIcon(code, { size: 44 })),
300
- el('span', { class: 'cat-wanted' }, icon('target', { size: 16 }), 'Wanted')),
301
- el('div', { class: 'cat-foot' }, el('a', { class: 'btn', href: requestUrl(name) }, icon('request', { size: 16 }), 'Request')));
302
  }
303
  const cur = F.leaf === code;
304
  const pick = () => setFilter(cur ? { domain: '', leaf: '' } : { domain: topOf(code), leaf: code });
305
  const lead = leadOf(st);
306
- const [n, scale] = bigParts(lead.v);
307
  const num = el('b', { class: 'cat-num' });
308
  setNumber(num, `cat:${code}`, lead.v, () => n);
309
  const meta = [];
310
- if (lead.k !== 'sources') meta.push(el('span', {}, plural(st.sources, 'source')));
311
- if (lead.k !== 'raw' && st.raw) meta.push(el('span', { class: 'muted' }, `+ ${fmtCompact(st.raw)} raw rows`));
312
- const readyText = st.ready ? `${fmtInt(st.ready)} ${st.ready === 1 ? 'unit' : 'units'}` : 'none yet';
313
  return el('li', { class: `card cat lift${cur ? ' current' : ''}` },
314
  el('button', { type: 'button', class: 'cat-main', 'aria-pressed': String(cur), onclick: pick },
315
  el('span', { class: 'cat-head' }, el('span', { class: 'cat-ico' }, domainIcon(code, { size: 22 })),
316
  cur ? el('span', { class: 'cat-check' }, icon('check', { size: 18 })) : null),
317
  el('span', { class: 'cat-name', title: name }, name),
318
- el('span', { class: 'cat-lead' }, num, el('span', { class: 'cat-num-label' }, scale ? `${scale} ${lead.word}` : lead.word)),
319
  el('span', { class: 'cat-meta' }, meta),
320
  el('span', { class: 'cat-bars' },
321
- smallBar('Task-ready', st.sources ? st.readySources / st.sources : 0, readyText,
322
- `${fmtInt(st.readySources)} of ${plural(st.sources, 'source')} task-ready`),
323
- smallBar(st.onReady ? 'Shippable' : 'Shippable, raw', st.shipOf ? st.ship / st.shipOf : 0, fmtPct(st.ship, st.shipOf),
324
- st.onReady ? 'of the task-ready units may be shipped' : 'of the raw rows may be shipped'))),
325
- el('div', { class: 'cat-foot' }, copyButton('Take a sample', command(commandArgs(null, { domain: code })))));
326
  }
327
  function renderCategories() {
328
- const cats = MOST_REQUESTED.map(([code]) => ({ code, st: catStats(code) }));
329
  const have = cats.filter((c) => c.st.sources).sort(bySize);
330
  const wanted = cats.filter((c) => !c.st.sources);
331
  clear($('req-cards')).append(...have.map(catCard), ...wanted.map(catCard));
@@ -338,15 +321,15 @@ function renderCategories() {
338
  const STACK_NAMED = 3;
339
  // parts ranked by size, largest darkest (one ordinal ramp of the slate blue); not classified yet is gray stripes
340
  const STEPS = ['var(--ours)', '#6f8797', '#8ea2ae', '#a8b6c0'];
341
- function capParts(entries, otherWord) {
342
  const named = entries.filter((e) => e.key !== NONE && e.value > 0).sort((x, y) => y.value - x.value);
343
  const none = entries.find((e) => e.key === NONE && e.value > 0);
344
  const head = named.length > STACK_NAMED + 1 ? named.slice(0, STACK_NAMED) : named;
345
  const rest = named.slice(head.length);
346
  const parts = head.map((e, i) => ({ ...e, color: STEPS[i] }));
347
- if (rest.length) parts.push({ key: OTHER, label: `Other ${plural(rest.length, otherWord)}`, value: sum(rest, (e) => e.value), sources: sum(rest, (e) => e.sources), color: STEPS[3] });
348
- if (none) parts.push({ ...none, label: 'Not classified yet', color: 'var(--unclassified)' });
349
- return parts.map((p) => ({ ...p, sub: `${fmtInt(p.value)} units` }));
350
  }
351
  function groupReady(T, keyOf) {
352
  const m = new Map();
@@ -373,11 +356,12 @@ function renderWhere() {
373
  tipLabel: [s.title, formName(formKeyOf(s))], onClick: () => openSource(s.id) }));
374
  if (rest.length) {
375
  const v = sum(rest, (s) => s.readyUnits);
376
- items.push({ key: OTHER, label: `Other ${plural(rest.length, 'source')}`, value: v, text: fmtInt(v), tipLabel: `Other ${plural(rest.length, 'source')}` });
 
377
  }
378
- figureWithToggle($('where-src'), 'where-src', 'Largest task-ready sources', {
379
- chart: () => (items.length ? hbars(items) : el('p', { class: 'small muted' }, 'none yet')),
380
- table: () => table(['Source', 'Form', { label: 'Task-ready units', r: true }],
381
  ranked.map((s) => [s.title, formName(formKeyOf(s)), fmtInt(s.readyUnits)])),
382
  });
383
  }
@@ -393,46 +377,50 @@ function renderReady() {
393
  const inside = ready - answered;
394
  const raw = sum(R, (s) => s.rawRows);
395
  const parts = [
396
- { value: T.length, color: 'var(--ours)', label: 'Task-ready', sub: `${plural(T.length, 'source')} · ${fmtInt(ready)} units` },
397
- { value: R.length, color: 'var(--other-light)', label: 'Raw corpora', sub: `${plural(R.length, 'source')} · ${fmtCompact(raw)} rows` },
398
  ];
399
  const inner = [
400
- { value: answered, color: 'var(--ours)', label: 'Answers apart', sub: `${fmtInt(answered)} units` },
401
- { value: inside, color: 'var(--fill-3)', label: 'Checks inside', sub: `${fmtInt(inside)} units` },
402
  ];
403
  const forms = groupReady(T, formKeyOf);
404
  const doms = groupReady(T, oneDomain);
405
- const formParts = capParts(forms.map((e) => ({ ...e, label: formName(e.key) })), 'form');
406
- const domParts = capParts(doms.map((e) => ({ ...e, label: domainName(e.key) })), 'domain');
407
  const bar = (ps, twoRows = false) => el('div', { class: 'split' },
408
  el('div', { class: 'stack', role: 'img', 'aria-label': ps.map((p) => `${p.label} ${p.sub}`).join(', ') },
409
  ps.filter((p) => p.value > 0).map((p) => withTip(el('span', { tabindex: '0', style: { flex: `${p.value} 1 0`, background: p.color } }), p.sub, p.label))),
410
  el('div', { class: `split-labels fixed${twoRows ? ' two-rows' : ''}` }, ps.map((p) => el('span', { title: `${p.label}: ${p.sub}` },
411
  el('span', { class: 'sw', style: { background: p.color } }), el('b', {}, p.label), el('span', { class: 'small' }, p.sub)))));
412
  const only = el('button', { type: 'button', class: 'btn small-btn', 'aria-pressed': String(F.status === 'ready'),
413
- onclick: () => setFilter({ status: F.status === 'ready' ? '' : 'ready' }) }, icon('pick', { size: 16 }), F.status === 'ready' ? 'Showing task-ready' : 'Only task-ready');
414
  const mini = (ico, text, first) => el('p', { class: `mini-title${first ? ' first' : ''}` }, icon(ico, { size: 16 }), text);
415
  const rows = (list, kind) => [...list].sort((x, y) => y.value - x.value).map((e) => [kind, e.label ?? '', fmtInt(e.sources), fmtInt(e.value)]);
416
- figureWithToggle($('ready-card'), 'ready', 'Task-ready or raw', {
417
- chart: () => el('div', {}, mini('sources', 'Sources', true), bar(parts), mini('check', 'Task-ready units', false), bar(inner),
418
- mini('stack', 'By form', false), bar(formParts, true), mini('browse', 'By domain', false), bar(domParts, true)),
419
- table: () => table(['Split', 'Part', { label: 'Sources', r: true }, { label: 'Task-ready units', r: true }], [
420
- ['Kind', 'Task-ready', fmtInt(T.length), fmtInt(ready)],
421
- ['Kind', `Raw corpora (${fmtInt(raw)} rows)`, fmtInt(R.length), ''],
422
- ['Units', 'Answers kept apart', fmtInt(T.filter((s) => s.answered).length), fmtInt(answered)],
423
- ['Units', 'Expected results inside', fmtInt(T.filter((s) => !s.answered).length), fmtInt(inside)],
424
- ...rows(forms.map((e) => ({ ...e, label: formName(e.key) })), 'Form'),
425
- ...rows(doms.map((e) => ({ ...e, label: domainName(e.key) })), 'Domain'),
426
  ]),
427
  }, () => el('div', { class: 'card-foot' }, only));
428
  }
429
- // ---- filter row
430
  function option(value, text) { return el('option', { value }, text); }
 
 
 
 
 
 
 
431
  function fillFilterRow() {
432
- clear($('f-domain')).append(option('', 'All'), domainRows().map((k) => option(k, domainName(k))));
433
- clear($('f-form')).append(option('', 'All'), formsByCount().map((k) => option(k, formName(k))));
434
- const tiers = [...new Set(IX.sources.map((s) => s.tier))].sort((a, b) => (TIER[a] || a).localeCompare(TIER[b] || b));
435
- clear($('f-tier')).append(option('', 'All'), tiers.map((t) => option(t, TIER[t] || human(t))));
436
  const on = (id, key) => $(id).addEventListener('change', (e) => {
437
  const patch = { [key]: e.target.value };
438
  if (key === 'domain') patch.leaf = '';
@@ -440,8 +428,8 @@ function fillFilterRow() {
440
  });
441
  on('f-domain', 'domain'); on('f-leaf', 'leaf'); on('f-form', 'form'); on('f-train', 'train'); on('f-redist', 'redist');
442
  on('f-tier', 'tier'); on('f-size', 'size'); on('f-status', 'status');
443
- let t = null;
444
- $('f-q').addEventListener('input', (e) => { clearTimeout(t); t = setTimeout(() => setFilter({ q: e.target.value.trim() }), 200); });
445
  $('f-clear').addEventListener('click', () => setFilter(Object.fromEntries(FILTER_KEYS.map((k) => [k, '']))));
446
  $('filter-toggle').addEventListener('click', () => {
447
  const open = !$('filters').classList.contains('open');
@@ -456,13 +444,13 @@ function syncFilterRow(sel) {
456
  if (document.activeElement !== $('f-q')) $('f-q').value = F.q;
457
  const leaf = $('f-leaf');
458
  const leaves = F.domain && F.domain !== NONE && F.domain !== MULTI ? leafKeys(F.domain) : [];
459
- clear(leaf).append(option('', 'All'), leaves.map((k) => option(k, domainName(k))));
460
  leaf.disabled = !leaves.length;
461
  leaf.value = F.leaf;
462
  const n = activeFilters().length;
463
  $('f-clear').disabled = !n;
464
- $('filter-state').textContent = n ? `${n} on · ${plural(sel.length, 'source')}` : 'none on';
465
- $('f-count').textContent = plural(sel.length, 'source');
466
  }
467
 
468
  // ---- axes of the map
@@ -525,28 +513,28 @@ function renderMap() {
525
  // gray row and column of the map, never the headline)
526
  const total = sum(base, (s) => metricOf(s, m));
527
  const covered = new Set(base.flatMap((s) => s.tops)).size;
528
- const amount = m === 'sources' ? plural(total, 'source') : m === 'rows' ? `${fmtCompact(total)} pointer rows` : fmtBytes(total);
529
- $('map-h').textContent = !base.length ? 'Nothing matches' : covered ? `${amount}, ${plural(covered, 'domain')}` : amount;
530
  for (const b of $('metric-seg').querySelectorAll('button')) b.setAttribute('aria-pressed', String(b.dataset.metric === m));
531
 
532
  const current = F.domain && F.form ? { row: F.domain, col: F.form } : null;
533
  const rowObjs = rows.map((k) => ({ key: k, label: domainName(k), muted: k === NONE }));
534
  const colObjs = cols.map((k) => ({ key: k, label: formName(k), head: formIcon(k, { size: 20 }), muted: k === NONE }));
535
- figureWithToggle($('heat-fig'), 'heat', `${METRIC_WORD[m]}: domain × form`, {
536
  chart: () => {
537
  const h = heatmap({
538
  rows: rowObjs, cols: colObjs, value: (r, c) => cell.get(`${r.key}|${c.key}`) || 0, fmt, current,
539
- metricName: METRIC_WORD[m],
540
  onPick: (r, c) => {
541
  const form = c.key === OTHER ? '' : c.key;
542
  if (F.domain === r.key && F.form === form) setFilter({ domain: '', leaf: '', form: '' });
543
  else setFilter({ domain: r.key, leaf: '', form });
544
  },
545
  });
546
- const key = el('ul', { class: 'icon-key', 'aria-label': 'Columns' }, colObjs.map((c) => el('li', {}, formIcon(c.key, { size: 16 }), c.label)));
547
  return el('div', { class: 'heat-body' }, h.grid, key, h.legend);
548
  },
549
- table: () => table(['Domain', 'Form', { label: METRIC_WORD[m], r: true }],
550
  rowObjs.flatMap((r) => colObjs.map((c) => [r, c, cell.get(`${r.key}|${c.key}`) || 0])).filter((x) => x[2] > 0)
551
  .map(([r, c, v]) => [r.label, c.label, fmt(v)])),
552
  });
@@ -573,9 +561,9 @@ function renderMap() {
573
  }
574
  }
575
  const domMax = Math.max(0, ...rows.map((r) => domVal.get(r)));
576
- figureWithToggle($('dom-fig'), 'dom', 'By domain', {
577
  chart: () => hbars(items, { max: domMax }),
578
- table: () => table(['Domain', { label: METRIC_WORD[m], r: true }], items.map((d) => [d.leaf ? ` ${d.label}` : d.label, d.text])),
579
  });
580
 
581
  const formBase = IX.sources.filter((s) => passes(s, { form: true }));
@@ -585,22 +573,22 @@ function renderMap() {
585
  const fShown = UI.allForms ? fAll : fAll.slice(0, FORM_BARS);
586
  const fItems = fShown.map((c) => ({ key: c, label: formName(c), icon: formIcon(c, { size: 16 }), value: fv.get(c) || 0, text: fmt(fv.get(c) || 0),
587
  current: F.form === c, muted: c === NONE, tipLabel: formName(c), onClick: () => setFilter({ form: F.form === c ? '' : c }) }));
588
- figureWithToggle($('form-fig'), 'form', 'By form', {
589
  chart: () => hbars(fItems),
590
- table: () => table(['Form', { label: METRIC_WORD[m], r: true }], fAll.map((c) => [formName(c), fmt(fv.get(c) || 0)])),
591
  }, () => (fAll.length > FORM_BARS ? el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.allForms),
592
- onclick: () => { UI.allForms = !UI.allForms; renderMap(); } }, UI.allForms ? 'Fewer' : `All ${fAll.length}`) : null));
593
  }
594
 
595
- // ---- 4. sources
596
  const COLS = [
597
- { key: 'title', label: 'Source' },
598
- { key: 'domain', label: 'Category' },
599
- { key: 'form', label: 'Form' },
600
- { key: 'units', label: 'Units or rows', r: true },
601
- { key: 'distinct', label: 'Distinct size', r: true },
602
- { key: 'tier', label: 'Licence' },
603
- { key: 'fit', label: 'Fit' },
604
  ];
605
  function fitScore(s) { const f = fitOf(s); return sum(FIT, (x) => ({ yes: 2, part: 1, unknown: 0.5, no: 0 })[f[x.k]]); }
606
  function sortValue(s, k) {
@@ -611,28 +599,24 @@ function sortValue(s, k) {
611
  // task-ready units above raw rows (never compared as one count): a raw corpus has fewer than 1e12 rows
612
  case 'units': return s.isTask ? 1e12 + s.readyUnits : s.rawRows;
613
  case 'distinct': return s.distinct ?? -1;
614
- case 'tier': return (TIER[s.tier] || s.tier).toLowerCase();
615
  case 'fit': return fitScore(s);
616
  default: return 0;
617
  }
618
  }
619
- const distinctText = (s) => (s.distinct == null ? 'Unknown' : fmtBytes(s.distinct));
620
- // #364: the same words as the index README, section Size (artpool/projection_templates/text.json, size_text)
621
- const SIZE_METHOD = "Distinct content is the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content "
622
- + '(the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the '
623
- + 'first source by registry order), counted at every release, per source in the catalog column distinct_bytes; archive '
624
- + 'members whose archive the same source also lists whole have other hashes, so they are not subtracted. tools/fetch.py '
625
- + 'removes duplicates over your own selection when it reads it.';
626
  /** Distinct content and rows without a size of the sources S: the rule of index_projection.size_totals. */
627
  function sizeTotals(S) {
628
  const known = S.filter((s) => s.distinct != null);
629
  return { distinct: known.length ? sum(known, (s) => s.distinct) : null, unsized: sum(S, (s) => s.rowsNoSize) };
630
  }
631
  function sizeNote(S) {
632
- const t = sizeTotals(S);
633
- if (t.distinct == null) return "Distinct content (the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content) is not known for this release yet.";
634
- const unsized = t.unsized ? ` ${fmtInt(t.unsized)} rows have no size and are not in that number.` : '';
635
- return `Distinct content: ${fmtBytes(t.distinct)}.${unsized} ${SIZE_METHOD}`;
636
  }
637
 
638
  function renderSources(sel) {
@@ -643,7 +627,7 @@ function renderSources(sel) {
643
  return (x < y ? -1 : x > y ? 1 : 0) * dir || a.title.localeCompare(b.title);
644
  });
645
  const nReady = sel.filter((s) => s.isTask).length;
646
- $('sources-h').textContent = sel.length ? `${plural(sel.length, 'source')}, ${fmtInt(nReady)} task-ready` : 'No source matches';
647
  const body = clear($('sources-body'));
648
  if (!sel.length) return;
649
  const shown = UI.showAll ? list : list.slice(0, FIRST_ROWS);
@@ -655,7 +639,7 @@ function renderSources(sel) {
655
  el('button', { type: 'button', onclick: () => {
656
  const d = on ? (F.dir === 'asc' ? 'desc' : 'asc') : (c.r || c.key === 'fit' ? 'desc' : 'asc');
657
  F.sort = c.key; F.dir = d; writeUrl(); renderSources(selected());
658
- } }, c.label, on ? (F.dir === 'asc' ? ' ↑' : ' ↓') : ''));
659
  }));
660
  const tbody = el('tbody');
661
  for (const s of shown) {
@@ -665,23 +649,23 @@ function renderSources(sel) {
665
  el('td', { class: 'title' },
666
  el('button', { type: 'button', 'aria-expanded': String(open), 'aria-controls': `d-${s.id}`, onclick: toggle },
667
  el('span', { class: 'caret', 'aria-hidden': 'true' }, open ? '▾' : '▸'), s.title),
668
- s.isTask ? el('span', { class: 'ready-tag' }, icon('check', { size: 14 }), 'Ready') : null),
669
- el('td', { 'data-label': 'Category', class: 'cat-cell' }, sourceDomainText(s)),
670
- el('td', { 'data-label': 'Form', class: 'form-cell' }, formIcon(formKeyOf(s), { size: 16 }), formName(formKeyOf(s))),
671
- el('td', { class: 'r num', 'data-label': s.isTask ? 'Units' : 'Rows' }, s.isTask
672
- ? [fmtInt(s.readyUnits), el('span', { class: 'unit-word' }, s.readyUnits === 1 ? ' unit' : ' units')]
673
- : el('span', { class: 'muted' }, fmtCompact(s.rawRows), el('span', { class: 'unit-word' }, ' raw rows'))),
674
- el('td', { class: 'r num', 'data-label': 'Distinct size' }, distinctText(s)),
675
- el('td', { 'data-label': 'Licence' }, TIER[s.tier] || human(s.tier)),
676
  el('td', { class: 'fit-cell' }, fitStrip(s))));
677
  if (open) tbody.append(el('tr', { class: 'detail-row' }, el('td', { colspan: String(COLS.length), id: `d-${s.id}` }, detail(s))));
678
  }
679
- body.append(el('div', { class: 'list-key' }, el('span', { class: 'small' }, 'Fit:'), FIT.map((x) => el('span', { class: 'fit-name' }, icon(x.icon, { size: 14 }), x.label)), fitKey()));
680
- body.append(el('table', { class: 'src-table' }, el('caption', { class: 'visually-hidden' }, 'Sources'), el('thead', {}, head), tbody));
681
  body.append(el('p', { class: 'small size-note' }, sizeNote(sel)));
682
  if (list.length > FIRST_ROWS) {
683
  body.append(el('p', { class: 'more' }, el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.showAll),
684
- onclick: () => { UI.showAll = !UI.showAll; renderSources(selected()); } }, UI.showAll ? `First ${FIRST_ROWS}` : `All ${fmtInt(list.length)}`)));
685
  }
686
  }
687
 
@@ -707,12 +691,12 @@ function sizeStrip(q) {
707
  const lo = Math.log10(Math.min(1e3, v('p10')));
708
  const hi = Math.log10(Math.max(v('p99'), v('p90') * 10, MAX_FILE * 2));
709
  const at = (x) => `${(100 * (Math.log10(x) - lo)) / (hi - lo)}%`;
710
- return el('div', { class: 'strip', role: 'img', 'aria-label': `median ${fmtBytes(v('p50'))}, 99% under ${fmtBytes(v('p99'))}` },
711
  el('span', { class: 'strip-axis' }),
712
  el('span', { class: 'strip-band', style: { left: at(v('p10')), width: `calc(${at(v('p90'))} - ${at(v('p10'))})` } }),
713
  el('span', { class: 'strip-limit', style: { left: at(MAX_FILE) } }),
714
  el('span', { class: 'strip-dot', style: { left: at(v('p50')) } }),
715
- el('span', { class: 'strip-lab', style: { left: at(v('p50')) } }, `median ${fmtBytes(v('p50'))}`),
716
  el('span', { class: 'strip-lab limit', style: { left: at(MAX_FILE) } }, '20 MB'));
717
  }
718
  function item(iconName, label, ...value) {
@@ -743,48 +727,55 @@ function detail(s) {
743
 
744
  // fit at a glance
745
  box.append(el('ul', { class: 'fit-row' }, FIT.map((x) => el('li', { class: `fit-item fit-${fit[x.k]}-item` },
746
- icon(x.icon, { size: 18 }), el('span', {}, x.label), fitDot(fit[x.k], x.label)))));
747
 
748
  const dl = el('dl', { class: 'd-list' });
749
  // 1 what it is
750
  const text = s.summary || s.description;
751
- dl.append(item('doc', 'What it is', text ? (text.length > 160 ? el('details', {}, el('summary', {}, firstSentence(text)), el('p', {}, text)) : text) : 'Not described'));
752
  // 2 what one unit is
 
 
 
 
753
  const unitWhat = s.isTask
754
- ? [el('span', {}, s.answered ? 'One task unit' : 'One unit with its checks'), el('span', { class: 'chips' },
755
- el('span', { class: 'chip' }, plural(s.roles.unit, 'unit file')), s.roles.companion ? el('span', { class: 'chip' }, plural(s.roles.companion, 'companion')) : null,
756
- s.roles.answer ? el('span', { class: 'chip' }, plural(s.roles.answer, 'answer')) : null)]
757
- : [el('span', {}, sentence(s.kinds.map((k) => UNIT_KIND[k] || k).join(', or '))), el('span', { class: 'chips' }, s.kinds.map((k) => el('span', { class: 'chip' }, `${fmtCompact(num(s.info.unit[k]))} ${k}${num(s.info.unit[k]) === 1 ? '' : 's'}`)))];
758
- dl.append(item('rows', 'One unit', ...unitWhat));
 
759
  // 3 where it comes from
760
  const origins = [...new Set(metas.map((m) => m.pool_source?.origin).filter(Boolean))];
761
  const hosts = [...new Set(rows.map((r) => hostOf(r.uri)).filter(Boolean))];
762
  const from = origins.length ? origins : hosts;
763
- dl.append(item('pin', 'Comes from', st.sample === 'loading' ? '…' : from.length ? el('span', { class: 'chips' }, from.slice(0, 3).map((h) => el('span', { class: 'chip mono-chip' }, h))) : 'Not recorded'));
764
  // 4 self-contained or companions
765
  const comps = [...new Set(rows.filter((r, i) => metas[i].role === 'companion').map((r) => r.uri.split('/').pop()))];
766
- dl.append(item('layers', 'Stands alone', s.isTask
767
- ? (s.roles.companion ? [el('span', {}, 'With companions'), el('span', { class: 'chips' }, (comps.length ? comps : ['LICENSE', 'README.md']).map((c) => el('span', { class: 'chip mono-chip' }, c)))] : 'Yes')
768
- : fit.alone === 'yes' ? 'Yes, each file' : 'Needs its container file'));
769
  // 5 where the answers are
770
- dl.append(item('check', 'Answers', s.answered ? el('span', { class: 'mono-chip chip' }, `answer/${s.id}/ · ${plural(s.answered, 'file')}`)
771
- : s.isTask ? 'Inside each unit (tests, expected output)' : el('span', { class: 'muted' }, 'None: raw material')));
772
  // 6 tools to open it
773
  const exts = Object.entries(info.by_ext || {}).map(([k, v]) => [k, num(v.rows) || 0]).sort((a, b) => b[1] - a[1]).map(([k]) => k);
774
  const tools = [];
775
- for (const e of exts) { if (TOOLS[e] && !tools.some((t) => t[1] === TOOLS[e])) tools.push([e, TOOLS[e]]); if (tools.length >= 3) break; }
776
- dl.append(item('tool', 'Opens with', tools.length ? el('span', { class: 'chips' }, tools.map(([e, t]) => el('span', { class: 'chip' }, el('span', { class: 'mono' }, e === '(none)' ? 'no ext' : `.${e}`), ` ${t}`))) : 'Any editor'));
777
  // 7 licence
778
  const lr = info.license_rows || {};
779
- const permit = (field) => PERMIT_ORDER.map((k) => ({ k, value: num(lr[field]?.[k]?.rows) || 0, color: PERMIT_COLOR[k], label: PERMIT[k] }));
780
- dl.append(item('training', 'Licence', el('span', { class: 'lic' }, el('b', {}, TIER[s.tier] || human(s.tier)), el('span', { class: 'small' }, lic.name || s.spdx || '')),
781
  el('span', { class: 'lic-bars' },
782
- el('span', { class: 'mini-title' }, 'Training'), stack(permit('training_ok'), s.rows),
783
- el('span', { class: 'mini-title' }, 'Shipping'), stack(permit('redistribution_ok'), s.rows))));
784
  // 8 size and units
785
  const q = info.size_quantiles_upper_bytes || {};
786
- dl.append(item('size', 'Size', el('span', { class: 'lic' }, el('b', {}, `${distinctText(s)} distinct`), el('span', { class: 'small' },
787
- `${s.isTask ? `${plural(s.readyUnits, 'unit')} · ${plural(s.rows, 'file')}` : `${fmtInt(s.rawRows)} raw rows`}${s.rowsNoSize ? ` · ${fmtCompact(s.rowsNoSize)} unsized` : ''}`)),
 
 
788
  q.p50 ? sizeStrip(q) : null));
789
  // 9 one example file
790
  let ex = null;
@@ -796,21 +787,22 @@ function detail(s) {
796
  ex = el('span', { class: 'example' }, href ? el('a', { href, rel: 'noopener', class: 'mono' }, name) : el('span', { class: 'mono' }, name),
797
  r.size_bytes != null ? el('span', { class: 'small' }, ` · ${fmtBytes(num(r.size_bytes))}`) : null);
798
  }
799
- dl.append(item('link', 'Example', st.sample === 'loading' ? '…' : st.sample?.error ? el('span', { class: 'problem-text' }, st.sample.error) : ex || 'No data file'));
800
  // 10 the command that takes this source
801
  const cmd = command(commandArgs(null, { sources: [s.id] }));
802
- dl.append(item('run', 'Take it', el('code', { class: 'cmd-inline' }, cmd), copyButton('Copy', cmd)));
803
  box.append(dl);
804
 
805
  // batches and the rows read, on request
806
  const b = st.batches;
807
- const more = el('details', { class: 'd-more' }, el('summary', {}, 'Batches and first rows'));
808
  if (Array.isArray(b)) {
809
- more.append(table(['Batch', { label: 'Rows', r: true }, 'Changed', 'State'],
810
- b.map((x) => [x.batch, x.rows == null ? 'not counted' : fmtInt(x.rows), x.date ? x.date.slice(0, 10) : '', x.status ? human(x.status) : ''])));
 
811
  }
812
  if (rows.length) {
813
- more.append(el('div', { class: 'table-wrap sample' }, table(['Address', { label: 'Size', r: true }, 'sha256'],
814
  rows.map((r) => [r.member_path ? `${r.uri} ! ${r.member_path}` : r.range_offset != null ? `${r.uri} @ ${num(r.range_offset)}+${num(r.range_length)}` : r.uri,
815
  r.size_bytes == null ? '' : fmtInt(num(r.size_bytes)), hex(r.sha256).slice(0, 16) + (r.sha256 ? '…' : '')]), st.file)));
816
  }
@@ -844,8 +836,8 @@ function selectionSql(sel) {
844
  const paths = all ? [`${root}/data/*/*/*.parquet`] : sel.map((s) => `${root}/data/${s.id}/*/*.parquet`);
845
  const chips = filterChips();
846
  return [
847
- `-- Agentic Env Index @ ${IX.sha.slice(0, 7)}: ${plural(sel.length, 'source')}, ${fmtInt(sum(sel, (s) => s.rows))} rows`,
848
- chips.length ? `-- ${chips.map(([k, v]) => `${k}: ${v}`).join('; ')}` : '-- all sources',
849
  `SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
850
  'FROM read_parquet([',
851
  paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
@@ -884,50 +876,51 @@ function folderTree(sel) {
884
  const src = first ? first.id : '<source>';
885
  const line = (depth, iconName, name, note) => el('li', { style: { paddingLeft: `${depth * 20}px` } }, icon(iconName, { size: 16 }),
886
  el('span', { class: 'tree-name' }, el('span', { class: 'mono' }, name), note ? el('span', { class: 'small' }, note) : null));
887
- return el('figure', { class: 'card tree-fig' }, el('p', { class: 'chart-title' }, 'What it writes'),
888
  el('ul', { class: 'tree' },
889
  line(0, 'folder', 'data/'),
890
- line(1, 'folder', `<domain>/${src}/`, 'units, README, LICENSE'),
891
- line(1, 'folder', `answer/${src}/`, 'answers, kept apart'),
892
  line(1, 'folder', '_meta/'),
893
- line(2, 'rows', 'manifest.jsonl', 'one line per file, sha256 checked'),
894
- line(2, 'rows', 'sizes.tsv', 'one line per source')),
895
- el('p', { class: 'small tree-note' }, el('span', { class: 'mono' }, '<domain>'), ': such as D3_data or D3'),
896
- el('p', { class: 'chart-title profile-title' }, 'Left out by default'),
897
- el('ul', { class: 'chips profile' }, ['Duplicates', 'Benchmark copies', 'No shipping licence', 'Not English', 'Personal data', 'Over 20 MB']
898
- .map((t) => el('li', { class: 'chip' }, t))),
899
  el('p', { class: 'small agent-links' }, icon('robot', { size: 16 }),
900
- el('a', { href: `${DATASET_URL}#for-agents` }, 'README: For agents'), ' · ', el('a', { href: 'llms.txt' }, 'llms.txt')));
901
  }
902
  function renderGet(sel) {
903
  const ready = sum(sel, (s) => s.readyUnits);
904
  const raw = sum(sel, (s) => s.rawRows);
905
- $('get-h').textContent = !sel.length ? 'Nothing to take' : sel.length === 1 ? `Take ${SAMPLE} units from ${sel[0].title}` : `Take ${SAMPLE} units from each of ${plural(sel.length, 'source')}`;
 
906
  const body = clear($('get-body'));
907
  if (!sel.length) return;
908
  const chips = filterChips();
909
- const tabs = segmented('', [['cmd', 'Command'], ['sql', 'SQL'], ['csv', 'CSV']], UI.tab, (v) => { UI.tab = v; vt(() => renderGet(selected())); });
910
  const pickRow = el('div', { class: 'pick-row' },
911
- el('span', { class: 'fact-line' }, icon('sources', { size: 18 }), el('b', {}, fmtInt(sel.length)), sel.length === 1 ? 'source' : 'sources'),
912
- el('span', { class: 'fact-line' }, icon('check', { size: 18 }), el('b', {}, fmtInt(ready)), ready === 1 ? 'task-ready unit' : 'task-ready units'),
913
- raw ? el('span', { class: 'fact-line muted' }, icon('rows', { size: 18 }), el('b', {}, fmtCompact(raw)), 'raw rows') : null,
914
- el('span', { class: 'chips' }, chips.length ? chips.map(([k, v]) => el('span', { class: 'chip', title: k }, v)) : el('span', { class: 'chip' }, 'Whole index')),
915
- el('a', { class: 'small', href: '#filters' }, 'Change'), tabs);
916
  body.append(pickRow);
917
  const codeCard = (iconName, title, code, n) => el('div', { class: 'code-card' },
918
  el('div', { class: 'code-head' }, el('span', { class: 'code-ico' }, icon(iconName, { size: 20 })),
919
  n ? el('span', { class: 'step-n' }, String(n)) : null, el('span', { class: 'code-title' }, title),
920
- copyButton('Copy', code, { dark: true })),
921
  el('pre', { class: 'code' }, el('code', {}, code)));
922
  if (UI.tab === 'cmd') {
923
  const sa = selectionArgs(sel);
924
  const dry = command(sa.args, true);
925
  const take = command(sa.args, false);
926
  body.append(el('div', { class: 'get-grid' },
927
- el('div', { class: 'cmd-steps' }, codeCard('search', 'Look first', dry, 1), codeCard('run', 'Take', take, 2)),
928
  folderTree(sel)));
929
  } else if (UI.tab === 'sql') {
930
- body.append(codeCard('database', 'DuckDB SQL, pinned to the commit', selectionSql(sel)));
931
  } else {
932
  const rows = sum(sel, (s) => s.rows);
933
  const key = sel.map((s) => s.id).join(',');
@@ -935,18 +928,18 @@ function renderGet(sel) {
935
  const tooMany = rows > CSV_MAX_ROWS;
936
  const btn = el('button', { type: 'button', class: 'btn solid', disabled: tooMany, onclick: async () => {
937
  btn.disabled = true;
938
- const say = (t) => { csvState.textContent = t; UI.csv = { key, text: t }; };
939
  try {
940
- const csv = await buildCsv(sel, (d, n) => say(`${d}/${n} files`));
941
  const blob = new Blob([csv], { type: 'text/csv' });
942
  const a = el('a', { href: URL.createObjectURL(blob), download: `agentic-env-index-${IX.sha.slice(0, 7)}-${sel.length === 1 ? sel[0].id : `${sel.length}-sources`}.csv` });
943
  document.body.append(a); a.click(); a.remove();
944
  setTimeout(() => URL.revokeObjectURL(a.href), 10000);
945
- say(`${fmtInt(csv.split('\n').length - 2)} rows`);
946
  } catch (e) { say(e.message); }
947
  btn.disabled = false;
948
- } }, icon('csv', { size: 16 }), `CSV · ${fmtInt(rows)} rows`);
949
- body.append(el('div', { class: 'card csv-card copy-row' }, icon('csv', { size: 28 }), btn, tooMany ? el('span', { class: 'small' }, `Over ${fmtShort(CSV_MAX_ROWS)} rows: use the command or SQL`) : csvState));
950
  }
951
  }
952
 
@@ -956,19 +949,19 @@ function renderExcluded() {
956
  const by = new Map();
957
  for (const r of rows) by.set(r.category, (by.get(r.category) || 0) + 1);
958
  const ranked = [...by].sort((a, b) => b[1] - a[1]);
959
- const name = (k) => CATEGORY[k] || human(k);
960
- $('excluded-h').textContent = rows.length ? `${fmtInt(rows.length)} reviewed and left out` : 'Nothing left out';
961
  const fig = clear($('excluded-fig'));
962
  if (!rows.length) return;
963
  const items = ranked.map(([k, v]) => ({ key: k, label: name(k), value: v, text: fmtInt(v), tipLabel: IX.requirements.get(k) || name(k) }));
964
- figureWithToggle(fig, 'excluded', 'Reason', {
965
  chart: () => hbars(items),
966
- table: () => table(['Reason', 'Requirement', { label: 'Items', r: true }], ranked.map(([k, v]) => [name(k), IX.requirements.get(k) || '', fmtInt(v)])),
967
  }, () => {
968
  const list = [...rows].sort((a, b) => (a.category || '').localeCompare(b.category || '') || (a.source || '').localeCompare(b.source || ''));
969
- return el('details', { class: 'long' }, el('summary', {}, `All ${fmtInt(rows.length)}`),
970
- el('div', { class: 'table-wrap' }, table(['Source', 'Unit', 'Requirement not met', 'Detail', 'Reviewed'],
971
- list.map((r) => [r.source, r.level === 'source' ? 'whole source' : r.unit, IX.requirements.get(r.category) || name(r.category), r.detail || '', r.reviewed || '']))));
972
  });
973
  }
974
  function snapshots() {
@@ -985,10 +978,11 @@ function snapshots() {
985
  }
986
  function renderHistory() {
987
  const snaps = snapshots();
988
- figureWithToggle($('history-fig'), 'history', 'History: sources after each snapshot', {
989
- chart: () => columns(snaps.map((x) => ({ key: x.date, label: x.date.slice(5), value: x.sources || 0, top: fmtInt(x.sources), sub: x.start ? 'start' : `+${fmtInt(x.added)}` }))),
990
- table: () => table(['Date', 'Entry', { label: 'Sources added', r: true }, { label: 'Rows added', r: true }],
991
- [...IX.changelog].reverse().map((e) => [e.date, e.kind === 'start' ? 'Log started' : sentence(e.title),
 
992
  e.kind === 'update' ? fmtInt(e.sourcesAdded) : e.kind === 'start' ? fmtInt(e.sources) : '',
993
  e.kind === 'update' ? fmtInt(e.rowsAdded) : e.kind === 'start' ? fmtInt(e.rows) : ''])),
994
  });
@@ -1007,16 +1001,17 @@ function renderHeader() {
1007
  });
1008
  if (IX.overrides.catalog || IX.overrides.vocabulary) {
1009
  const chip = $('preview-chip');
1010
- clear(chip).append(el('span', { class: 'dot other', 'aria-hidden': 'true' }), 'Preview data');
1011
  chip.title = [IX.overrides.catalog, IX.overrides.vocabulary].filter(Boolean).join(' ');
1012
  chip.hidden = false;
1013
  }
1014
  }
 
1015
  function updateRequestLinks() {
1016
- const chips = filterChips();
1017
  const u = new URL(`${GITHUB}/issues/new`);
1018
  u.searchParams.set('template', 'data_request.yml');
1019
- if (chips.length) u.searchParams.set('what', `Like this catalog selection (${chips.map(([k, v]) => `${k}: ${v}`).join('; ')}), but: `);
1020
  for (const a of document.querySelectorAll('.request-link')) a.href = u.toString();
1021
  }
1022
  function progress() {
@@ -1057,8 +1052,36 @@ function renderScoped() {
1057
  updateRequestLinks();
1058
  }
1059
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1060
  async function main() {
1061
  readUrl();
 
 
 
1062
  decorate();
1063
  reveal();
1064
  let ticking = false;
@@ -1068,11 +1091,10 @@ async function main() {
1068
  addEventListener('resize', onScroll);
1069
  progress();
1070
  try {
1071
- IX = await loadIndex(new URLSearchParams(location.search), (msg) => { const n = $('notice'); n.textContent = msg; n.hidden = false; });
1072
  } catch (e) {
1073
- const n = $('notice');
1074
- clear(n).append(el('span', { class: 'dot problem', 'aria-hidden': 'true' }), `Could not read the index: ${e.message}`);
1075
- n.hidden = false;
1076
  console.error(e);
1077
  return;
1078
  }
 
2
  // at a glance, take it with one command. Graphics first; every chart keeps a short label and a table view.
3
  import { DATASET_URL, fileUrl, loadBatchDates, loadIndex, num, parquetRows, readColumns, REPO, topOf } from './data.js';
4
  import {
5
+ clear, columns, compactParts, el, figureWithToggle, fmtBytes, fmtCompact, fmtDate, fmtInt, fmtOr, fmtPct, fmtShort, heatmap,
6
+ hbars, hideTip, meter, segmented, stack, table, vt, withTip,
7
  } from './charts.js';
8
  import { decorate, drawHero, markFromIndex, setNumber } from './first.js';
9
+ import { onLanguage, setUpSwitch, t, tEn, translatePage } from './i18n.js';
10
  import { domainIcon, formIcon, icon } from './icons.js';
11
 
12
+ // The categories the group asks for most, shown first. Their names, and those of the top-level domains, are fallbacks
13
+ // in the locale files (domain.<code>) for when the vocabulary has none.
14
+ const MOST_REQUESTED = ['D10.2', 'D4.1', 'D4.5', 'D3.1', 'D3.6', 'D2.5', 'D2.6', 'D2.7', 'D2.8', 'D8.2'];
 
 
 
 
 
 
 
 
 
15
  const GITHUB = 'https://github.com/msra-sg/agentic-env-pipeline';
16
  const CSV_MAX_ROWS = 200000;
17
  const ROW_COLUMNS = ['source', 'batch', 'uri', 'range_offset', 'range_length', 'member_path', 'size_bytes', 'sha256'];
 
23
  const MAX_LISTED = 30;
24
  const $ = (id) => document.getElementById(id);
25
 
26
+ // ---- plain names (codes only inside the SQL and the commands); the words are in the locale files, by code:
27
+ // tier.<tier>, permit.<answer>, reason.<category>, unitKind.<kind> and unitCount.<kind>
28
+ const tierName = (k, tt = t) => tt(`tier.${k}`, { defaultValue: human(k) });
 
 
 
29
  const PERMIT_ORDER = ['yes', 'no', 'conditional', 'unknown'];
30
  const PERMIT_COLOR = { yes: 'var(--ours)', no: 'var(--problem)', conditional: 'var(--fill-3)', unknown: 'var(--other)' };
31
+ const reasonName = (k) => t(`reason.${k}`, { defaultValue: human(k) });
32
+ // the kinds of unit of a raw corpus (sources.json unit) -> their key in the locale files
33
+ const UNIT_KIND = { file: 'file', 'API response': 'apiResponse', 'WARC record': 'warcRecord', 'archive member': 'archiveMember', 'byte range': 'byteRange' };
34
+ // file type -> the tool that opens it (TOOL_WORDS: not a tool's name, a key of the locale files)
 
 
 
 
35
  const TOOLS = {
36
  xlsx: 'openpyxl', xlsm: 'openpyxl', xls: 'LibreOffice', ods: 'LibreOffice', csv: 'DuckDB', tsv: 'DuckDB', parquet: 'DuckDB',
37
  json: 'jq', jsonl: 'jq', ndjson: 'jq', pdf: 'pdftotext', doc: 'LibreOffice', docx: 'python-docx', ppt: 'LibreOffice',
38
  pptx: 'python-pptx', gz: 'gzip', tgz: 'tar', tar: 'tar', bz2: 'bzip2', xz: 'xz', zst: 'zstd', zip: 'unzip', whl: 'unzip',
39
  jar: 'unzip', '7z': '7-Zip', crate: 'tar', deb: 'dpkg-deb', rpm: 'rpm2cpio', warc: 'warcio', arc: 'warcio', pgn: 'python-chess',
40
+ py: 'Python', '(none)': 'file(1)', xml: 'xmllint',
41
+ yaml: 'yq', yml: 'yq', evtx: 'python-evtx', log: 'grep', rego: 'OPA', sql: 'DuckDB', db: 'SQLite',
42
  safetensors: 'safetensors', final: 'TensorBoard', ipynb: 'Jupyter', eml: 'Python email', mbox: 'Python mailbox',
43
  ics: 'icalendar', e01: 'The Sleuth Kit', img: 'The Sleuth Kit', dd: 'The Sleuth Kit', raw: 'The Sleuth Kit',
44
  ys: 'Yosys', v: 'Icarus Verilog', sv: 'Verilator', go: 'Go', rs: 'Cargo', zig: 'Zig', lua: 'Lua', rb: 'Ruby', c: 'gcc',
45
+ tcl: 'tclsh', smt2: 'Z3', mzn: 'MiniZinc', dzn: 'MiniZinc', lp: 'HiGHS', mps: 'HiGHS',
46
  };
47
+ const TOOL_WORDS = { md: 'tool.editor', txt: 'tool.editor', toml: 'tool.editor', html: 'tool.browser', htm: 'tool.browser', test: 'tool.suiteRunner' };
48
+ const toolOf = (ext) => TOOLS[ext] || (TOOL_WORDS[ext] ? t(TOOL_WORDS[ext]) : null);
49
  const NONE = '_none';
50
  const MULTI = '_multi';
51
  const OTHER = '_other';
52
  const human = (code) => String(code).replace(/[_-]+/g, ' ').replace(/^./, (c) => c.toUpperCase());
53
  function sentence(s) {
54
+ let x = String(s).trim();
55
+ if (!/\s/.test(x)) x = x.replace(/[-_]+/g, ' ');
56
+ x = x.replace(/\b(ci|ctf|etl|ml|qa|sql|api)\b/gi, (w) => (w.toLowerCase() === 'qa' ? 'Q&A' : w.toUpperCase()));
57
+ return x.charAt(0).toUpperCase() + x.slice(1);
58
  }
59
+ /** A count with its noun: key_one and key_other in the locale files, {{n}} the formatted number. */
60
+ const plural = (key, n, tt = t) => tt(key, { count: n, n: fmtInt(n) });
61
  const sum = (xs, f) => xs.reduce((a, x) => a + f(x), 0);
62
 
63
  let IX = null;
 
87
  const n = IX.vocab[sec].names.get(code);
88
  return n ? sentence(n) : null;
89
  };
90
+ function domainName(key, tt = t) {
91
+ if (key === NONE) return tt('name.notClassified');
92
+ if (key === MULTI) return vocabName('domain', 'multi') || tt('name.severalDomains');
93
+ return vocabName('domain', key) || tt(`domain.${key}`, { defaultValue: key });
 
94
  }
95
+ function formName(key, tt = t) {
96
+ if (key === NONE || key == null) return tt('name.notClassified');
97
+ if (key === OTHER) return tt('name.otherForms');
98
  return vocabName('form', key) || sentence(key);
99
  }
100
  const domainKeysOf = (s) => (s.domainKind === 'codes' ? s.tops : [s.domainKind === 'multi' ? MULTI : NONE]);
 
108
  }
109
  const inCategory = (s, code) => (code.includes('.') ? s.codes.includes(code) : s.tops.includes(code));
110
 
111
+ // ---- fit: what decides whether a source can feed a task (yes, part, no, unknown); labels fit.<k>, states fitWord.<state>
112
  const FIT = [
113
+ { k: 'ship', icon: 'share' },
114
+ { k: 'bench', icon: 'target' },
115
+ { k: 'pii', icon: 'person' },
116
+ { k: 'size', icon: 'size' },
117
+ { k: 'alone', icon: 'doc' },
118
+ { k: 'answers', icon: 'check' },
119
+ { k: 'login', icon: 'link' },
120
  ];
121
+ const fitLabel = (k) => t(`fit.${k}`);
122
+ const fitWord = (state) => t(`fitWord.${state}`);
123
  function fitOf(s) {
124
  const q = s.info.size_quantiles_upper_bytes || {};
125
  const c = s.contamination?.level;
 
137
  };
138
  }
139
  function fitDot(state, label) {
140
+ return el('span', { class: `fit fit-${state}`, role: 'img', 'aria-label': t('fit.dot', { label, state: fitWord(state) }) });
141
  }
142
  function fitStrip(s) {
143
  const f = fitOf(s);
144
+ return el('span', { class: 'fit-strip' }, FIT.map((x) => withTip(fitDot(f[x.k], fitLabel(x.k)), fitWord(f[x.k]), fitLabel(x.k))));
145
  }
146
  function fitKey() {
147
+ return el('span', { class: 'fit-key' }, ['yes', 'part', 'no', 'unknown'].map((k) => el('span', {}, el('span', { class: `fit fit-${k}`, 'aria-hidden': 'true' }), fitWord(k))));
148
  }
149
 
150
  // ---- filters
 
181
  const selected = () => IX.sources.filter((s) => passes(s));
182
  const FILTER_KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q'];
183
  const activeFilters = () => FILTER_KEYS.filter((k) => F[k]);
184
+ /** The filters that are on, as [label, value] in the words of tt (t, or tEn for the issue the page prefills). */
185
+ function filterChips(tt = t) {
186
  const out = [];
187
+ const word = (key) => tt(key, { defaultValue: '' });
188
+ if (F.domain) out.push([tt('chip.domain'), F.leaf ? domainName(F.leaf, tt) : domainName(F.domain, tt)]);
189
+ if (F.form) out.push([tt('chip.form'), formName(F.form, tt)]);
190
+ if (F.train) out.push([tt('chip.training'), word(`filter.permit.${F.train}`)]);
191
+ if (F.redist) out.push([tt('chip.redistribution'), word(`filter.permit.${F.redist}`)]);
192
+ if (F.tier) out.push([tt('chip.tier'), tierName(F.tier, tt)]);
193
+ if (F.size) out.push([tt('chip.size'), word(`filter.size.${F.size}`)]);
194
+ if (F.status) out.push([tt('chip.kind'), word(`filter.status.${F.status}`)]);
195
+ if (F.q) out.push([tt('chip.search'), F.q]);
196
  return out;
197
  }
198
  function metricOf(s, m = F.metric) { return m === 'sources' ? 1 : m === 'rows' ? s.rows : s.distinct || 0; }
199
  const metricFmt = (m = F.metric) => (m === 'bytes' ? fmtBytes : (v) => fmtInt(v));
200
+ const metricWord = (m) => t(`metric.${m}`);
201
 
202
  function setFilter(patch) {
203
  Object.assign(F, patch);
 
230
  return { args: commandArgs(sel, { domain, form: F.form }), listed: false };
231
  }
232
  async function copy(text, state) {
233
+ try { await navigator.clipboard.writeText(text); state.textContent = t('copy.done'); } catch { state.textContent = t('copy.select'); }
234
  setTimeout(() => { state.textContent = ''; }, 2400);
235
  }
236
  function copyButton(label, text, { dark = false, iconName = 'copy' } = {}) {
 
254
  return { sources: S.length, readySources: S.filter((s) => s.isTask).length, ready, raw, onReady, ship, shipOf: onReady ? ready : raw };
255
  }
256
  const bySize = (a, b) => b.st.ready - a.st.ready || b.st.raw - a.st.raw || b.st.sources - a.st.sources;
257
+ function requestUrl(code) {
258
  const req = new URL(`${GITHUB}/issues/new`);
259
  req.searchParams.set('template', 'data_request.yml');
260
+ req.searchParams.set('what', tEn('request.more', { name: domainName(code, tEn) }));
261
  return req.toString();
262
  }
 
 
 
 
 
 
263
  /** The largest true number of a category: task-ready units, raw rows or sources. */
264
  function leadOf(st) {
265
  const c = [
266
+ { v: st.ready, k: 'ready', word: t('cat.lead.ready', { count: st.ready }) },
267
+ { v: st.raw, k: 'raw', word: t('cat.lead.raw', { count: st.raw }) },
268
+ { v: st.sources, k: 'sources', word: t('cat.lead.sources', { count: st.sources }) },
269
  ].filter((x) => x.v > 0);
270
  return c.sort((x, y) => y.v - x.v)[0];
271
  }
 
280
  el('div', { class: 'cat-main' },
281
  el('span', { class: 'cat-name', title: name }, name),
282
  el('span', { class: 'wanted-art', 'aria-hidden': 'true' }, domainIcon(code, { size: 44 })),
283
+ el('span', { class: 'cat-wanted' }, icon('target', { size: 16 }), t('cat.wanted'))),
284
+ el('div', { class: 'cat-foot' }, el('a', { class: 'btn', href: requestUrl(code) }, icon('request', { size: 16 }), t('cat.request'))));
285
  }
286
  const cur = F.leaf === code;
287
  const pick = () => setFilter(cur ? { domain: '', leaf: '' } : { domain: topOf(code), leaf: code });
288
  const lead = leadOf(st);
289
+ const [n, scale] = compactParts(lead.v);
290
  const num = el('b', { class: 'cat-num' });
291
  setNumber(num, `cat:${code}`, lead.v, () => n);
292
  const meta = [];
293
+ if (lead.k !== 'sources') meta.push(el('span', {}, plural('n.source', st.sources)));
294
+ if (lead.k !== 'raw' && st.raw) meta.push(el('span', { class: 'muted' }, t('cat.plusRaw', { count: st.raw, n: fmtCompact(st.raw) })));
295
+ const readyText = st.ready ? plural('n.unit', st.ready) : t('common.noneYet');
296
  return el('li', { class: `card cat lift${cur ? ' current' : ''}` },
297
  el('button', { type: 'button', class: 'cat-main', 'aria-pressed': String(cur), onclick: pick },
298
  el('span', { class: 'cat-head' }, el('span', { class: 'cat-ico' }, domainIcon(code, { size: 22 })),
299
  cur ? el('span', { class: 'cat-check' }, icon('check', { size: 18 })) : null),
300
  el('span', { class: 'cat-name', title: name }, name),
301
+ el('span', { class: 'cat-lead' }, num, el('span', { class: 'cat-num-label' }, scale ? t('cat.scaled', { scale, word: lead.word }) : lead.word)),
302
  el('span', { class: 'cat-meta' }, meta),
303
  el('span', { class: 'cat-bars' },
304
+ smallBar(t('kind.ready'), st.sources ? st.readySources / st.sources : 0, readyText,
305
+ t('cat.readyTip', { count: st.sources, ready: fmtInt(st.readySources), n: fmtInt(st.sources) })),
306
+ smallBar(st.onReady ? t('cat.shippable') : t('cat.shippableRaw'), st.shipOf ? st.ship / st.shipOf : 0, fmtPct(st.ship, st.shipOf),
307
+ st.onReady ? t('cat.shipTipReady') : t('cat.shipTipRaw')))),
308
+ el('div', { class: 'cat-foot' }, copyButton(t('cat.sample'), command(commandArgs(null, { domain: code })))));
309
  }
310
  function renderCategories() {
311
+ const cats = MOST_REQUESTED.map((code) => ({ code, st: catStats(code) }));
312
  const have = cats.filter((c) => c.st.sources).sort(bySize);
313
  const wanted = cats.filter((c) => !c.st.sources);
314
  clear($('req-cards')).append(...have.map(catCard), ...wanted.map(catCard));
 
321
  const STACK_NAMED = 3;
322
  // parts ranked by size, largest darkest (one ordinal ramp of the slate blue); not classified yet is gray stripes
323
  const STEPS = ['var(--ours)', '#6f8797', '#8ea2ae', '#a8b6c0'];
324
+ function capParts(entries, otherKey) {
325
  const named = entries.filter((e) => e.key !== NONE && e.value > 0).sort((x, y) => y.value - x.value);
326
  const none = entries.find((e) => e.key === NONE && e.value > 0);
327
  const head = named.length > STACK_NAMED + 1 ? named.slice(0, STACK_NAMED) : named;
328
  const rest = named.slice(head.length);
329
  const parts = head.map((e, i) => ({ ...e, color: STEPS[i] }));
330
+ if (rest.length) parts.push({ key: OTHER, label: plural(otherKey, rest.length), value: sum(rest, (e) => e.value), sources: sum(rest, (e) => e.sources), color: STEPS[3] });
331
+ if (none) parts.push({ ...none, label: t('stack.notClassified'), color: 'var(--unclassified)' });
332
+ return parts.map((p) => ({ ...p, sub: plural('n.unit', p.value) }));
333
  }
334
  function groupReady(T, keyOf) {
335
  const m = new Map();
 
356
  tipLabel: [s.title, formName(formKeyOf(s))], onClick: () => openSource(s.id) }));
357
  if (rest.length) {
358
  const v = sum(rest, (s) => s.readyUnits);
359
+ const other = plural('where.otherSources', rest.length);
360
+ items.push({ key: OTHER, label: other, value: v, text: fmtInt(v), tipLabel: other });
361
  }
362
+ figureWithToggle($('where-src'), 'where-src', t('where.title'), {
363
+ chart: () => (items.length ? hbars(items) : el('p', { class: 'small muted' }, t('common.noneYet'))),
364
+ table: () => table([t('th.source'), t('th.form'), { label: t('th.readyUnits'), r: true }],
365
  ranked.map((s) => [s.title, formName(formKeyOf(s)), fmtInt(s.readyUnits)])),
366
  });
367
  }
 
377
  const inside = ready - answered;
378
  const raw = sum(R, (s) => s.rawRows);
379
  const parts = [
380
+ { value: T.length, color: 'var(--ours)', label: t('kind.ready'), sub: `${plural('n.source', T.length)} · ${plural('n.unit', ready)}` },
381
+ { value: R.length, color: 'var(--other-light)', label: t('ready.rawCorpora'), sub: `${plural('n.source', R.length)} · ${t('n.rowsCompact', { count: raw, n: fmtCompact(raw) })}` },
382
  ];
383
  const inner = [
384
+ { value: answered, color: 'var(--ours)', label: t('ready.answersApart'), sub: plural('n.unit', answered) },
385
+ { value: inside, color: 'var(--fill-3)', label: t('ready.checksInside'), sub: plural('n.unit', inside) },
386
  ];
387
  const forms = groupReady(T, formKeyOf);
388
  const doms = groupReady(T, oneDomain);
389
+ const formParts = capParts(forms.map((e) => ({ ...e, label: formName(e.key) })), 'stack.otherForms');
390
+ const domParts = capParts(doms.map((e) => ({ ...e, label: domainName(e.key) })), 'stack.otherDomains');
391
  const bar = (ps, twoRows = false) => el('div', { class: 'split' },
392
  el('div', { class: 'stack', role: 'img', 'aria-label': ps.map((p) => `${p.label} ${p.sub}`).join(', ') },
393
  ps.filter((p) => p.value > 0).map((p) => withTip(el('span', { tabindex: '0', style: { flex: `${p.value} 1 0`, background: p.color } }), p.sub, p.label))),
394
  el('div', { class: `split-labels fixed${twoRows ? ' two-rows' : ''}` }, ps.map((p) => el('span', { title: `${p.label}: ${p.sub}` },
395
  el('span', { class: 'sw', style: { background: p.color } }), el('b', {}, p.label), el('span', { class: 'small' }, p.sub)))));
396
  const only = el('button', { type: 'button', class: 'btn small-btn', 'aria-pressed': String(F.status === 'ready'),
397
+ onclick: () => setFilter({ status: F.status === 'ready' ? '' : 'ready' }) }, icon('pick', { size: 16 }), F.status === 'ready' ? t('ready.showing') : t('ready.only'));
398
  const mini = (ico, text, first) => el('p', { class: `mini-title${first ? ' first' : ''}` }, icon(ico, { size: 16 }), text);
399
  const rows = (list, kind) => [...list].sort((x, y) => y.value - x.value).map((e) => [kind, e.label ?? '', fmtInt(e.sources), fmtInt(e.value)]);
400
+ figureWithToggle($('ready-card'), 'ready', t('ready.title'), {
401
+ chart: () => el('div', {}, mini('sources', t('metric.sources'), true), bar(parts), mini('check', t('th.readyUnits'), false), bar(inner),
402
+ mini('stack', t('ready.byForm'), false), bar(formParts, true), mini('browse', t('ready.byDomain'), false), bar(domParts, true)),
403
+ table: () => table([t('ready.split'), t('ready.part'), { label: t('metric.sources'), r: true }, { label: t('th.readyUnits'), r: true }], [
404
+ [t('ready.kind'), t('kind.ready'), fmtInt(T.length), fmtInt(ready)],
405
+ [t('ready.kind'), t('ready.rawRow', { count: raw, n: fmtInt(raw) }), fmtInt(R.length), ''],
406
+ [t('ready.units'), t('ready.answersKept'), fmtInt(T.filter((s) => s.answered).length), fmtInt(answered)],
407
+ [t('ready.units'), t('ready.resultsInside'), fmtInt(T.filter((s) => !s.answered).length), fmtInt(inside)],
408
+ ...rows(forms.map((e) => ({ ...e, label: formName(e.key) })), t('th.form')),
409
+ ...rows(doms.map((e) => ({ ...e, label: domainName(e.key) })), t('th.domain')),
410
  ]),
411
  }, () => el('div', { class: 'card-foot' }, only));
412
  }
413
+ // ---- filter row: the options (again in a new language), and their listeners (once)
414
  function option(value, text) { return el('option', { value }, text); }
415
+ function fillFilterOptions() {
416
+ const all = t('filter.all');
417
+ clear($('f-domain')).append(option('', all), domainRows().map((k) => option(k, domainName(k))));
418
+ clear($('f-form')).append(option('', all), formsByCount().map((k) => option(k, formName(k))));
419
+ const tiers = [...new Set(IX.sources.map((s) => s.tier))].sort((a, b) => tierName(a).localeCompare(tierName(b)));
420
+ clear($('f-tier')).append(option('', all), tiers.map((k) => option(k, tierName(k))));
421
+ }
422
  function fillFilterRow() {
423
+ fillFilterOptions();
 
 
 
424
  const on = (id, key) => $(id).addEventListener('change', (e) => {
425
  const patch = { [key]: e.target.value };
426
  if (key === 'domain') patch.leaf = '';
 
428
  });
429
  on('f-domain', 'domain'); on('f-leaf', 'leaf'); on('f-form', 'form'); on('f-train', 'train'); on('f-redist', 'redist');
430
  on('f-tier', 'tier'); on('f-size', 'size'); on('f-status', 'status');
431
+ let timer = null;
432
+ $('f-q').addEventListener('input', (e) => { clearTimeout(timer); timer = setTimeout(() => setFilter({ q: e.target.value.trim() }), 200); });
433
  $('f-clear').addEventListener('click', () => setFilter(Object.fromEntries(FILTER_KEYS.map((k) => [k, '']))));
434
  $('filter-toggle').addEventListener('click', () => {
435
  const open = !$('filters').classList.contains('open');
 
444
  if (document.activeElement !== $('f-q')) $('f-q').value = F.q;
445
  const leaf = $('f-leaf');
446
  const leaves = F.domain && F.domain !== NONE && F.domain !== MULTI ? leafKeys(F.domain) : [];
447
+ clear(leaf).append(option('', t('filter.all')), leaves.map((k) => option(k, domainName(k))));
448
  leaf.disabled = !leaves.length;
449
  leaf.value = F.leaf;
450
  const n = activeFilters().length;
451
  $('f-clear').disabled = !n;
452
+ $('filter-state').textContent = n ? `${t('filter.on', { count: n, n: fmtInt(n) })} · ${plural('n.source', sel.length)}` : t('filter.noneOn');
453
+ $('f-count').textContent = plural('n.source', sel.length);
454
  }
455
 
456
  // ---- axes of the map
 
513
  // gray row and column of the map, never the headline)
514
  const total = sum(base, (s) => metricOf(s, m));
515
  const covered = new Set(base.flatMap((s) => s.tops)).size;
516
+ const amount = m === 'sources' ? plural('n.source', total) : m === 'rows' ? t('map.rows', { count: total, n: fmtCompact(total) }) : fmtBytes(total);
517
+ $('map-h').textContent = !base.length ? t('map.nothing') : covered ? t('map.head', { amount, domains: plural('n.domain', covered) }) : amount;
518
  for (const b of $('metric-seg').querySelectorAll('button')) b.setAttribute('aria-pressed', String(b.dataset.metric === m));
519
 
520
  const current = F.domain && F.form ? { row: F.domain, col: F.form } : null;
521
  const rowObjs = rows.map((k) => ({ key: k, label: domainName(k), muted: k === NONE }));
522
  const colObjs = cols.map((k) => ({ key: k, label: formName(k), head: formIcon(k, { size: 20 }), muted: k === NONE }));
523
+ figureWithToggle($('heat-fig'), 'heat', t('map.title', { metric: metricWord(m) }), {
524
  chart: () => {
525
  const h = heatmap({
526
  rows: rowObjs, cols: colObjs, value: (r, c) => cell.get(`${r.key}|${c.key}`) || 0, fmt, current,
527
+ metricName: metricWord(m),
528
  onPick: (r, c) => {
529
  const form = c.key === OTHER ? '' : c.key;
530
  if (F.domain === r.key && F.form === form) setFilter({ domain: '', leaf: '', form: '' });
531
  else setFilter({ domain: r.key, leaf: '', form });
532
  },
533
  });
534
+ const key = el('ul', { class: 'icon-key', 'aria-label': t('map.columns') }, colObjs.map((c) => el('li', {}, formIcon(c.key, { size: 16 }), c.label)));
535
  return el('div', { class: 'heat-body' }, h.grid, key, h.legend);
536
  },
537
+ table: () => table([t('th.domain'), t('th.form'), { label: metricWord(m), r: true }],
538
  rowObjs.flatMap((r) => colObjs.map((c) => [r, c, cell.get(`${r.key}|${c.key}`) || 0])).filter((x) => x[2] > 0)
539
  .map(([r, c, v]) => [r.label, c.label, fmt(v)])),
540
  });
 
561
  }
562
  }
563
  const domMax = Math.max(0, ...rows.map((r) => domVal.get(r)));
564
+ figureWithToggle($('dom-fig'), 'dom', t('ready.byDomain'), {
565
  chart: () => hbars(items, { max: domMax }),
566
+ table: () => table([t('th.domain'), { label: metricWord(m), r: true }], items.map((d) => [d.leaf ? ` ${d.label}` : d.label, d.text])),
567
  });
568
 
569
  const formBase = IX.sources.filter((s) => passes(s, { form: true }));
 
573
  const fShown = UI.allForms ? fAll : fAll.slice(0, FORM_BARS);
574
  const fItems = fShown.map((c) => ({ key: c, label: formName(c), icon: formIcon(c, { size: 16 }), value: fv.get(c) || 0, text: fmt(fv.get(c) || 0),
575
  current: F.form === c, muted: c === NONE, tipLabel: formName(c), onClick: () => setFilter({ form: F.form === c ? '' : c }) }));
576
+ figureWithToggle($('form-fig'), 'form', t('ready.byForm'), {
577
  chart: () => hbars(fItems),
578
+ table: () => table([t('th.form'), { label: metricWord(m), r: true }], fAll.map((c) => [formName(c), fmt(fv.get(c) || 0)])),
579
  }, () => (fAll.length > FORM_BARS ? el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.allForms),
580
+ onclick: () => { UI.allForms = !UI.allForms; renderMap(); } }, UI.allForms ? t('common.fewer') : t('common.all', { n: fmtInt(fAll.length) })) : null));
581
  }
582
 
583
+ // ---- 4. sources; the column labels are col.<key> in the locale files
584
  const COLS = [
585
+ { key: 'title' },
586
+ { key: 'domain' },
587
+ { key: 'form' },
588
+ { key: 'units', r: true },
589
+ { key: 'distinct', r: true },
590
+ { key: 'tier' },
591
+ { key: 'fit' },
592
  ];
593
  function fitScore(s) { const f = fitOf(s); return sum(FIT, (x) => ({ yes: 2, part: 1, unknown: 0.5, no: 0 })[f[x.k]]); }
594
  function sortValue(s, k) {
 
599
  // task-ready units above raw rows (never compared as one count): a raw corpus has fewer than 1e12 rows
600
  case 'units': return s.isTask ? 1e12 + s.readyUnits : s.rawRows;
601
  case 'distinct': return s.distinct ?? -1;
602
+ case 'tier': return tierName(s.tier).toLowerCase();
603
  case 'fit': return fitScore(s);
604
  default: return 0;
605
  }
606
  }
607
+ const distinctText = (s) => (s.distinct == null ? t('common.unknown') : fmtBytes(s.distinct));
608
+ // #364: the same words as the index README, section Size (artpool/projection_templates/text.json, size_text), in the
609
+ // locale files as size.method
 
 
 
 
610
  /** Distinct content and rows without a size of the sources S: the rule of index_projection.size_totals. */
611
  function sizeTotals(S) {
612
  const known = S.filter((s) => s.distinct != null);
613
  return { distinct: known.length ? sum(known, (s) => s.distinct) : null, unsized: sum(S, (s) => s.rowsNoSize) };
614
  }
615
  function sizeNote(S) {
616
+ const s = sizeTotals(S);
617
+ if (s.distinct == null) return t('size.unknown');
618
+ const values = { size: fmtBytes(s.distinct), method: t('size.method') };
619
+ return s.unsized ? t('size.noteUnsized', { ...values, unsized: t('size.unsized', { count: s.unsized, n: fmtInt(s.unsized) }) }) : t('size.note', values);
620
  }
621
 
622
  function renderSources(sel) {
 
627
  return (x < y ? -1 : x > y ? 1 : 0) * dir || a.title.localeCompare(b.title);
628
  });
629
  const nReady = sel.filter((s) => s.isTask).length;
630
+ $('sources-h').textContent = sel.length ? t('sources.head', { sources: plural('n.source', sel.length), ready: fmtInt(nReady) }) : t('sources.none');
631
  const body = clear($('sources-body'));
632
  if (!sel.length) return;
633
  const shown = UI.showAll ? list : list.slice(0, FIRST_ROWS);
 
639
  el('button', { type: 'button', onclick: () => {
640
  const d = on ? (F.dir === 'asc' ? 'desc' : 'asc') : (c.r || c.key === 'fit' ? 'desc' : 'asc');
641
  F.sort = c.key; F.dir = d; writeUrl(); renderSources(selected());
642
+ } }, t(`col.${c.key}`), on ? (F.dir === 'asc' ? ' ↑' : ' ↓') : ''));
643
  }));
644
  const tbody = el('tbody');
645
  for (const s of shown) {
 
649
  el('td', { class: 'title' },
650
  el('button', { type: 'button', 'aria-expanded': String(open), 'aria-controls': `d-${s.id}`, onclick: toggle },
651
  el('span', { class: 'caret', 'aria-hidden': 'true' }, open ? '▾' : '▸'), s.title),
652
+ s.isTask ? el('span', { class: 'ready-tag' }, icon('check', { size: 14 }), t('sources.ready')) : null),
653
+ el('td', { 'data-label': t('col.domain'), class: 'cat-cell' }, sourceDomainText(s)),
654
+ el('td', { 'data-label': t('col.form'), class: 'form-cell' }, formIcon(formKeyOf(s), { size: 16 }), formName(formKeyOf(s))),
655
+ el('td', { class: 'r num', 'data-label': s.isTask ? t('sources.unitsLabel') : t('sources.rowsLabel') }, s.isTask
656
+ ? [fmtInt(s.readyUnits), el('span', { class: 'unit-word' }, t('sources.unitWord', { count: s.readyUnits }))]
657
+ : el('span', { class: 'muted' }, fmtCompact(s.rawRows), el('span', { class: 'unit-word' }, t('sources.rawWord', { count: s.rawRows })))),
658
+ el('td', { class: 'r num', 'data-label': t('col.distinct') }, distinctText(s)),
659
+ el('td', { 'data-label': t('col.tier') }, tierName(s.tier)),
660
  el('td', { class: 'fit-cell' }, fitStrip(s))));
661
  if (open) tbody.append(el('tr', { class: 'detail-row' }, el('td', { colspan: String(COLS.length), id: `d-${s.id}` }, detail(s))));
662
  }
663
+ body.append(el('div', { class: 'list-key' }, el('span', { class: 'small' }, t('sources.fitKey')), FIT.map((x) => el('span', { class: 'fit-name' }, icon(x.icon, { size: 14 }), fitLabel(x.k))), fitKey()));
664
+ body.append(el('table', { class: 'src-table' }, el('caption', { class: 'visually-hidden' }, t('metric.sources')), el('thead', {}, head), tbody));
665
  body.append(el('p', { class: 'small size-note' }, sizeNote(sel)));
666
  if (list.length > FIRST_ROWS) {
667
  body.append(el('p', { class: 'more' }, el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.showAll),
668
+ onclick: () => { UI.showAll = !UI.showAll; renderSources(selected()); } }, UI.showAll ? t('common.first', { n: fmtInt(FIRST_ROWS) }) : t('common.all', { n: fmtInt(list.length) }))));
669
  }
670
  }
671
 
 
691
  const lo = Math.log10(Math.min(1e3, v('p10')));
692
  const hi = Math.log10(Math.max(v('p99'), v('p90') * 10, MAX_FILE * 2));
693
  const at = (x) => `${(100 * (Math.log10(x) - lo)) / (hi - lo)}%`;
694
+ return el('div', { class: 'strip', role: 'img', 'aria-label': t('detail.stripAria', { median: fmtBytes(v('p50')), p99: fmtBytes(v('p99')) }) },
695
  el('span', { class: 'strip-axis' }),
696
  el('span', { class: 'strip-band', style: { left: at(v('p10')), width: `calc(${at(v('p90'))} - ${at(v('p10'))})` } }),
697
  el('span', { class: 'strip-limit', style: { left: at(MAX_FILE) } }),
698
  el('span', { class: 'strip-dot', style: { left: at(v('p50')) } }),
699
+ el('span', { class: 'strip-lab', style: { left: at(v('p50')) } }, t('detail.median', { size: fmtBytes(v('p50')) })),
700
  el('span', { class: 'strip-lab limit', style: { left: at(MAX_FILE) } }, '20 MB'));
701
  }
702
  function item(iconName, label, ...value) {
 
727
 
728
  // fit at a glance
729
  box.append(el('ul', { class: 'fit-row' }, FIT.map((x) => el('li', { class: `fit-item fit-${fit[x.k]}-item` },
730
+ icon(x.icon, { size: 18 }), el('span', {}, fitLabel(x.k)), fitDot(fit[x.k], fitLabel(x.k))))));
731
 
732
  const dl = el('dl', { class: 'd-list' });
733
  // 1 what it is
734
  const text = s.summary || s.description;
735
+ dl.append(item('doc', t('detail.what'), text ? (text.length > 160 ? el('details', {}, el('summary', {}, firstSentence(text)), el('p', {}, text)) : text) : t('detail.notDescribed')));
736
  // 2 what one unit is
737
+ const kindCount = (k) => {
738
+ const n = num(s.info.unit[k]);
739
+ return t(UNIT_KIND[k] ? `unitCount.${UNIT_KIND[k]}` : 'unitCount.code', { count: n, n: fmtCompact(n), kind: k });
740
+ };
741
  const unitWhat = s.isTask
742
+ ? [el('span', {}, s.answered ? t('detail.taskUnit') : t('detail.unitWithChecks')), el('span', { class: 'chips' },
743
+ el('span', { class: 'chip' }, plural('n.unitFile', s.roles.unit)), s.roles.companion ? el('span', { class: 'chip' }, plural('n.companion', s.roles.companion)) : null,
744
+ s.roles.answer ? el('span', { class: 'chip' }, plural('n.answer', s.roles.answer)) : null)]
745
+ : [el('span', {}, sentence(fmtOr(s.kinds.map((k) => (UNIT_KIND[k] ? t(`unitKind.${UNIT_KIND[k]}`) : k))))),
746
+ el('span', { class: 'chips' }, s.kinds.map((k) => el('span', { class: 'chip' }, kindCount(k))))];
747
+ dl.append(item('rows', t('detail.oneUnit'), ...unitWhat));
748
  // 3 where it comes from
749
  const origins = [...new Set(metas.map((m) => m.pool_source?.origin).filter(Boolean))];
750
  const hosts = [...new Set(rows.map((r) => hostOf(r.uri)).filter(Boolean))];
751
  const from = origins.length ? origins : hosts;
752
+ dl.append(item('pin', t('detail.from'), st.sample === 'loading' ? '…' : from.length ? el('span', { class: 'chips' }, from.slice(0, 3).map((h) => el('span', { class: 'chip mono-chip' }, h))) : t('detail.notRecorded')));
753
  // 4 self-contained or companions
754
  const comps = [...new Set(rows.filter((r, i) => metas[i].role === 'companion').map((r) => r.uri.split('/').pop()))];
755
+ dl.append(item('layers', t('detail.alone'), s.isTask
756
+ ? (s.roles.companion ? [el('span', {}, t('detail.withCompanions')), el('span', { class: 'chips' }, (comps.length ? comps : ['LICENSE', 'README.md']).map((c) => el('span', { class: 'chip mono-chip' }, c)))] : t('detail.yes'))
757
+ : fit.alone === 'yes' ? t('detail.eachFile') : t('detail.container')));
758
  // 5 where the answers are
759
+ dl.append(item('check', t('detail.answers'), s.answered ? el('span', { class: 'mono-chip chip' }, `answer/${s.id}/ · ${plural('n.file', s.answered)}`)
760
+ : s.isTask ? t('detail.answersInside') : el('span', { class: 'muted' }, t('detail.answersNone'))));
761
  // 6 tools to open it
762
  const exts = Object.entries(info.by_ext || {}).map(([k, v]) => [k, num(v.rows) || 0]).sort((a, b) => b[1] - a[1]).map(([k]) => k);
763
  const tools = [];
764
+ for (const e of exts) { const tool = toolOf(e); if (tool && !tools.some((x) => x[1] === tool)) tools.push([e, tool]); if (tools.length >= 3) break; }
765
+ dl.append(item('tool', t('detail.opensWith'), tools.length ? el('span', { class: 'chips' }, tools.map(([e, tool]) => el('span', { class: 'chip' }, el('span', { class: 'mono' }, e === '(none)' ? t('detail.noExt') : `.${e}`), ` ${tool}`))) : t('detail.anyEditor')));
766
  // 7 licence
767
  const lr = info.license_rows || {};
768
+ const permit = (field) => PERMIT_ORDER.map((k) => ({ k, value: num(lr[field]?.[k]?.rows) || 0, color: PERMIT_COLOR[k], label: t(`permit.${k}`) }));
769
+ dl.append(item('training', t('detail.licence'), el('span', { class: 'lic' }, el('b', {}, tierName(s.tier)), el('span', { class: 'small' }, lic.name || s.spdx || '')),
770
  el('span', { class: 'lic-bars' },
771
+ el('span', { class: 'mini-title' }, t('detail.training')), stack(permit('training_ok'), s.rows),
772
+ el('span', { class: 'mini-title' }, t('detail.shipping')), stack(permit('redistribution_ok'), s.rows))));
773
  // 8 size and units
774
  const q = info.size_quantiles_upper_bytes || {};
775
+ const amounts = s.isTask ? `${plural('n.unit', s.readyUnits)} · ${plural('n.file', s.rows)}` : plural('n.rawRows', s.rawRows);
776
+ const unsized = s.rowsNoSize ? ` · ${t('detail.unsized', { n: fmtCompact(s.rowsNoSize) })}` : '';
777
+ dl.append(item('size', t('detail.size'), el('span', { class: 'lic' }, el('b', {}, t('detail.distinct', { size: distinctText(s) })), el('span', { class: 'small' },
778
+ `${amounts}${unsized}`)),
779
  q.p50 ? sizeStrip(q) : null));
780
  // 9 one example file
781
  let ex = null;
 
787
  ex = el('span', { class: 'example' }, href ? el('a', { href, rel: 'noopener', class: 'mono' }, name) : el('span', { class: 'mono' }, name),
788
  r.size_bytes != null ? el('span', { class: 'small' }, ` · ${fmtBytes(num(r.size_bytes))}`) : null);
789
  }
790
+ dl.append(item('link', t('detail.example'), st.sample === 'loading' ? '…' : st.sample?.error ? el('span', { class: 'problem-text' }, st.sample.error) : ex || t('detail.noFile')));
791
  // 10 the command that takes this source
792
  const cmd = command(commandArgs(null, { sources: [s.id] }));
793
+ dl.append(item('run', t('detail.take'), el('code', { class: 'cmd-inline' }, cmd), copyButton(t('copy.button'), cmd)));
794
  box.append(dl);
795
 
796
  // batches and the rows read, on request
797
  const b = st.batches;
798
+ const more = el('details', { class: 'd-more' }, el('summary', {}, t('detail.more')));
799
  if (Array.isArray(b)) {
800
+ more.append(table([t('th.batch'), { label: t('th.rows'), r: true }, t('th.changed'), t('th.state')],
801
+ b.map((x) => [x.batch, x.rows == null ? t('detail.notCounted') : fmtInt(x.rows), fmtDate(x.date),
802
+ x.status ? t(`batch.${x.status}`, { defaultValue: human(x.status) }) : ''])));
803
  }
804
  if (rows.length) {
805
+ more.append(el('div', { class: 'table-wrap sample' }, table([t('th.address'), { label: t('th.size'), r: true }, 'sha256'],
806
  rows.map((r) => [r.member_path ? `${r.uri} ! ${r.member_path}` : r.range_offset != null ? `${r.uri} @ ${num(r.range_offset)}+${num(r.range_length)}` : r.uri,
807
  r.size_bytes == null ? '' : fmtInt(num(r.size_bytes)), hex(r.sha256).slice(0, 16) + (r.sha256 ? '…' : '')]), st.file)));
808
  }
 
836
  const paths = all ? [`${root}/data/*/*/*.parquet`] : sel.map((s) => `${root}/data/${s.id}/*/*.parquet`);
837
  const chips = filterChips();
838
  return [
839
+ `-- ${t('sql.head', { sha: IX.sha.slice(0, 7), sources: plural('n.source', sel.length), rows: plural('n.row', sum(sel, (s) => s.rows)) })}`,
840
+ `-- ${chips.length ? chips.map(([k, v]) => t('sql.chip', { k, v })).join('; ') : t('sql.all')}`,
841
  `SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
842
  'FROM read_parquet([',
843
  paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
 
876
  const src = first ? first.id : '<source>';
877
  const line = (depth, iconName, name, note) => el('li', { style: { paddingLeft: `${depth * 20}px` } }, icon(iconName, { size: 16 }),
878
  el('span', { class: 'tree-name' }, el('span', { class: 'mono' }, name), note ? el('span', { class: 'small' }, note) : null));
879
+ return el('figure', { class: 'card tree-fig' }, el('p', { class: 'chart-title' }, t('tree.title')),
880
  el('ul', { class: 'tree' },
881
  line(0, 'folder', 'data/'),
882
+ line(1, 'folder', `<domain>/${src}/`, t('tree.units')),
883
+ line(1, 'folder', `answer/${src}/`, t('tree.answers')),
884
  line(1, 'folder', '_meta/'),
885
+ line(2, 'rows', 'manifest.jsonl', t('tree.manifest')),
886
+ line(2, 'rows', 'sizes.tsv', t('tree.sizes'))),
887
+ el('p', { class: 'small tree-note' }, el('span', { class: 'mono' }, '<domain>'), t('tree.domainNote')),
888
+ el('p', { class: 'chart-title profile-title' }, t('tree.leftOut')),
889
+ el('ul', { class: 'chips profile' }, ['duplicates', 'benchmark', 'licence', 'english', 'personal', 'large']
890
+ .map((k) => el('li', { class: 'chip' }, t(`profile.${k}`)))),
891
  el('p', { class: 'small agent-links' }, icon('robot', { size: 16 }),
892
+ el('a', { href: `${DATASET_URL}#for-agents` }, t('agents.readme')), ' · ', el('a', { href: 'llms.txt' }, 'llms.txt')));
893
  }
894
  function renderGet(sel) {
895
  const ready = sum(sel, (s) => s.readyUnits);
896
  const raw = sum(sel, (s) => s.rawRows);
897
+ $('get-h').textContent = !sel.length ? t('get.nothing') : sel.length === 1 ? t('get.takeOne', { count: SAMPLE, title: sel[0].title })
898
+ : t('get.takeEach', { count: SAMPLE, sources: plural('n.source', sel.length) });
899
  const body = clear($('get-body'));
900
  if (!sel.length) return;
901
  const chips = filterChips();
902
+ const tabs = segmented('', [['cmd', t('get.command')], ['sql', 'SQL'], ['csv', 'CSV']], UI.tab, (v) => { UI.tab = v; vt(() => renderGet(selected())); });
903
  const pickRow = el('div', { class: 'pick-row' },
904
+ el('span', { class: 'fact-line' }, icon('sources', { size: 18 }), el('b', {}, fmtInt(sel.length)), t('get.sourceWord', { count: sel.length })),
905
+ el('span', { class: 'fact-line' }, icon('check', { size: 18 }), el('b', {}, fmtInt(ready)), t('cat.lead.ready', { count: ready })),
906
+ raw ? el('span', { class: 'fact-line muted' }, icon('rows', { size: 18 }), el('b', {}, fmtCompact(raw)), t('cat.lead.raw', { count: raw })) : null,
907
+ el('span', { class: 'chips' }, chips.length ? chips.map(([k, v]) => el('span', { class: 'chip', title: k }, v)) : el('span', { class: 'chip' }, t('get.wholeIndex'))),
908
+ el('a', { class: 'small', href: '#filters' }, t('get.change')), tabs);
909
  body.append(pickRow);
910
  const codeCard = (iconName, title, code, n) => el('div', { class: 'code-card' },
911
  el('div', { class: 'code-head' }, el('span', { class: 'code-ico' }, icon(iconName, { size: 20 })),
912
  n ? el('span', { class: 'step-n' }, String(n)) : null, el('span', { class: 'code-title' }, title),
913
+ copyButton(t('copy.button'), code, { dark: true })),
914
  el('pre', { class: 'code' }, el('code', {}, code)));
915
  if (UI.tab === 'cmd') {
916
  const sa = selectionArgs(sel);
917
  const dry = command(sa.args, true);
918
  const take = command(sa.args, false);
919
  body.append(el('div', { class: 'get-grid' },
920
+ el('div', { class: 'cmd-steps' }, codeCard('search', t('get.look'), dry, 1), codeCard('run', t('get.take'), take, 2)),
921
  folderTree(sel)));
922
  } else if (UI.tab === 'sql') {
923
+ body.append(codeCard('database', t('get.sql'), selectionSql(sel)));
924
  } else {
925
  const rows = sum(sel, (s) => s.rows);
926
  const key = sel.map((s) => s.id).join(',');
 
928
  const tooMany = rows > CSV_MAX_ROWS;
929
  const btn = el('button', { type: 'button', class: 'btn solid', disabled: tooMany, onclick: async () => {
930
  btn.disabled = true;
931
+ const say = (text) => { csvState.textContent = text; UI.csv = { key, text }; };
932
  try {
933
+ const csv = await buildCsv(sel, (d, n) => say(t('csv.progress', { count: n, done: fmtInt(d), n: fmtInt(n) })));
934
  const blob = new Blob([csv], { type: 'text/csv' });
935
  const a = el('a', { href: URL.createObjectURL(blob), download: `agentic-env-index-${IX.sha.slice(0, 7)}-${sel.length === 1 ? sel[0].id : `${sel.length}-sources`}.csv` });
936
  document.body.append(a); a.click(); a.remove();
937
  setTimeout(() => URL.revokeObjectURL(a.href), 10000);
938
+ say(plural('n.row', csv.split('\n').length - 2));
939
  } catch (e) { say(e.message); }
940
  btn.disabled = false;
941
+ } }, icon('csv', { size: 16 }), t('csv.button', { rows: plural('n.row', rows) }));
942
+ body.append(el('div', { class: 'card csv-card copy-row' }, icon('csv', { size: 28 }), btn, tooMany ? el('span', { class: 'small' }, t('csv.tooMany', { n: fmtShort(CSV_MAX_ROWS) })) : csvState));
943
  }
944
  }
945
 
 
949
  const by = new Map();
950
  for (const r of rows) by.set(r.category, (by.get(r.category) || 0) + 1);
951
  const ranked = [...by].sort((a, b) => b[1] - a[1]);
952
+ const name = reasonName;
953
+ $('excluded-h').textContent = rows.length ? t('excluded.head', { count: rows.length, n: fmtInt(rows.length) }) : t('excluded.none');
954
  const fig = clear($('excluded-fig'));
955
  if (!rows.length) return;
956
  const items = ranked.map(([k, v]) => ({ key: k, label: name(k), value: v, text: fmtInt(v), tipLabel: IX.requirements.get(k) || name(k) }));
957
+ figureWithToggle(fig, 'excluded', t('excluded.reason'), {
958
  chart: () => hbars(items),
959
+ table: () => table([t('excluded.reason'), t('excluded.requirement'), { label: t('excluded.items'), r: true }], ranked.map(([k, v]) => [name(k), IX.requirements.get(k) || '', fmtInt(v)])),
960
  }, () => {
961
  const list = [...rows].sort((a, b) => (a.category || '').localeCompare(b.category || '') || (a.source || '').localeCompare(b.source || ''));
962
+ return el('details', { class: 'long' }, el('summary', {}, t('common.all', { n: fmtInt(rows.length) })),
963
+ el('div', { class: 'table-wrap' }, table([t('th.source'), t('excluded.unit'), t('excluded.notMet'), t('excluded.detail'), t('excluded.reviewed')],
964
+ list.map((r) => [r.source, r.level === 'source' ? t('excluded.wholeSource') : r.unit, IX.requirements.get(r.category) || name(r.category), r.detail || '', r.reviewed || '']))));
965
  });
966
  }
967
  function snapshots() {
 
978
  }
979
  function renderHistory() {
980
  const snaps = snapshots();
981
+ figureWithToggle($('history-fig'), 'history', t('history.title'), {
982
+ chart: () => columns(snaps.map((x) => ({ key: x.date, label: fmtDate(x.date, { month: 'numeric', day: 'numeric' }), value: x.sources || 0, top: fmtInt(x.sources),
983
+ sub: x.start ? t('history.start') : `+${fmtInt(x.added)}` }))),
984
+ table: () => table([t('history.date'), t('history.entry'), { label: t('history.sourcesAdded'), r: true }, { label: t('history.rowsAdded'), r: true }],
985
+ [...IX.changelog].reverse().map((e) => [fmtDate(e.date), e.kind === 'start' ? t('history.started') : sentence(e.title),
986
  e.kind === 'update' ? fmtInt(e.sourcesAdded) : e.kind === 'start' ? fmtInt(e.sources) : '',
987
  e.kind === 'update' ? fmtInt(e.rowsAdded) : e.kind === 'start' ? fmtInt(e.rows) : ''])),
988
  });
 
1001
  });
1002
  if (IX.overrides.catalog || IX.overrides.vocabulary) {
1003
  const chip = $('preview-chip');
1004
+ clear(chip).append(el('span', { class: 'dot other', 'aria-hidden': 'true' }), t('hero.preview'));
1005
  chip.title = [IX.overrides.catalog, IX.overrides.vocabulary].filter(Boolean).join(' ');
1006
  chip.hidden = false;
1007
  }
1008
  }
1009
+ // the issue a request link opens is written in English, the language of the hub
1010
  function updateRequestLinks() {
1011
+ const chips = filterChips(tEn);
1012
  const u = new URL(`${GITHUB}/issues/new`);
1013
  u.searchParams.set('template', 'data_request.yml');
1014
+ if (chips.length) u.searchParams.set('what', tEn('request.like', { chips: chips.map(([k, v]) => `${k}: ${v}`).join('; ') }));
1015
  for (const a of document.querySelectorAll('.request-link')) a.href = u.toString();
1016
  }
1017
  function progress() {
 
1052
  updateRequestLinks();
1053
  }
1054
 
1055
+ // the note above the page (a part of the index not read, or the index not read at all), kept so a new language
1056
+ // writes it again: {key, values, problem}
1057
+ let notice = null;
1058
+ function drawNotice() {
1059
+ if (!notice) return;
1060
+ const n = $('notice');
1061
+ clear(n).append(notice.problem ? el('span', { class: 'dot problem', 'aria-hidden': 'true' }) : null, t(notice.key, notice.values));
1062
+ n.hidden = false;
1063
+ }
1064
+
1065
+ /** The page in the language just chosen: the words of index.html, then every part drawn from the index. */
1066
+ function renderLanguage() {
1067
+ translatePage();
1068
+ drawNotice();
1069
+ if (!IX) return;
1070
+ hideTip();
1071
+ renderHeader();
1072
+ renderCategories();
1073
+ renderReady();
1074
+ fillFilterOptions();
1075
+ renderScoped();
1076
+ renderExcluded();
1077
+ renderHistory();
1078
+ }
1079
+
1080
  async function main() {
1081
  readUrl();
1082
+ translatePage();
1083
+ setUpSwitch();
1084
+ onLanguage(renderLanguage);
1085
  decorate();
1086
  reveal();
1087
  let ticking = false;
 
1091
  addEventListener('resize', onScroll);
1092
  progress();
1093
  try {
1094
+ IX = await loadIndex(new URLSearchParams(location.search), (key, values) => { notice = { key, values }; drawNotice(); });
1095
  } catch (e) {
1096
+ notice = { key: 'notice.failed', values: { error: e.message }, problem: true };
1097
+ drawNotice();
 
1098
  console.error(e);
1099
  return;
1100
  }
charts.js CHANGED
@@ -1,5 +1,6 @@
1
  // Small chart kit: elements, number formats, tooltip, chart/table toggle, thin bars, heatmap.
2
  // Every string from the data goes in with textContent (through el), never innerHTML.
 
3
 
4
  export function el(tag, attrs = {}, ...kids) {
5
  const n = document.createElement(tag);
@@ -20,41 +21,52 @@ export function el(tag, attrs = {}, ...kids) {
20
 
21
  export const clear = (n) => { while (n.firstChild) n.firstChild.remove(); return n; };
22
 
23
- // ---- numbers
24
- const intFmt = new Intl.NumberFormat('en-US');
25
- export const fmtInt = (v) => (v == null ? 'unknown' : intFmt.format(Math.round(v)));
 
 
 
 
 
 
 
26
  export function fmtCompact(v) {
27
- if (v == null) return 'unknown';
28
- const a = Math.abs(v);
29
- const three = (x) => (x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2));
30
- if (a >= 1e9) return `${three(v / 1e9)} billion`;
31
- if (a >= 1e6) return `${three(v / 1e6)} million`;
32
- if (a >= 1e4) return `${three(v / 1e3)} thousand`;
33
- return intFmt.format(v);
34
  }
35
- export function fmtShort(v) {
36
- if (v == null) return '';
37
- const a = Math.abs(v);
38
- const two = (x) => (x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(0) : x.toFixed(1));
39
- if (a >= 1e9) return `${two(v / 1e9)}B`;
40
- if (a >= 1e6) return `${two(v / 1e6)}M`;
41
- if (a >= 1e3) return `${two(v / 1e3)}K`;
42
- return String(Math.round(v));
43
  }
 
44
  export function fmtBytes(v) {
45
- if (v == null) return 'unknown';
46
  const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB'];
47
  let i = 0;
48
  let x = v;
49
  while (x >= 1000 && i < units.length - 1) { x /= 1000; i++; }
50
- const s = i === 0 ? String(x) : x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
51
  return `${s} ${units[i]}`;
52
  }
53
  export const fmtPct = (part, whole) => {
54
- if (!whole) return '0%';
55
- const p = (100 * part) / whole;
56
- return `${p > 0 && p < 1 ? '<1' : Math.round(p)}%`;
 
57
  };
 
 
 
 
 
 
 
 
58
 
59
  // ---- tooltip: the value leads, the label follows; also shown on keyboard focus
60
  const tip = () => document.getElementById('tip');
@@ -106,7 +118,7 @@ export function figureWithToggle(fig, key, title, views, extra) {
106
  clear(fig).append(
107
  el('div', { class: 'chart-head' },
108
  el('p', { class: 'chart-title' }, title),
109
- segmented('View', [['chart', 'Chart'], ['table', 'Table']], mode, (v) => { viewState.set(key, v); vt(draw); })),
110
  mode === 'chart' ? views.chart() : el('div', { class: 'table-wrap' }, views.table()),
111
  );
112
  const more = extra ? extra() : null;
@@ -177,7 +189,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
177
  for (const r of rows) for (const c of cols) vals.push(value(r, c));
178
  const scale = logBins(vals);
179
  const maxV = Math.max(0, ...vals);
180
- const grid = el('div', { class: 'heat', role: 'grid', 'aria-label': `${metricName} by domain and form`,
181
  style: { gridTemplateColumns: `minmax(min-content, 10rem) repeat(${cols.length}, minmax(22px, 3rem))`,
182
  gridTemplateRows: `auto repeat(${rows.length}, minmax(34px, 1fr))` } });
183
  grid.append(el('div', { class: 'corner', role: 'presentation' }));
@@ -196,7 +208,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
196
  const isCur = current && current.row === r.key && current.col === c.key;
197
  const cell = el('button', {
198
  type: 'button', class: `cell${fill ? '' : ' empty'}${isCur ? ' current' : ''}`, role: 'gridcell', tabindex: '-1',
199
- 'aria-label': `${r.label}, ${c.label}: ${fmt(v)}`, 'aria-pressed': isCur ? 'true' : 'false',
200
  style: fill ? { background: fill } : null,
201
  onclick: () => onPick(r, c),
202
  });
@@ -234,7 +246,7 @@ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName })
234
  const ticks = el('span', { class: 'ticks', 'aria-hidden': 'true' },
235
  el('span', {}, '0'), scale.edges.slice(0, RAMP.length).map((e) => el('span', {}, fmtShort(e))));
236
  const legend = el('div', { class: 'scale' },
237
- el('span', {}, `${metricName}, log scale`),
238
  el('span', {}, steps, ticks));
239
  return { grid, legend };
240
  }
 
1
  // Small chart kit: elements, number formats, tooltip, chart/table toggle, thin bars, heatmap.
2
  // Every string from the data goes in with textContent (through el), never innerHTML.
3
+ import { locale, t } from './i18n.js';
4
 
5
  export function el(tag, attrs = {}, ...kids) {
6
  const n = document.createElement(tag);
 
21
 
22
  export const clear = (n) => { while (n.firstChild) n.firstChild.remove(); return n; };
23
 
24
+ // ---- numbers and dates, by Intl in the current language
25
+ const formats = new Map();
26
+ const nf = (opts = {}) => {
27
+ const k = `${locale()} ${JSON.stringify(opts)}`;
28
+ if (!formats.has(k)) formats.set(k, new Intl.NumberFormat(locale(), opts));
29
+ return formats.get(k);
30
+ };
31
+ // three significant digits and a scale word: 8.97 million in English, 897 and the sign for ten thousand in Chinese
32
+ const COMPACT = { notation: 'compact', compactDisplay: 'long', minimumSignificantDigits: 3, maximumSignificantDigits: 3 };
33
+ export const fmtInt = (v) => (v == null ? t('num.unknown') : nf().format(Math.round(v)));
34
  export function fmtCompact(v) {
35
+ if (v == null) return t('num.unknown');
36
+ return Math.abs(v) >= 1e4 ? nf(COMPACT).format(v) : nf().format(v);
 
 
 
 
 
37
  }
38
+ /** fmtCompact split into its digits and its scale word (8970000 -> ['8.97', 'million'] in English); small numbers
39
+ * whole, with no scale word. */
40
+ export function compactParts(v) {
41
+ if (Math.abs(v) < 1e4) return [fmtInt(v), ''];
42
+ const parts = nf(COMPACT).formatToParts(v);
43
+ const join = (keep) => parts.filter((p) => keep(p.type)).map((p) => p.value).join('');
44
+ return [join((k) => k !== 'compact' && k !== 'literal'), join((k) => k === 'compact')];
 
45
  }
46
+ export const fmtShort = (v) => (v == null ? '' : nf({ notation: 'compact' }).format(v));
47
  export function fmtBytes(v) {
48
+ if (v == null) return t('num.unknown');
49
  const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB'];
50
  let i = 0;
51
  let x = v;
52
  while (x >= 1000 && i < units.length - 1) { x /= 1000; i++; }
53
+ const s = i === 0 ? nf().format(x) : nf({ minimumSignificantDigits: 3, maximumSignificantDigits: 3 }).format(x);
54
  return `${s} ${units[i]}`;
55
  }
56
  export const fmtPct = (part, whole) => {
57
+ const pct = (p) => nf({ style: 'percent', maximumFractionDigits: 0 }).format(p);
58
+ if (!whole) return pct(0);
59
+ const p = part / whole;
60
+ return p > 0 && p < 0.01 ? `<${pct(0.01)}` : pct(p);
61
  };
62
+ /** A date of the index (YYYY-MM-DD at the start of an ISO time), in UTC; opts: Intl.DateTimeFormat options. */
63
+ export function fmtDate(iso, opts = { day: 'numeric', month: 'short', year: 'numeric' }) {
64
+ if (!iso) return '';
65
+ const d = new Date(`${iso.slice(0, 10)}T00:00:00Z`);
66
+ return Number.isNaN(d.getTime()) ? iso.slice(0, 10) : new Intl.DateTimeFormat(locale(), { ...opts, timeZone: 'UTC' }).format(d);
67
+ }
68
+ /** Alternatives as one phrase: a, b or c. */
69
+ export const fmtOr = (list) => new Intl.ListFormat(locale(), { type: 'disjunction' }).format(list);
70
 
71
  // ---- tooltip: the value leads, the label follows; also shown on keyboard focus
72
  const tip = () => document.getElementById('tip');
 
118
  clear(fig).append(
119
  el('div', { class: 'chart-head' },
120
  el('p', { class: 'chart-title' }, title),
121
+ segmented(t('chart.view'), [['chart', t('chart.chart')], ['table', t('chart.table')]], mode, (v) => { viewState.set(key, v); vt(draw); })),
122
  mode === 'chart' ? views.chart() : el('div', { class: 'table-wrap' }, views.table()),
123
  );
124
  const more = extra ? extra() : null;
 
189
  for (const r of rows) for (const c of cols) vals.push(value(r, c));
190
  const scale = logBins(vals);
191
  const maxV = Math.max(0, ...vals);
192
+ const grid = el('div', { class: 'heat', role: 'grid', 'aria-label': t('heat.aria', { metric: metricName }),
193
  style: { gridTemplateColumns: `minmax(min-content, 10rem) repeat(${cols.length}, minmax(22px, 3rem))`,
194
  gridTemplateRows: `auto repeat(${rows.length}, minmax(34px, 1fr))` } });
195
  grid.append(el('div', { class: 'corner', role: 'presentation' }));
 
208
  const isCur = current && current.row === r.key && current.col === c.key;
209
  const cell = el('button', {
210
  type: 'button', class: `cell${fill ? '' : ' empty'}${isCur ? ' current' : ''}`, role: 'gridcell', tabindex: '-1',
211
+ 'aria-label': t('heat.cell', { row: r.label, col: c.label, value: fmt(v) }), 'aria-pressed': isCur ? 'true' : 'false',
212
  style: fill ? { background: fill } : null,
213
  onclick: () => onPick(r, c),
214
  });
 
246
  const ticks = el('span', { class: 'ticks', 'aria-hidden': 'true' },
247
  el('span', {}, '0'), scale.edges.slice(0, RAMP.length).map((e) => el('span', {}, fmtShort(e))));
248
  const legend = el('div', { class: 'scale' },
249
+ el('span', {}, t('heat.scale', { metric: metricName })),
250
  el('span', {}, steps, ticks));
251
  return { grid, legend };
252
  }
data.js CHANGED
@@ -1,6 +1,7 @@
1
  // Reading the index in the browser. Every file comes from one commit of the dataset, so every number agrees.
2
  import { asyncBufferFromUrl, parquetMetadataAsync, parquetReadObjects } from 'https://cdn.jsdelivr.net/npm/hyparquet@1.31.5/+esm';
3
  import { compressors } from 'https://cdn.jsdelivr.net/npm/hyparquet-compressors@1.1.2/+esm';
 
4
 
5
  export const REPO = 'Battam/agentic-env-index';
6
  export const HF = 'https://huggingface.co';
@@ -47,7 +48,7 @@ export async function readColumns(url, cols, range = {}) {
47
  /** The commit to read: ?rev=<sha or branch>, else main. Returns {sha, date, files}. */
48
  export async function loadRevision(rev) {
49
  const j = await getJson(`${HF}/api/datasets/${REPO}/revision/${encodeURIComponent(rev || 'main')}`);
50
- if (!/^[0-9a-f]{40}$/.test(j.sha || '')) throw new Error('the Hub gave no commit id');
51
  return { sha: j.sha, date: j.lastModified || null, files: (j.siblings || []).map((s) => s.rfilename) };
52
  }
53
 
@@ -194,7 +195,8 @@ async function readCatalog(url) {
194
  return readParquet(url);
195
  }
196
 
197
- /** Everything the page shows, from one commit. params: URLSearchParams (rev, catalog, vocabulary). */
 
198
  export async function loadIndex(params, onNote) {
199
  const rev = await loadRevision(params.get('rev'));
200
  const at = (p) => fileUrl(rev.sha, p);
@@ -210,7 +212,7 @@ export async function loadIndex(params, onNote) {
210
  readParquet(at('meta/not_included.parquet')).catch(() => []),
211
  getText(at('CHANGELOG.md')).catch(() => ''),
212
  getText(at('README.md')).catch(() => ''),
213
- vocabUrl ? getJson(vocabUrl).catch((e) => { onNote?.(`The vocabulary could not be read (${e.message}); codes are shown as they are.`); return null; }) : null,
214
  ]);
215
 
216
  const vocab = parseVocabulary(vocabJson || {});
 
1
  // Reading the index in the browser. Every file comes from one commit of the dataset, so every number agrees.
2
  import { asyncBufferFromUrl, parquetMetadataAsync, parquetReadObjects } from 'https://cdn.jsdelivr.net/npm/hyparquet@1.31.5/+esm';
3
  import { compressors } from 'https://cdn.jsdelivr.net/npm/hyparquet-compressors@1.1.2/+esm';
4
+ import { t } from './i18n.js';
5
 
6
  export const REPO = 'Battam/agentic-env-index';
7
  export const HF = 'https://huggingface.co';
 
48
  /** The commit to read: ?rev=<sha or branch>, else main. Returns {sha, date, files}. */
49
  export async function loadRevision(rev) {
50
  const j = await getJson(`${HF}/api/datasets/${REPO}/revision/${encodeURIComponent(rev || 'main')}`);
51
+ if (!/^[0-9a-f]{40}$/.test(j.sha || '')) throw new Error(t('error.noCommit'));
52
  return { sha: j.sha, date: j.lastModified || null, files: (j.siblings || []).map((s) => s.rfilename) };
53
  }
54
 
 
195
  return readParquet(url);
196
  }
197
 
198
+ /** Everything the page shows, from one commit. params: URLSearchParams (rev, catalog, vocabulary); onNote(key, values):
199
+ * a note for the reader, as a key of the locale files. */
200
  export async function loadIndex(params, onNote) {
201
  const rev = await loadRevision(params.get('rev'));
202
  const at = (p) => fileUrl(rev.sha, p);
 
212
  readParquet(at('meta/not_included.parquet')).catch(() => []),
213
  getText(at('CHANGELOG.md')).catch(() => ''),
214
  getText(at('README.md')).catch(() => ''),
215
+ vocabUrl ? getJson(vocabUrl).catch((e) => { onNote?.('notice.vocabulary', { error: e.message }); return null; }) : null,
216
  ]);
217
 
218
  const vocab = parseVocabulary(vocabJson || {});
first.js CHANGED
@@ -1,7 +1,8 @@
1
  // The first screen, drawn before the index is read: the pictograms, and the hero from first.json (written by the deploy
2
  // step from the index commit of the day). app.js redraws the hero from the index itself when it has read it; the numbers
3
  // are the same when both name the same index commit (portal.py first_screen and data.js count alike).
4
- import { fmtBytes, fmtCompact, fmtInt } from './charts.js';
 
5
  import { icon } from './icons.js';
6
 
7
  const reduced = () => matchMedia('(prefers-reduced-motion: reduce)').matches;
@@ -33,27 +34,23 @@ export function setNumber(node, key, value, fmt) {
33
  node.addEventListener('animationend', () => node.classList.remove('updated'), { once: true });
34
  }
35
 
36
- function dateText(iso) {
37
- if (!iso) return '';
38
- const d = new Date(`${iso.slice(0, 10)}T00:00:00Z`);
39
- return Number.isNaN(d.getTime()) ? iso.slice(0, 10) : d.toLocaleDateString('en-GB', { day: 'numeric', month: 'short', year: 'numeric', timeZone: 'UTC' });
40
- }
41
-
42
  /** The size line (#364): distinct content and the rows without a size, not in it; the words of the index README,
43
  * section Size. */
44
  function drawSize(v) {
45
  const node = document.getElementById('hero-size');
46
  if (!node || !('distinct_bytes' in v)) return;
47
- if (v.distinct_bytes == null) { node.replaceChildren('Distinct content not known yet'); return; }
48
  const big = document.createElement('b');
49
- const unsized = v.unsized_rows ? ` · ${fmtInt(v.unsized_rows)} rows have no size and are not in that number` : '';
50
- node.replaceChildren(big, ` distinct content${unsized}`);
51
  setNumber(big, 'distinct_bytes', v.distinct_bytes, fmtBytes);
52
  }
53
 
 
54
  /** v: {ready_units, ready_sources, answers, raw_rows, sources, distinct_bytes, unsized_rows, index, date}. */
55
  export function drawHero(v) {
56
  if (!v) return;
 
57
  setNumber(document.getElementById('hero-big'), 'ready_units', v.ready_units, fmtInt);
58
  setNumber(document.getElementById('material-n'), 'material_units', v.ready_units, fmtInt);
59
  drawSize(v);
@@ -63,15 +60,17 @@ export function drawHero(v) {
63
  }
64
  const pin = document.getElementById('pin-link');
65
  if (v.index && pin) {
66
- pin.textContent = dateText(v.date);
67
  pin.href = `https://huggingface.co/datasets/Battam/agentic-env-index/tree/${v.index}`;
68
- pin.title = `Commit ${v.index}`;
69
  }
70
  const foot = document.getElementById('foot-pin');
71
- if (v.index && foot) foot.textContent = `Index ${v.index.slice(0, 7)} · ${dateText(v.date)}`;
72
  }
73
 
74
  decorate();
 
 
75
  let drawnFromIndex = false;
76
  export const markFromIndex = () => { drawnFromIndex = true; };
77
  // first.json describes main of the published index; with ?rev=, ?catalog= or ?vocabulary= the page shows other data, so
 
1
  // The first screen, drawn before the index is read: the pictograms, and the hero from first.json (written by the deploy
2
  // step from the index commit of the day). app.js redraws the hero from the index itself when it has read it; the numbers
3
  // are the same when both name the same index commit (portal.py first_screen and data.js count alike).
4
+ import { fmtBytes, fmtCompact, fmtDate, fmtInt } from './charts.js';
5
+ import { onLanguage, t } from './i18n.js';
6
  import { icon } from './icons.js';
7
 
8
  const reduced = () => matchMedia('(prefers-reduced-motion: reduce)').matches;
 
34
  node.addEventListener('animationend', () => node.classList.remove('updated'), { once: true });
35
  }
36
 
 
 
 
 
 
 
37
  /** The size line (#364): distinct content and the rows without a size, not in it; the words of the index README,
38
  * section Size. */
39
  function drawSize(v) {
40
  const node = document.getElementById('hero-size');
41
  if (!node || !('distinct_bytes' in v)) return;
42
+ if (v.distinct_bytes == null) { node.replaceChildren(t('hero.sizeUnknown')); return; }
43
  const big = document.createElement('b');
44
+ const unsized = v.unsized_rows ? ` · ${t('hero.unsized', { count: v.unsized_rows, n: fmtInt(v.unsized_rows) })}` : '';
45
+ node.replaceChildren(big, ` ${t('hero.distinct')}${unsized}`);
46
  setNumber(big, 'distinct_bytes', v.distinct_bytes, fmtBytes);
47
  }
48
 
49
+ let last = null;
50
  /** v: {ready_units, ready_sources, answers, raw_rows, sources, distinct_bytes, unsized_rows, index, date}. */
51
  export function drawHero(v) {
52
  if (!v) return;
53
+ last = v;
54
  setNumber(document.getElementById('hero-big'), 'ready_units', v.ready_units, fmtInt);
55
  setNumber(document.getElementById('material-n'), 'material_units', v.ready_units, fmtInt);
56
  drawSize(v);
 
60
  }
61
  const pin = document.getElementById('pin-link');
62
  if (v.index && pin) {
63
+ pin.textContent = fmtDate(v.date);
64
  pin.href = `https://huggingface.co/datasets/Battam/agentic-env-index/tree/${v.index}`;
65
+ pin.title = t('hero.commit', { sha: v.index });
66
  }
67
  const foot = document.getElementById('foot-pin');
68
+ if (v.index && foot) foot.textContent = t('foot.index', { sha: v.index.slice(0, 7), date: fmtDate(v.date) });
69
  }
70
 
71
  decorate();
72
+ // the hero in the new language: the same numbers, written again
73
+ onLanguage(() => drawHero(last));
74
  let drawnFromIndex = false;
75
  export const markFromIndex = () => { drawnFromIndex = true; };
76
  // first.json describes main of the published index; with ?rev=, ?catalog= or ?vocabulary= the page shows other data, so
first.json CHANGED
@@ -1,13 +1,13 @@
1
  {
2
  "answers": 590,
3
  "date": "2026-10-29",
4
- "distinct_bytes": 3786142106650529,
5
  "format": "portal-first/1",
6
- "index": "75e29bbc75408ca6b9a6097bd5227763860bd43a",
7
- "raw_rows": 1148924572,
8
  "raw_sources": 89,
9
- "ready_sources": 1779,
10
- "ready_units": 3952,
11
- "sources": 1868,
12
- "unsized_rows": 281312740
13
  }
 
1
  {
2
  "answers": 590,
3
  "date": "2026-10-29",
4
+ "distinct_bytes": 3786223653585058,
5
  "format": "portal-first/1",
6
+ "index": "02b52f1fb8ec44ec91c6593f759faff0be42b200",
7
+ "raw_rows": 1150137524,
8
  "raw_sources": 89,
9
+ "ready_sources": 1783,
10
+ "ready_units": 3962,
11
+ "sources": 1872,
12
+ "unsized_rows": 281582670
13
  }
i18n.js ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // The page's languages, English and Simplified Chinese, by i18next; the reader's choice by its browser language
2
+ // detector: the query string first (?lng=zh), then localStorage, and kept in localStorage. Every word the page writes
3
+ // comes from t by key (locales/en.json, locales/zh.json); text read from the dataset stays as it is.
4
+ // Standard tool: the page's languages by i18next, with i18next-browser-languagedetector for the choice and its memory.
5
+ import i18next from 'https://cdn.jsdelivr.net/npm/i18next@26.4.5/+esm';
6
+ import LanguageDetector from 'https://cdn.jsdelivr.net/npm/i18next-browser-languagedetector@8.2.1/+esm';
7
+
8
+ const LANGS = ['en', 'zh'];
9
+ const HTML_LANG = { en: 'en', zh: 'zh-CN' };
10
+
11
+ const resources = Object.fromEntries(await Promise.all(LANGS.map(async (lng) => {
12
+ const r = await fetch(new URL(`locales/${lng}.json`, import.meta.url));
13
+ return [lng, { translation: await r.json() }];
14
+ })));
15
+
16
+ await i18next.use(LanguageDetector).init({
17
+ resources,
18
+ fallbackLng: 'en',
19
+ supportedLngs: LANGS,
20
+ // flat keys: a key may hold a code with dots (domain.D10.2) or colons
21
+ keySeparator: false,
22
+ nsSeparator: false,
23
+ // every text goes into the page with textContent, never innerHTML
24
+ interpolation: { escapeValue: false },
25
+ detection: { order: ['querystring', 'localStorage'], lookupQuerystring: 'lng', caches: ['localStorage'] },
26
+ });
27
+
28
+ export const t = (...args) => i18next.t(...args);
29
+ /** The words of the issue texts the page prefills: the hub writes English. */
30
+ export const tEn = i18next.getFixedT('en');
31
+ export const lang = () => i18next.resolvedLanguage;
32
+ /** The current language as a BCP 47 tag, for <html lang> and Intl. */
33
+ export const locale = () => HTML_LANG[lang()];
34
+ export const onLanguage = (fn) => i18next.on('languageChanged', fn);
35
+
36
+ /** The page's own words in index.html (its <title> too): [data-i18n] sets the text, [data-i18n-attr] attributes
37
+ * ("name:key;name:key"); <html lang> and the language switch. */
38
+ export function translatePage() {
39
+ document.documentElement.lang = locale();
40
+ for (const n of document.querySelectorAll('[data-i18n]')) n.textContent = t(n.dataset.i18n);
41
+ for (const n of document.querySelectorAll('[data-i18n-attr]')) {
42
+ for (const pair of n.dataset.i18nAttr.split(';')) {
43
+ const [name, key] = pair.split(':');
44
+ n.setAttribute(name, t(key));
45
+ }
46
+ }
47
+ const other = LANGS.find((l) => l !== lang());
48
+ const button = document.getElementById('lang-switch');
49
+ button.textContent = i18next.getFixedT(other)('lang.name');
50
+ button.lang = HTML_LANG[other];
51
+ button.title = t('lang.switch');
52
+ button.hidden = false;
53
+ }
54
+
55
+ /** The switch: the other language, without a reload. A ?lng= in the address goes, so the choice kept in localStorage
56
+ * holds on the next visit. */
57
+ export function setUpSwitch() {
58
+ document.getElementById('lang-switch').addEventListener('click', () => {
59
+ const p = new URLSearchParams(location.search);
60
+ if (p.has('lng')) {
61
+ p.delete('lng');
62
+ const s = p.toString();
63
+ history.replaceState(null, '', s ? `?${s}` : location.pathname);
64
+ }
65
+ i18next.changeLanguage(LANGS.find((l) => l !== lang()));
66
+ });
67
+ }
index.html CHANGED
@@ -3,33 +3,37 @@
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
- <title>Agentic Env Catalog</title>
7
  <link rel="icon" href="data:,">
8
- <link rel="alternate" type="text/plain" href="llms.txt" title="For agents">
9
- <meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it.">
10
  <meta name="theme-color" content="#112532">
11
  <link rel="preconnect" href="https://fonts.googleapis.com">
12
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
13
  <link rel="preconnect" href="https://huggingface.co">
14
  <link rel="preload" href="first.json" as="fetch" crossorigin>
 
 
15
  <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=IBM+Plex+Mono:wght@500&family=Noto+Sans+SC:wght@400;500;700&display=swap">
16
  <link rel="stylesheet" href="style.css">
17
  <script type="module" src="first.js" async></script>
18
  <script type="module" src="app.js"></script>
19
  </head>
20
  <body>
21
- <nav class="topnav" aria-label="Page sections">
22
  <div class="wrap nav-row">
23
- <a class="brand" href="#top" data-icon="layers" data-size="22"><span>Agentic Env Catalog</span></a>
24
  <ol class="progress">
25
- <li><a href="#material" data-part="overview" data-icon="overview"><span class="pl">Overview</span><span class="track"><span class="fill"></span></span></a></li>
26
- <li><a href="#map" data-part="browse" data-icon="browse"><span class="pl">Browse</span><span class="track"><span class="fill"></span></span></a></li>
27
- <li><a href="#get" data-part="get" data-icon="get"><span class="pl">Get the data</span><span class="track"><span class="fill"></span></span></a></li>
28
- <li><a href="#excluded" data-part="excluded" data-icon="excluded"><span class="pl">Not included</span><span class="track"><span class="fill"></span></span></a></li>
29
  </ol>
30
  <div class="nav-links">
31
- <a href="llms.txt" data-icon="robot" data-size="16">For agents</a>
32
- <a class="btn solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
 
 
33
  </div>
34
  </div>
35
  </nav>
@@ -38,30 +42,30 @@
38
  <header class="hero" id="top" data-part="overview">
39
  <div class="wrap hero-grid">
40
  <div class="hero-main">
41
- <p class="eyebrow" data-icon="sources" data-size="18"><span>The Agentic Env Index, by category</span></p>
42
- <h1>Real material for agent tasks</h1>
43
  <div class="hero-number">
44
  <span class="display" id="hero-big"><span class="sk sk-display"></span></span>
45
- <span class="display-label">task-ready units</span>
46
  </div>
47
  <p class="hero-size" id="hero-size"><span class="sk sk-line"></span></p>
48
  <div class="hero-cta">
49
- <a class="btn hero-btn" href="#material" data-icon="browse" data-size="18">Browse categories</a>
50
- <a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18">Get the data</a>
51
  </div>
52
  </div>
53
- <ul class="hero-facts" id="hero-facts" aria-label="Facts">
54
- <li class="fact-tile" data-k="ready_sources"><span class="fact-ico" data-icon="check" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">sources ready</span></li>
55
- <li class="fact-tile" data-k="answers"><span class="fact-ico" data-icon="target" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">units with an answer</span></li>
56
- <li class="fact-tile" data-k="raw_rows"><span class="fact-ico" data-icon="rows" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">raw rows</span></li>
57
- <li class="fact-tile" data-k="sources"><span class="fact-ico" data-icon="database" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label">sources indexed</span></li>
58
  </ul>
59
  </div>
60
  <div class="wrap hero-foot">
61
- <span class="hero-pin">Index of <a id="pin-link" href="https://huggingface.co/datasets/Battam/agentic-env-index"><span class="sk sk-word"></span></a></span>
62
  <span class="chip chip-dark" id="preview-chip" hidden></span>
63
- <a href="https://huggingface.co/datasets/Battam/agentic-env-index">Dataset</a>
64
- <a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md">Catalog README</a>
65
  <span class="notice" id="notice" hidden></span>
66
  </div>
67
  </header>
@@ -69,14 +73,14 @@
69
  <section class="band band-canvas" id="material" data-part="overview" aria-labelledby="material-h">
70
  <div class="wrap">
71
  <div class="band-head reveal">
72
- <h2 id="material-h">Where the <span id="material-n"><span class="sk sk-num"></span></span> task-ready units are</h2>
73
  </div>
74
  <!-- two cards: the task-ready units stacked four ways, and the largest task-ready sources; the skeletons have the final cards' sizes, so nothing moves when the index arrives -->
75
  <div class="where-grid">
76
  <figure class="card chart-card where-card" id="ready-card"><div class="sk sk-block"></div></figure>
77
  <figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
78
  </div>
79
- <div class="band-head sub-head reveal"><h3 id="cats-h">Most requested categories</h3></div>
80
  <!-- one card per most requested category (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
81
  <ul class="cat-grid" id="req-cards">
82
  <li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
@@ -86,41 +90,41 @@
86
 
87
  <div class="band band-white" id="browse" data-part="browse">
88
  <div class="wrap browse-grid">
89
- <aside class="filters" id="filters" aria-label="Filters">
90
  <button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row" data-icon="pick" data-size="18">
91
- <span>Filters</span><span class="state" id="filter-state">none on</span>
92
  </button>
93
  <div class="filter-row" id="filter-row">
94
- <p class="filter-title" data-icon="pick" data-size="18"><span>Filters</span></p>
95
- <label>Domain<select id="f-domain"><option value="">All</option></select></label>
96
- <label>Subdomain<select id="f-leaf" disabled><option value="">All</option></select></label>
97
- <label>Form<select id="f-form"><option value="">All</option></select></label>
98
- <label>Kind<select id="f-status">
99
- <option value="">Any</option>
100
- <option value="ready">Task-ready</option>
101
- <option value="raw">Raw corpus</option>
102
- <option value="flagged">Flagged</option>
103
  </select></label>
104
- <label>Training<select id="f-train">
105
- <option value="">Any</option>
106
- <option value="yes">Allowed, all rows</option>
107
- <option value="cond">Allowed or conditional</option>
108
  </select></label>
109
- <label>Shipping<select id="f-redist">
110
- <option value="">Any</option>
111
- <option value="yes">Allowed, all rows</option>
112
- <option value="cond">Allowed or conditional</option>
113
  </select></label>
114
- <label>Licence tier<select id="f-tier"><option value="">All</option></select></label>
115
- <label>Source size<select id="f-size">
116
- <option value="">Any</option>
117
- <option value="lt1g">Under 1 GB</option>
118
- <option value="1g1t">1 GB to 1 TB</option>
119
- <option value="ge1t">1 TB or more</option>
120
- <option value="unknown">Not published</option>
121
  </select></label>
122
- <label class="wide">Search<input id="f-q" type="search" placeholder="Title or text" autocomplete="off"></label>
123
- <div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled>Clear</button></div>
124
  </div>
125
  </aside>
126
 
@@ -128,11 +132,11 @@
128
  <section id="map" aria-labelledby="map-h">
129
  <div class="band-head row reveal">
130
  <h2 id="map-h"><span class="sk sk-line"></span></h2>
131
- <span class="seg-wrap"><span class="seg-label" id="metric-label">Count</span>
132
  <span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
133
- <button type="button" data-metric="sources" aria-pressed="true">Sources</button>
134
- <button type="button" data-metric="rows" aria-pressed="false">Pointer rows</button>
135
- <button type="button" data-metric="bytes" aria-pressed="false">Size</button>
136
  </span>
137
  </span>
138
  </div>
@@ -160,12 +164,12 @@
160
  <div class="card agents-card lift reveal" id="agents">
161
  <span class="agents-ico" data-icon="robot" data-size="36"></span>
162
  <div class="agents-text">
163
- <h3 id="agents-h">For agents</h3>
164
- <p class="small">Catalog columns · the one command · the default profile · where answers land</p>
165
  </div>
166
  <div class="agents-actions">
167
  <a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a>
168
- <a class="btn" href="https://huggingface.co/datasets/Battam/agentic-env-index#for-agents" data-icon="doc" data-size="16">README: For agents</a>
169
  </div>
170
  </div>
171
  </div>
@@ -187,16 +191,16 @@
187
  <div class="foot-cta">
188
  <span class="foot-ico" data-icon="request" data-size="32"></span>
189
  <div>
190
- <h2>Missing material?</h2>
191
- <p>Ask for a category, a form or a source.</p>
192
  </div>
193
- <a class="btn on-dark-solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
194
  </div>
195
  <ul class="foot-links">
196
- <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index">Dataset</a></li>
197
- <li><a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md">Catalog README</a></li>
198
- <li><a href="llms.txt">For agents</a></li>
199
- <li><a href="#top">Back to top</a></li>
200
  </ul>
201
  <p class="foot-pin mono" id="foot-pin"></p>
202
  </div>
 
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title data-i18n="page.title">Agentic Env Catalog</title>
7
  <link rel="icon" href="data:,">
8
+ <link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents">
9
+ <meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it." data-i18n-attr="content:page.description">
10
  <meta name="theme-color" content="#112532">
11
  <link rel="preconnect" href="https://fonts.googleapis.com">
12
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
13
  <link rel="preconnect" href="https://huggingface.co">
14
  <link rel="preload" href="first.json" as="fetch" crossorigin>
15
+ <link rel="preload" href="locales/en.json" as="fetch" crossorigin>
16
+ <link rel="preload" href="locales/zh.json" as="fetch" crossorigin>
17
  <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=IBM+Plex+Mono:wght@500&family=Noto+Sans+SC:wght@400;500;700&display=swap">
18
  <link rel="stylesheet" href="style.css">
19
  <script type="module" src="first.js" async></script>
20
  <script type="module" src="app.js"></script>
21
  </head>
22
  <body>
23
+ <nav class="topnav" aria-label="Page sections" data-i18n-attr="aria-label:nav.sections">
24
  <div class="wrap nav-row">
25
+ <a class="brand" href="#top" data-icon="layers" data-size="22"><span data-i18n="page.title">Agentic Env Catalog</span></a>
26
  <ol class="progress">
27
+ <li><a href="#material" data-part="overview" data-icon="overview"><span class="pl" data-i18n="nav.overview">Overview</span><span class="track"><span class="fill"></span></span></a></li>
28
+ <li><a href="#map" data-part="browse" data-icon="browse"><span class="pl" data-i18n="nav.browse">Browse</span><span class="track"><span class="fill"></span></span></a></li>
29
+ <li><a href="#get" data-part="get" data-icon="get"><span class="pl" data-i18n="nav.get">Get the data</span><span class="track"><span class="fill"></span></span></a></li>
30
+ <li><a href="#excluded" data-part="excluded" data-icon="excluded"><span class="pl" data-i18n="nav.excluded">Not included</span><span class="track"><span class="fill"></span></span></a></li>
31
  </ol>
32
  <div class="nav-links">
33
+ <a href="llms.txt" data-icon="robot" data-size="16"><span data-i18n="nav.agents">For agents</span></a>
34
+ <a class="btn solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml" data-i18n="nav.request">Request data</a>
35
+ <!-- the language switch: the other language's name, in that language (i18n.js) -->
36
+ <button class="btn lang-btn" id="lang-switch" type="button" hidden></button>
37
  </div>
38
  </div>
39
  </nav>
 
42
  <header class="hero" id="top" data-part="overview">
43
  <div class="wrap hero-grid">
44
  <div class="hero-main">
45
+ <p class="eyebrow" data-icon="sources" data-size="18"><span data-i18n="hero.eyebrow">The Agentic Env Index, by category</span></p>
46
+ <h1 data-i18n="hero.title">Real material for agent tasks</h1>
47
  <div class="hero-number">
48
  <span class="display" id="hero-big"><span class="sk sk-display"></span></span>
49
+ <span class="display-label" data-i18n="hero.units">task-ready units</span>
50
  </div>
51
  <p class="hero-size" id="hero-size"><span class="sk sk-line"></span></p>
52
  <div class="hero-cta">
53
+ <a class="btn hero-btn" href="#material" data-icon="browse" data-size="18"><span data-i18n="hero.browse">Browse categories</span></a>
54
+ <a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18"><span data-i18n="nav.get">Get the data</span></a>
55
  </div>
56
  </div>
57
+ <ul class="hero-facts" id="hero-facts" aria-label="Facts" data-i18n-attr="aria-label:hero.facts">
58
+ <li class="fact-tile" data-k="ready_sources"><span class="fact-ico" data-icon="check" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.ready_sources">sources ready</span></li>
59
+ <li class="fact-tile" data-k="answers"><span class="fact-ico" data-icon="target" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.answers">units with an answer</span></li>
60
+ <li class="fact-tile" data-k="raw_rows"><span class="fact-ico" data-icon="rows" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.raw_rows">raw rows</span></li>
61
+ <li class="fact-tile" data-k="sources"><span class="fact-ico" data-icon="database" data-size="22"></span><b class="fact-num"><span class="sk sk-num"></span></b><span class="fact-label" data-i18n="fact.sources">sources indexed</span></li>
62
  </ul>
63
  </div>
64
  <div class="wrap hero-foot">
65
+ <span class="hero-pin"><span data-i18n="hero.indexOf">Index of</span> <a id="pin-link" href="https://huggingface.co/datasets/Battam/agentic-env-index"><span class="sk sk-word"></span></a></span>
66
  <span class="chip chip-dark" id="preview-chip" hidden></span>
67
+ <a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a>
68
+ <a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md" data-i18n="link.readme">Catalog README</a>
69
  <span class="notice" id="notice" hidden></span>
70
  </div>
71
  </header>
 
73
  <section class="band band-canvas" id="material" data-part="overview" aria-labelledby="material-h">
74
  <div class="wrap">
75
  <div class="band-head reveal">
76
+ <h2 id="material-h"><span data-i18n="material.before">Where the</span> <span id="material-n"><span class="sk sk-num"></span></span> <span data-i18n="material.after">task-ready units are</span></h2>
77
  </div>
78
  <!-- two cards: the task-ready units stacked four ways, and the largest task-ready sources; the skeletons have the final cards' sizes, so nothing moves when the index arrives -->
79
  <div class="where-grid">
80
  <figure class="card chart-card where-card" id="ready-card"><div class="sk sk-block"></div></figure>
81
  <figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
82
  </div>
83
+ <div class="band-head sub-head reveal"><h3 id="cats-h" data-i18n="material.cats">Most requested categories</h3></div>
84
  <!-- one card per most requested category (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
85
  <ul class="cat-grid" id="req-cards">
86
  <li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
 
90
 
91
  <div class="band band-white" id="browse" data-part="browse">
92
  <div class="wrap browse-grid">
93
+ <aside class="filters" id="filters" aria-label="Filters" data-i18n-attr="aria-label:filter.title">
94
  <button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row" data-icon="pick" data-size="18">
95
+ <span data-i18n="filter.title">Filters</span><span class="state" id="filter-state" data-i18n="filter.noneOn">none on</span>
96
  </button>
97
  <div class="filter-row" id="filter-row">
98
+ <p class="filter-title" data-icon="pick" data-size="18"><span data-i18n="filter.title">Filters</span></p>
99
+ <label><span data-i18n="filter.domain">Domain</span><select id="f-domain"><option value="" data-i18n="filter.all">All</option></select></label>
100
+ <label><span data-i18n="filter.leaf">Subdomain</span><select id="f-leaf" disabled><option value="" data-i18n="filter.all">All</option></select></label>
101
+ <label><span data-i18n="filter.form">Form</span><select id="f-form"><option value="" data-i18n="filter.all">All</option></select></label>
102
+ <label><span data-i18n="filter.kind">Kind</span><select id="f-status">
103
+ <option value="" data-i18n="filter.any">Any</option>
104
+ <option value="ready" data-i18n="filter.status.ready">Task-ready</option>
105
+ <option value="raw" data-i18n="filter.status.raw">Raw corpus</option>
106
+ <option value="flagged" data-i18n="filter.status.flagged">Flagged</option>
107
  </select></label>
108
+ <label><span data-i18n="filter.training">Training</span><select id="f-train">
109
+ <option value="" data-i18n="filter.any">Any</option>
110
+ <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
111
+ <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
112
  </select></label>
113
+ <label><span data-i18n="filter.shipping">Shipping</span><select id="f-redist">
114
+ <option value="" data-i18n="filter.any">Any</option>
115
+ <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
116
+ <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
117
  </select></label>
118
+ <label><span data-i18n="filter.tier">Licence tier</span><select id="f-tier"><option value="" data-i18n="filter.all">All</option></select></label>
119
+ <label><span data-i18n="filter.size">Source size</span><select id="f-size">
120
+ <option value="" data-i18n="filter.any">Any</option>
121
+ <option value="lt1g" data-i18n="filter.size.lt1g">Under 1 GB</option>
122
+ <option value="1g1t" data-i18n="filter.size.1g1t">1 GB to 1 TB</option>
123
+ <option value="ge1t" data-i18n="filter.size.ge1t">1 TB or more</option>
124
+ <option value="unknown" data-i18n="filter.size.unknown">Not published</option>
125
  </select></label>
126
+ <label class="wide"><span data-i18n="filter.search">Search</span><input id="f-q" type="search" placeholder="Title or text" data-i18n-attr="placeholder:filter.searchHint" autocomplete="off"></label>
127
+ <div class="filter-foot"><span class="count" id="f-count" aria-live="polite"></span><button class="clear" id="f-clear" type="button" disabled data-i18n="filter.clear">Clear</button></div>
128
  </div>
129
  </aside>
130
 
 
132
  <section id="map" aria-labelledby="map-h">
133
  <div class="band-head row reveal">
134
  <h2 id="map-h"><span class="sk sk-line"></span></h2>
135
+ <span class="seg-wrap"><span class="seg-label" id="metric-label" data-i18n="metric.label">Count</span>
136
  <span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
137
+ <button type="button" data-metric="sources" aria-pressed="true" data-i18n="metric.sources">Sources</button>
138
+ <button type="button" data-metric="rows" aria-pressed="false" data-i18n="metric.rows">Pointer rows</button>
139
+ <button type="button" data-metric="bytes" aria-pressed="false" data-i18n="metric.bytes">Size</button>
140
  </span>
141
  </span>
142
  </div>
 
164
  <div class="card agents-card lift reveal" id="agents">
165
  <span class="agents-ico" data-icon="robot" data-size="36"></span>
166
  <div class="agents-text">
167
+ <h3 id="agents-h" data-i18n="nav.agents">For agents</h3>
168
+ <p class="small" data-i18n="agents.text">Catalog columns · the one command · the default profile · where answers land</p>
169
  </div>
170
  <div class="agents-actions">
171
  <a class="btn solid" href="llms.txt" data-icon="doc" data-size="16">llms.txt</a>
172
+ <a class="btn" href="https://huggingface.co/datasets/Battam/agentic-env-index#for-agents" data-icon="doc" data-size="16"><span data-i18n="agents.readme">README: For agents</span></a>
173
  </div>
174
  </div>
175
  </div>
 
191
  <div class="foot-cta">
192
  <span class="foot-ico" data-icon="request" data-size="32"></span>
193
  <div>
194
+ <h2 data-i18n="foot.missing">Missing material?</h2>
195
+ <p data-i18n="foot.ask">Ask for a category, a form or a source.</p>
196
  </div>
197
+ <a class="btn on-dark-solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml" data-i18n="nav.request">Request data</a>
198
  </div>
199
  <ul class="foot-links">
200
+ <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a></li>
201
+ <li><a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md" data-i18n="link.readme">Catalog README</a></li>
202
+ <li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li>
203
+ <li><a href="#top" data-i18n="foot.top">Back to top</a></li>
204
  </ul>
205
  <p class="foot-pin mono" id="foot-pin"></p>
206
  </div>
llms.txt CHANGED
@@ -1,11 +1,11 @@
1
  # Agentic Env Index: for agents
2
 
3
  - Pick sources from `meta/catalog.parquet` (one row per source, about 50 KB; do not download `meta/` whole). Columns: `source`; `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a raw corpus has one unit per row), `answers` (units that have an answer, the expected results kept apart) and `files` (files of every role); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (`not scanned` when no comparison is recorded for the source); `license.ship`; `pii.level`; `distinct_bytes` (distinct content) and `bytes` (referenced, the sum over rows). The vocabulary is `meta/vocabulary.json`: every domain code with its name, what it covers and its parent, and every form with what it means.
4
- - Size of this release: Distinct content: 3.79 PB. 281,312,740 rows have no size and are not in that number. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it.
5
  - Task-ready sources are collections whose rows carry the roles unit, companion and answer (`answers` above 0, or `units` below `files`): each unit comes with its expected results. Every other source is a raw corpus to mine, one unit per row.
6
  - Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--domain`, `--form`, `--tag`, `--ext`; `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
7
  - It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
8
- - The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, collections awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; `--all` turns the profile off, and each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet). Compared with the benchmark list (keys of 2026-10-24): 1,868 of 1,868 sources, 1,779 of 1,779 task-ready sources. The profile also removes every row whose data file was not compared (`--allow-benchmark` keeps them, unchecked).
9
  - Output: `./data/<domain>/<source>/...` holds the units and their companion files (README, LICENSE); `./data/answer/<source>/...` holds the answers and expected results, apart from the material; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, answer, origin_kind, leaf, role, unit_group, answer_for); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
10
  - Never put `answer/` into the environment the agent works in: the answers are there to check its result, not to be read by it. Each kind of answer, and what else to keep out:
11
  - Answer files (role `answer`; an answer in JSON names its kind in the field `kind` or `answer_kind`): the expected result of one unit, such as an output, a value or a verdict. Keep out of the environment also any copy of it, and any file that holds the same result.
 
1
  # Agentic Env Index: for agents
2
 
3
  - Pick sources from `meta/catalog.parquet` (one row per source, about 50 KB; do not download `meta/` whole). Columns: `source`; `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a raw corpus has one unit per row), `answers` (units that have an answer, the expected results kept apart) and `files` (files of every role); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (`not scanned` when no comparison is recorded for the source); `license.ship`; `pii.level`; `distinct_bytes` (distinct content) and `bytes` (referenced, the sum over rows). The vocabulary is `meta/vocabulary.json`: every domain code with its name, what it covers and its parent, and every form with what it means.
4
+ - Size of this release: Distinct content: 3.79 PB. 281,582,670 rows have no size and are not in that number. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it.
5
  - Task-ready sources are collections whose rows carry the roles unit, companion and answer (`answers` above 0, or `units` below `files`): each unit comes with its expected results. Every other source is a raw corpus to mine, one unit per row.
6
  - Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--domain`, `--form`, `--tag`, `--ext`; `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
7
  - It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
8
+ - The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, collections awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; `--all` turns the profile off, and each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet). Compared with the benchmark list (keys of 2026-10-24): 1,872 of 1,872 sources, 1,783 of 1,783 task-ready sources. The profile also removes every row whose data file was not compared (`--allow-benchmark` keeps them, unchecked).
9
  - Output: `./data/<domain>/<source>/...` holds the units and their companion files (README, LICENSE); `./data/answer/<source>/...` holds the answers and expected results, apart from the material; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, answer, origin_kind, leaf, role, unit_group, answer_for); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
10
  - Never put `answer/` into the environment the agent works in: the answers are there to check its result, not to be read by it. Each kind of answer, and what else to keep out:
11
  - Answer files (role `answer`; an answer in JSON names its kind in the field `kind` or `answer_kind`): the expected result of one unit, such as an output, a value or a verdict. Keep out of the environment also any copy of it, and any file that holds the same result.
locales/en.json ADDED
@@ -0,0 +1,379 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "lang.name": "English",
3
+ "lang.switch": "Show this page in Chinese",
4
+
5
+ "page.title": "Agentic Env Catalog",
6
+ "page.description": "Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it.",
7
+
8
+ "nav.sections": "Page sections",
9
+ "nav.overview": "Overview",
10
+ "nav.browse": "Browse",
11
+ "nav.get": "Get the data",
12
+ "nav.excluded": "Not included",
13
+ "nav.agents": "For agents",
14
+ "nav.request": "Request data",
15
+
16
+ "hero.eyebrow": "The Agentic Env Index, by category",
17
+ "hero.title": "Real material for agent tasks",
18
+ "hero.units": "task-ready units",
19
+ "hero.browse": "Browse categories",
20
+ "hero.facts": "Facts",
21
+ "hero.indexOf": "Index of",
22
+ "hero.sizeUnknown": "Distinct content not known yet",
23
+ "hero.distinct": "distinct content",
24
+ "hero.unsized_one": "{{n}} row has no size and is not in that number",
25
+ "hero.unsized_other": "{{n}} rows have no size and are not in that number",
26
+ "hero.commit": "Commit {{sha}}",
27
+ "hero.preview": "Preview data",
28
+ "fact.ready_sources": "sources ready",
29
+ "fact.answers": "units with an answer",
30
+ "fact.raw_rows": "raw rows",
31
+ "fact.sources": "sources indexed",
32
+ "link.dataset": "Dataset",
33
+ "link.readme": "Catalog README",
34
+
35
+ "foot.missing": "Missing material?",
36
+ "foot.ask": "Ask for a category, a form or a source.",
37
+ "foot.top": "Back to top",
38
+ "foot.index": "Index {{sha}} · {{date}}",
39
+
40
+ "notice.vocabulary": "The vocabulary could not be read ({{error}}); codes are shown as they are.",
41
+ "notice.failed": "Could not read the index: {{error}}",
42
+ "error.noCommit": "the Hub gave no commit id",
43
+
44
+ "num.unknown": "unknown",
45
+ "common.unknown": "Unknown",
46
+ "common.noneYet": "none yet",
47
+ "common.all": "All {{n}}",
48
+ "common.first": "First {{n}}",
49
+ "common.fewer": "Fewer",
50
+ "copy.button": "Copy",
51
+ "copy.done": "Copied",
52
+ "copy.select": "Select the text",
53
+ "chart.view": "View",
54
+ "chart.chart": "Chart",
55
+ "chart.table": "Table",
56
+ "heat.aria": "{{metric}} by domain and form",
57
+ "heat.scale": "{{metric}}, log scale",
58
+ "heat.cell": "{{row}}, {{col}}: {{value}}",
59
+
60
+ "n.source_one": "{{n}} source",
61
+ "n.source_other": "{{n}} sources",
62
+ "n.domain_one": "{{n}} domain",
63
+ "n.domain_other": "{{n}} domains",
64
+ "n.unit_one": "{{n}} unit",
65
+ "n.unit_other": "{{n}} units",
66
+ "n.unitFile_one": "{{n}} unit file",
67
+ "n.unitFile_other": "{{n}} unit files",
68
+ "n.companion_one": "{{n}} companion",
69
+ "n.companion_other": "{{n}} companions",
70
+ "n.answer_one": "{{n}} answer",
71
+ "n.answer_other": "{{n}} answers",
72
+ "n.file_one": "{{n}} file",
73
+ "n.file_other": "{{n}} files",
74
+ "n.row_one": "{{n}} row",
75
+ "n.row_other": "{{n}} rows",
76
+ "n.rawRows_one": "{{n}} raw row",
77
+ "n.rawRows_other": "{{n}} raw rows",
78
+ "n.rowsCompact_one": "{{n}} row",
79
+ "n.rowsCompact_other": "{{n}} rows",
80
+
81
+ "name.notClassified": "Not classified",
82
+ "name.severalDomains": "Several domains",
83
+ "name.otherForms": "Other forms",
84
+
85
+ "domain.D1": "Software engineering",
86
+ "domain.D2": "System administration",
87
+ "domain.D3": "Data science",
88
+ "domain.D4": "Machine learning",
89
+ "domain.D5": "Science",
90
+ "domain.D6": "Operations",
91
+ "domain.D7": "Hardware",
92
+ "domain.D8": "Security",
93
+ "domain.D9": "Media",
94
+ "domain.D10": "Misc",
95
+ "domain.D10.2": "Constraint scheduling",
96
+ "domain.D4.1": "Training and debugging",
97
+ "domain.D4.5": "Paper reproductions",
98
+ "domain.D3.1": "ETL and multi-source merging",
99
+ "domain.D3.6": "Visualisation and reports",
100
+ "domain.D2.5": "File systems and recovery",
101
+ "domain.D2.6": "Crash and performance analysis",
102
+ "domain.D2.7": "Virtualisation and emulators",
103
+ "domain.D2.8": "Large text and log editing",
104
+ "domain.D8.2": "Reverse engineering and binaries",
105
+
106
+ "tier.permissive": "Permissive",
107
+ "tier.share-alike": "Share-alike",
108
+ "tier.nc-research": "Non-commercial",
109
+ "tier.no-training": "No training",
110
+ "tier.internal-only": "Internal only",
111
+ "tier.mixed": "Mixed",
112
+ "tier.unknown": "Not stated",
113
+ "permit.yes": "Allowed",
114
+ "permit.no": "Not allowed",
115
+ "permit.conditional": "Conditional",
116
+ "permit.unknown": "Not stated",
117
+ "reason.not_english": "Not English",
118
+ "reason.licence_restricted": "Licence too tight",
119
+ "reason.access_gated": "Behind a login",
120
+ "reason.personal_data": "Personal data",
121
+ "reason.answer_not_verifiable": "Answer uncheckable",
122
+ "reason.answer_cells_insufficient": "Few answer cells",
123
+ "reason.not_in_scope": "Out of scope",
124
+ "reason.below_threshold": "Too little activity",
125
+ "reason.series_cap": "Series cap",
126
+ "reason.duplicate": "Duplicate",
127
+ "reason.other": "Other",
128
+ "unitKind.file": "a file",
129
+ "unitKind.apiResponse": "an API response",
130
+ "unitKind.warcRecord": "a record in a web archive",
131
+ "unitKind.archiveMember": "a file inside an archive",
132
+ "unitKind.byteRange": "a byte range of a bigger file",
133
+ "unitCount.file_one": "{{n}} file",
134
+ "unitCount.file_other": "{{n}} files",
135
+ "unitCount.apiResponse_one": "{{n}} API response",
136
+ "unitCount.apiResponse_other": "{{n}} API responses",
137
+ "unitCount.warcRecord_one": "{{n}} WARC record",
138
+ "unitCount.warcRecord_other": "{{n}} WARC records",
139
+ "unitCount.archiveMember_one": "{{n}} archive member",
140
+ "unitCount.archiveMember_other": "{{n}} archive members",
141
+ "unitCount.byteRange_one": "{{n}} byte range",
142
+ "unitCount.byteRange_other": "{{n}} byte ranges",
143
+ "unitCount.code_one": "{{n}} {{kind}}",
144
+ "unitCount.code_other": "{{n}} {{kind}}s",
145
+ "tool.editor": "any editor",
146
+ "tool.browser": "a browser",
147
+ "tool.suiteRunner": "the suite runner",
148
+ "batch.collecting": "Collecting",
149
+
150
+ "fit.ship": "Shippable licence",
151
+ "fit.bench": "No benchmark overlap",
152
+ "fit.pii": "No personal data",
153
+ "fit.size": "Files up to 20 MB",
154
+ "fit.alone": "Unit stands alone",
155
+ "fit.answers": "Answers or expected results",
156
+ "fit.login": "No login",
157
+ "fit.dot": "{{label}}: {{state}}",
158
+ "fitWord.yes": "Fits",
159
+ "fitWord.part": "Partly",
160
+ "fitWord.no": "Does not fit",
161
+ "fitWord.unknown": "Not known",
162
+
163
+ "material.before": "Where the",
164
+ "material.after": "task-ready units are",
165
+ "material.cats": "Most requested categories",
166
+ "kind.ready": "Task-ready",
167
+ "cat.lead.ready_one": "task-ready unit",
168
+ "cat.lead.ready_other": "task-ready units",
169
+ "cat.lead.raw_one": "raw row",
170
+ "cat.lead.raw_other": "raw rows",
171
+ "cat.lead.sources_one": "source",
172
+ "cat.lead.sources_other": "sources",
173
+ "cat.scaled": "{{scale}} {{word}}",
174
+ "cat.wanted": "Wanted",
175
+ "cat.request": "Request",
176
+ "cat.plusRaw_one": "+ {{n}} raw row",
177
+ "cat.plusRaw_other": "+ {{n}} raw rows",
178
+ "cat.readyTip_one": "{{ready}} of {{n}} source task-ready",
179
+ "cat.readyTip_other": "{{ready}} of {{n}} sources task-ready",
180
+ "cat.shippable": "Shippable",
181
+ "cat.shippableRaw": "Shippable, raw",
182
+ "cat.shipTipReady": "of the task-ready units may be shipped",
183
+ "cat.shipTipRaw": "of the raw rows may be shipped",
184
+ "cat.sample": "Take a sample",
185
+ "request.more": "More material for: {{name}}",
186
+ "request.like": "Like this catalog selection ({{chips}}), but: ",
187
+
188
+ "stack.otherForms_one": "Other {{n}} form",
189
+ "stack.otherForms_other": "Other {{n}} forms",
190
+ "stack.otherDomains_one": "Other {{n}} domain",
191
+ "stack.otherDomains_other": "Other {{n}} domains",
192
+ "stack.notClassified": "Not classified yet",
193
+ "where.title": "Largest task-ready sources",
194
+ "where.otherSources_one": "Other {{n}} source",
195
+ "where.otherSources_other": "Other {{n}} sources",
196
+ "ready.title": "Task-ready or raw",
197
+ "ready.rawCorpora": "Raw corpora",
198
+ "ready.answersApart": "Answers apart",
199
+ "ready.checksInside": "Checks inside",
200
+ "ready.showing": "Showing task-ready",
201
+ "ready.only": "Only task-ready",
202
+ "ready.byForm": "By form",
203
+ "ready.byDomain": "By domain",
204
+ "ready.split": "Split",
205
+ "ready.part": "Part",
206
+ "ready.kind": "Kind",
207
+ "ready.units": "Units",
208
+ "ready.rawRow_one": "Raw corpora ({{n}} row)",
209
+ "ready.rawRow_other": "Raw corpora ({{n}} rows)",
210
+ "ready.answersKept": "Answers kept apart",
211
+ "ready.resultsInside": "Expected results inside",
212
+ "th.source": "Source",
213
+ "th.form": "Form",
214
+ "th.domain": "Domain",
215
+ "th.readyUnits": "Task-ready units",
216
+ "th.batch": "Batch",
217
+ "th.rows": "Rows",
218
+ "th.changed": "Changed",
219
+ "th.state": "State",
220
+ "th.address": "Address",
221
+ "th.size": "Size",
222
+
223
+ "filter.title": "Filters",
224
+ "filter.noneOn": "none on",
225
+ "filter.on_one": "{{n}} on",
226
+ "filter.on_other": "{{n}} on",
227
+ "filter.all": "All",
228
+ "filter.any": "Any",
229
+ "filter.domain": "Domain",
230
+ "filter.leaf": "Subdomain",
231
+ "filter.form": "Form",
232
+ "filter.kind": "Kind",
233
+ "filter.status.ready": "Task-ready",
234
+ "filter.status.raw": "Raw corpus",
235
+ "filter.status.flagged": "Flagged",
236
+ "filter.training": "Training",
237
+ "filter.shipping": "Shipping",
238
+ "filter.permit.yes": "Allowed, all rows",
239
+ "filter.permit.cond": "Allowed or conditional",
240
+ "filter.tier": "Licence tier",
241
+ "filter.size": "Source size",
242
+ "filter.size.lt1g": "Under 1 GB",
243
+ "filter.size.1g1t": "1 GB to 1 TB",
244
+ "filter.size.ge1t": "1 TB or more",
245
+ "filter.size.unknown": "Not published",
246
+ "filter.search": "Search",
247
+ "filter.searchHint": "Title or text",
248
+ "filter.clear": "Clear",
249
+ "chip.domain": "Domain",
250
+ "chip.form": "Form",
251
+ "chip.training": "Training",
252
+ "chip.redistribution": "Redistribution",
253
+ "chip.tier": "Tier",
254
+ "chip.size": "Size",
255
+ "chip.kind": "Kind",
256
+ "chip.search": "Search",
257
+
258
+ "metric.label": "Count",
259
+ "metric.sources": "Sources",
260
+ "metric.rows": "Pointer rows",
261
+ "metric.bytes": "Size",
262
+ "map.rows_one": "{{n}} pointer row",
263
+ "map.rows_other": "{{n}} pointer rows",
264
+ "map.nothing": "Nothing matches",
265
+ "map.head": "{{amount}}, {{domains}}",
266
+ "map.title": "{{metric}}: domain × form",
267
+ "map.columns": "Columns",
268
+
269
+ "col.title": "Source",
270
+ "col.domain": "Category",
271
+ "col.form": "Form",
272
+ "col.units": "Units or rows",
273
+ "col.distinct": "Distinct size",
274
+ "col.tier": "Licence",
275
+ "col.fit": "Fit",
276
+ "sources.head": "{{sources}}, {{ready}} task-ready",
277
+ "sources.none": "No source matches",
278
+ "sources.ready": "Ready",
279
+ "sources.unitsLabel": "Units",
280
+ "sources.rowsLabel": "Rows",
281
+ "sources.unitWord_one": " unit",
282
+ "sources.unitWord_other": " units",
283
+ "sources.rawWord_one": " raw row",
284
+ "sources.rawWord_other": " raw rows",
285
+ "sources.fitKey": "Fit:",
286
+ "size.method": "Distinct content is the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column distinct_bytes; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. tools/fetch.py removes duplicates over your own selection when it reads it.",
287
+ "size.unknown": "Distinct content (the sum of size_bytes over the rows minus the sizes of the rows that repeat another row's content) is not known for this release yet.",
288
+ "size.note": "Distinct content: {{size}}. {{method}}",
289
+ "size.noteUnsized": "Distinct content: {{size}}. {{unsized}} {{method}}",
290
+ "size.unsized_one": "{{n}} row has no size and is not in that number.",
291
+ "size.unsized_other": "{{n}} rows have no size and are not in that number.",
292
+
293
+ "detail.what": "What it is",
294
+ "detail.notDescribed": "Not described",
295
+ "detail.oneUnit": "One unit",
296
+ "detail.taskUnit": "One task unit",
297
+ "detail.unitWithChecks": "One unit with its checks",
298
+ "detail.from": "Comes from",
299
+ "detail.notRecorded": "Not recorded",
300
+ "detail.alone": "Stands alone",
301
+ "detail.withCompanions": "With companions",
302
+ "detail.yes": "Yes",
303
+ "detail.eachFile": "Yes, each file",
304
+ "detail.container": "Needs its container file",
305
+ "detail.answers": "Answers",
306
+ "detail.answersInside": "Inside each unit (tests, expected output)",
307
+ "detail.answersNone": "None: raw material",
308
+ "detail.opensWith": "Opens with",
309
+ "detail.noExt": "no ext",
310
+ "detail.anyEditor": "Any editor",
311
+ "detail.licence": "Licence",
312
+ "detail.training": "Training",
313
+ "detail.shipping": "Shipping",
314
+ "detail.size": "Size",
315
+ "detail.distinct": "{{size}} distinct",
316
+ "detail.unsized": "{{n}} unsized",
317
+ "detail.median": "median {{size}}",
318
+ "detail.stripAria": "median {{median}}, 99% under {{p99}}",
319
+ "detail.example": "Example",
320
+ "detail.noFile": "No data file",
321
+ "detail.take": "Take it",
322
+ "detail.more": "Batches and first rows",
323
+ "detail.notCounted": "not counted",
324
+
325
+ "get.nothing": "Nothing to take",
326
+ "get.takeOne_one": "Take {{count}} unit from {{title}}",
327
+ "get.takeOne_other": "Take {{count}} units from {{title}}",
328
+ "get.takeEach_one": "Take {{count}} unit from each of {{sources}}",
329
+ "get.takeEach_other": "Take {{count}} units from each of {{sources}}",
330
+ "get.command": "Command",
331
+ "get.sourceWord_one": "source",
332
+ "get.sourceWord_other": "sources",
333
+ "get.wholeIndex": "Whole index",
334
+ "get.change": "Change",
335
+ "get.look": "Look first",
336
+ "get.take": "Take",
337
+ "get.sql": "DuckDB SQL, pinned to the commit",
338
+ "sql.head": "Agentic Env Index @ {{sha}}: {{sources}}, {{rows}}",
339
+ "sql.all": "all sources",
340
+ "sql.chip": "{{k}}: {{v}}",
341
+ "csv.progress_one": "{{done}}/{{n}} file",
342
+ "csv.progress_other": "{{done}}/{{n}} files",
343
+ "csv.button": "CSV · {{rows}}",
344
+ "csv.tooMany": "Over {{n}} rows: use the command or SQL",
345
+ "tree.title": "What it writes",
346
+ "tree.units": "units, README, LICENSE",
347
+ "tree.answers": "answers, kept apart",
348
+ "tree.manifest": "one line per file, sha256 checked",
349
+ "tree.sizes": "one line per source",
350
+ "tree.domainNote": ": such as D3_data or D3",
351
+ "tree.leftOut": "Left out by default",
352
+ "profile.duplicates": "Duplicates",
353
+ "profile.benchmark": "Benchmark copies",
354
+ "profile.licence": "No shipping licence",
355
+ "profile.english": "Not English",
356
+ "profile.personal": "Personal data",
357
+ "profile.large": "Over 20 MB",
358
+ "agents.text": "Catalog columns · the one command · the default profile · where answers land",
359
+ "agents.readme": "README: For agents",
360
+
361
+ "excluded.head_one": "{{n}} reviewed and left out",
362
+ "excluded.head_other": "{{n}} reviewed and left out",
363
+ "excluded.none": "Nothing left out",
364
+ "excluded.reason": "Reason",
365
+ "excluded.requirement": "Requirement",
366
+ "excluded.items": "Items",
367
+ "excluded.unit": "Unit",
368
+ "excluded.notMet": "Requirement not met",
369
+ "excluded.detail": "Detail",
370
+ "excluded.reviewed": "Reviewed",
371
+ "excluded.wholeSource": "whole source",
372
+ "history.title": "History: sources after each snapshot",
373
+ "history.start": "start",
374
+ "history.date": "Date",
375
+ "history.entry": "Entry",
376
+ "history.sourcesAdded": "Sources added",
377
+ "history.rowsAdded": "Rows added",
378
+ "history.started": "Log started"
379
+ }
locales/zh.json ADDED
@@ -0,0 +1,343 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "lang.name": "中文",
3
+ "lang.switch": "以英文显示本页",
4
+
5
+ "page.title": "Agentic Env 数据目录",
6
+ "page.description": "来自公开指针索引 Battam/agentic-env-index 的终端智能体任务真实素材:按类别浏览,一眼看出是否适配,一条命令即可获取。",
7
+
8
+ "nav.sections": "页面分区",
9
+ "nav.overview": "概览",
10
+ "nav.browse": "浏览",
11
+ "nav.get": "获取数据",
12
+ "nav.excluded": "未收录",
13
+ "nav.agents": "给智能体",
14
+ "nav.request": "申请数据",
15
+
16
+ "hero.eyebrow": "Agentic Env 索引,按类别",
17
+ "hero.title": "面向智能体任务的真实素材",
18
+ "hero.units": "个任务就绪单元",
19
+ "hero.browse": "浏览类别",
20
+ "hero.facts": "概况",
21
+ "hero.indexOf": "索引版本",
22
+ "hero.sizeUnknown": "去重后内容大小尚未统计",
23
+ "hero.distinct": "去重后内容",
24
+ "hero.unsized_other": "{{n}} 行没有大小,未计入该数字",
25
+ "hero.commit": "提交 {{sha}}",
26
+ "hero.preview": "预览数据",
27
+ "fact.ready_sources": "个就绪数据源",
28
+ "fact.answers": "个带答案的单元",
29
+ "fact.raw_rows": "行原始数据",
30
+ "fact.sources": "个已索引数据源",
31
+ "link.dataset": "数据集",
32
+ "link.readme": "目录 README",
33
+
34
+ "foot.missing": "缺少素材?",
35
+ "foot.ask": "可以申请一个类别、一种形态或一个数据源。",
36
+ "foot.top": "回到顶部",
37
+ "foot.index": "索引 {{sha}} · {{date}}",
38
+
39
+ "notice.vocabulary": "无法读取词表({{error}});代码按原样显示。",
40
+ "notice.failed": "无法读取索引:{{error}}",
41
+ "error.noCommit": "Hub 未返回提交 ID",
42
+
43
+ "num.unknown": "未知",
44
+ "common.unknown": "未知",
45
+ "common.noneYet": "暂无",
46
+ "common.all": "全部 {{n}} 项",
47
+ "common.first": "前 {{n}} 项",
48
+ "common.fewer": "收起",
49
+ "copy.button": "复制",
50
+ "copy.done": "已复制",
51
+ "copy.select": "请手动选中文本",
52
+ "chart.view": "视图",
53
+ "chart.chart": "图表",
54
+ "chart.table": "表格",
55
+ "heat.aria": "按领域和形态统计的{{metric}}",
56
+ "heat.scale": "{{metric}},对数刻度",
57
+ "heat.cell": "{{row}},{{col}}:{{value}}",
58
+
59
+ "n.source_other": "{{n}} 个数据源",
60
+ "n.domain_other": "{{n}} 个领域",
61
+ "n.unit_other": "{{n}} 个单元",
62
+ "n.unitFile_other": "{{n}} 个单元文件",
63
+ "n.companion_other": "{{n}} 个配套文件",
64
+ "n.answer_other": "{{n}} 个答案",
65
+ "n.file_other": "{{n}} 个文件",
66
+ "n.row_other": "{{n}} 行",
67
+ "n.rawRows_other": "{{n}} 行原始数据",
68
+ "n.rowsCompact_other": "{{n}} 行",
69
+
70
+ "name.notClassified": "未分类",
71
+ "name.severalDomains": "多个领域",
72
+ "name.otherForms": "其他形态",
73
+
74
+ "domain.D1": "软件工程",
75
+ "domain.D2": "系统管理",
76
+ "domain.D3": "数据科学",
77
+ "domain.D4": "机器学习",
78
+ "domain.D5": "科学",
79
+ "domain.D6": "运维",
80
+ "domain.D7": "硬件",
81
+ "domain.D8": "安全",
82
+ "domain.D9": "媒体",
83
+ "domain.D10": "其他",
84
+ "domain.D10.2": "约束调度",
85
+ "domain.D4.1": "训练与调试",
86
+ "domain.D4.5": "论文复现",
87
+ "domain.D3.1": "ETL 与多源合并",
88
+ "domain.D3.6": "可视化与报告",
89
+ "domain.D2.5": "文件系统与数据恢复",
90
+ "domain.D2.6": "崩溃与性能分析",
91
+ "domain.D2.7": "虚拟化与模拟器",
92
+ "domain.D2.8": "大文本与日志编辑",
93
+ "domain.D8.2": "逆向工程与二进制",
94
+
95
+ "tier.permissive": "宽松",
96
+ "tier.share-alike": "相同方式共享",
97
+ "tier.nc-research": "非商业",
98
+ "tier.no-training": "禁止训练",
99
+ "tier.internal-only": "仅限内部",
100
+ "tier.mixed": "混合",
101
+ "tier.unknown": "未注明",
102
+ "permit.yes": "允许",
103
+ "permit.no": "不允许",
104
+ "permit.conditional": "有条件",
105
+ "permit.unknown": "未注明",
106
+ "reason.not_english": "非英文",
107
+ "reason.licence_restricted": "许可证限制过严",
108
+ "reason.access_gated": "需要登录",
109
+ "reason.personal_data": "含个人数据",
110
+ "reason.answer_not_verifiable": "答案无法核验",
111
+ "reason.answer_cells_insufficient": "答案单元格过少",
112
+ "reason.not_in_scope": "不在范围内",
113
+ "reason.below_threshold": "活跃度过低",
114
+ "reason.series_cap": "系列数量上限",
115
+ "reason.duplicate": "重复",
116
+ "reason.other": "其他",
117
+ "unitKind.file": "一个文件",
118
+ "unitKind.apiResponse": "一条 API 响应",
119
+ "unitKind.warcRecord": "Web 归档中的一条记录",
120
+ "unitKind.archiveMember": "压缩包内的一个文件",
121
+ "unitKind.byteRange": "大文件中的一段字节范围",
122
+ "unitCount.file_other": "{{n}} 个文件",
123
+ "unitCount.apiResponse_other": "{{n}} 条 API 响应",
124
+ "unitCount.warcRecord_other": "{{n}} 条 WARC 记录",
125
+ "unitCount.archiveMember_other": "{{n}} 个压缩包成员",
126
+ "unitCount.byteRange_other": "{{n}} 段字节范围",
127
+ "unitCount.code_other": "{{n}} 个 {{kind}}",
128
+ "tool.editor": "任意编辑器",
129
+ "tool.browser": "浏览器",
130
+ "tool.suiteRunner": "测试套件运行器",
131
+ "batch.collecting": "收集中",
132
+
133
+ "fit.ship": "许可证允许分发",
134
+ "fit.bench": "与基准无重合",
135
+ "fit.pii": "无个人数据",
136
+ "fit.size": "文件不超过 20 MB",
137
+ "fit.alone": "单元可独立使用",
138
+ "fit.answers": "有答案或预期结果",
139
+ "fit.login": "无需登录",
140
+ "fit.dot": "{{label}}:{{state}}",
141
+ "fitWord.yes": "适配",
142
+ "fitWord.part": "部分适配",
143
+ "fitWord.no": "不适配",
144
+ "fitWord.unknown": "未知",
145
+
146
+ "material.before": "这",
147
+ "material.after": "个任务就绪单元的分布",
148
+ "material.cats": "需求最多的类别",
149
+ "kind.ready": "任务就绪",
150
+ "cat.lead.ready_other": "个任务就绪单元",
151
+ "cat.lead.raw_other": "行原始数据",
152
+ "cat.lead.sources_other": "个数据源",
153
+ "cat.scaled": "{{scale}}{{word}}",
154
+ "cat.wanted": "征集中",
155
+ "cat.request": "申请",
156
+ "cat.plusRaw_other": "另有 {{n}} 行原始数据",
157
+ "cat.readyTip_other": "{{n}} 个数据源中有 {{ready}} 个任务就绪",
158
+ "cat.shippable": "可分发",
159
+ "cat.shippableRaw": "可分发(原始)",
160
+ "cat.shipTipReady": "的任务就绪单元可以分发",
161
+ "cat.shipTipRaw": "的原始数据行可以分发",
162
+ "cat.sample": "取样",
163
+ "request.more": "更多素材:{{name}}",
164
+ "request.like": "类似当前目录选择({{chips}}),但:",
165
+
166
+ "stack.otherForms_other": "其他 {{n}} 种形态",
167
+ "stack.otherDomains_other": "其他 {{n}} 个领域",
168
+ "stack.notClassified": "尚未分类",
169
+ "where.title": "最大的任务就绪数据源",
170
+ "where.otherSources_other": "其他 {{n}} 个数据源",
171
+ "ready.title": "任务就绪还是原始",
172
+ "ready.rawCorpora": "原始语料",
173
+ "ready.answersApart": "答案单独存放",
174
+ "ready.checksInside": "检查在单元内",
175
+ "ready.showing": "正在只看任务就绪",
176
+ "ready.only": "只看任务就绪",
177
+ "ready.byForm": "按形态",
178
+ "ready.byDomain": "按领域",
179
+ "ready.split": "划分",
180
+ "ready.part": "部分",
181
+ "ready.kind": "类型",
182
+ "ready.units": "单元",
183
+ "ready.rawRow_other": "原始语料({{n}} 行)",
184
+ "ready.answersKept": "答案单独存放",
185
+ "ready.resultsInside": "预期结果在单元内",
186
+ "th.source": "数据源",
187
+ "th.form": "形态",
188
+ "th.domain": "领域",
189
+ "th.readyUnits": "任务就绪单元",
190
+ "th.batch": "批次",
191
+ "th.rows": "行数",
192
+ "th.changed": "更新于",
193
+ "th.state": "状态",
194
+ "th.address": "地址",
195
+ "th.size": "大小",
196
+
197
+ "filter.title": "筛选",
198
+ "filter.noneOn": "未启用",
199
+ "filter.on_other": "已启用 {{n}} 项",
200
+ "filter.all": "全部",
201
+ "filter.any": "不限",
202
+ "filter.domain": "领域",
203
+ "filter.leaf": "子领域",
204
+ "filter.form": "形态",
205
+ "filter.kind": "类型",
206
+ "filter.status.ready": "任务就绪",
207
+ "filter.status.raw": "原始语料",
208
+ "filter.status.flagged": "已标记",
209
+ "filter.training": "训练",
210
+ "filter.shipping": "分发",
211
+ "filter.permit.yes": "允许,所有行",
212
+ "filter.permit.cond": "允许或有条件",
213
+ "filter.tier": "许可证等级",
214
+ "filter.size": "数据源大小",
215
+ "filter.size.lt1g": "小于 1 GB",
216
+ "filter.size.1g1t": "1 GB 到 1 TB",
217
+ "filter.size.ge1t": "1 TB 及以上",
218
+ "filter.size.unknown": "未公布",
219
+ "filter.search": "搜索",
220
+ "filter.searchHint": "标题或正文",
221
+ "filter.clear": "清除",
222
+ "chip.domain": "领域",
223
+ "chip.form": "形态",
224
+ "chip.training": "训练",
225
+ "chip.redistribution": "再分发",
226
+ "chip.tier": "等级",
227
+ "chip.size": "大小",
228
+ "chip.kind": "类型",
229
+ "chip.search": "搜索",
230
+
231
+ "metric.label": "统计",
232
+ "metric.sources": "数据源",
233
+ "metric.rows": "指针行",
234
+ "metric.bytes": "大小",
235
+ "map.rows_other": "{{n}} 条指针行",
236
+ "map.nothing": "没有匹配项",
237
+ "map.head": "{{amount}},{{domains}}",
238
+ "map.title": "{{metric}}:领域 × 形态",
239
+ "map.columns": "列",
240
+
241
+ "col.title": "数据源",
242
+ "col.domain": "类别",
243
+ "col.form": "形态",
244
+ "col.units": "单元或行数",
245
+ "col.distinct": "去重后大小",
246
+ "col.tier": "许可证",
247
+ "col.fit": "适配",
248
+ "sources.head": "{{sources}},其中 {{ready}} 个任务就绪",
249
+ "sources.none": "没有匹配的数据源",
250
+ "sources.ready": "就绪",
251
+ "sources.unitsLabel": "单元",
252
+ "sources.rowsLabel": "行数",
253
+ "sources.unitWord_other": " 个单元",
254
+ "sources.rawWord_other": " 行原始数据",
255
+ "sources.fitKey": "适配:",
256
+ "size.method": "去重后内容是各行 size_bytes 之和,减去与其他行内容重复的行的大小(在同一哈希范围内依次比较 sha256、sha1、md5 加大小;每份内容只保留一份,归入注册顺序中的第一个数据源),每次发布时统计,按数据源记在目录列 distinct_bytes 中;若同一数据源还整体列出了某个压缩包,其成员的哈希与压缩包不同,因此不会被减去。tools/fetch.py 读取时会在你自己的选择范围内去重。",
257
+ "size.unknown": "本次发布的去重后内容(各行 size_bytes 之和减去与其他行内容重复的行的大小)尚未统计。",
258
+ "size.note": "去重后内容:{{size}}。{{method}}",
259
+ "size.noteUnsized": "去重后内容:{{size}}。{{unsized}}{{method}}",
260
+ "size.unsized_other": "{{n}} 行没有大小,未计入该数字。",
261
+
262
+ "detail.what": "简介",
263
+ "detail.notDescribed": "暂无描述",
264
+ "detail.oneUnit": "一个单元",
265
+ "detail.taskUnit": "一个任务单元",
266
+ "detail.unitWithChecks": "一个带检查的单元",
267
+ "detail.from": "来源",
268
+ "detail.notRecorded": "未记录",
269
+ "detail.alone": "可独立使用",
270
+ "detail.withCompanions": "需配套文件",
271
+ "detail.yes": "是",
272
+ "detail.eachFile": "是,每个文件都可以",
273
+ "detail.container": "需要其所在的容器文件",
274
+ "detail.answers": "答案",
275
+ "detail.answersInside": "在每个单元内(测试、预期输出)",
276
+ "detail.answersNone": "无:原始素材",
277
+ "detail.opensWith": "打开方式",
278
+ "detail.noExt": "无扩展名",
279
+ "detail.anyEditor": "任意编辑器",
280
+ "detail.licence": "许可证",
281
+ "detail.training": "训练",
282
+ "detail.shipping": "分发",
283
+ "detail.size": "大小",
284
+ "detail.distinct": "去重后 {{size}}",
285
+ "detail.unsized": "{{n}} 行无大小",
286
+ "detail.median": "中位数 {{size}}",
287
+ "detail.stripAria": "中位数 {{median}},99% 小于 {{p99}}",
288
+ "detail.example": "示例",
289
+ "detail.noFile": "无数据文件",
290
+ "detail.take": "获取",
291
+ "detail.more": "批次与前几行",
292
+ "detail.notCounted": "未统计",
293
+
294
+ "get.nothing": "没有可获取的内容",
295
+ "get.takeOne_other": "从 {{title}} 获取 {{count}} 个单元",
296
+ "get.takeEach_other": "从 {{sources}}中各获取 {{count}} 个单元",
297
+ "get.command": "命令",
298
+ "get.sourceWord_other": "个数据源",
299
+ "get.wholeIndex": "整个索引",
300
+ "get.change": "修改",
301
+ "get.look": "先预览",
302
+ "get.take": "获取",
303
+ "get.sql": "DuckDB SQL,固定到该提交",
304
+ "sql.head": "Agentic Env 索引 @ {{sha}}:{{sources}},{{rows}}",
305
+ "sql.all": "全部数据源",
306
+ "sql.chip": "{{k}}:{{v}}",
307
+ "csv.progress_other": "{{done}}/{{n}} 个文件",
308
+ "csv.button": "CSV · {{rows}}",
309
+ "csv.tooMany": "超过 {{n}} 行:请使用命令或 SQL",
310
+ "tree.title": "写入内容",
311
+ "tree.units": "单元、README、LICENSE",
312
+ "tree.answers": "答案,单独存放",
313
+ "tree.manifest": "每个文件一行,已校验 sha256",
314
+ "tree.sizes": "每个数据源一行",
315
+ "tree.domainNote": ":例如 D3_data 或 D3",
316
+ "tree.leftOut": "默认排除",
317
+ "profile.duplicates": "重复内容",
318
+ "profile.benchmark": "基准副本",
319
+ "profile.licence": "许可证不允许分发",
320
+ "profile.english": "非英文",
321
+ "profile.personal": "个人数据",
322
+ "profile.large": "超过 20 MB",
323
+ "agents.text": "目录字段 · 一条命令 · 默认配置 · 答案存放位置",
324
+ "agents.readme": "README:给智能体",
325
+
326
+ "excluded.head_other": "已审核并排除 {{n}} 项",
327
+ "excluded.none": "没有排除项",
328
+ "excluded.reason": "原因",
329
+ "excluded.requirement": "要求",
330
+ "excluded.items": "项数",
331
+ "excluded.unit": "单元",
332
+ "excluded.notMet": "未满足的要求",
333
+ "excluded.detail": "详情",
334
+ "excluded.reviewed": "审核日期",
335
+ "excluded.wholeSource": "整个数据源",
336
+ "history.title": "历史:每次快照后的数据源数",
337
+ "history.start": "起点",
338
+ "history.date": "日期",
339
+ "history.entry": "条目",
340
+ "history.sourcesAdded": "新增数据源",
341
+ "history.rowsAdded": "新增行数",
342
+ "history.started": "日志开始"
343
+ }
portal.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
- "commit": "7915a71d8ec281a21702dcee20a64e74130cf7ef",
3
- "digest": "31ed6b99c727afdb55491eece238f46314f674ecab52486cc1a394ada47f0bdf",
4
- "files": 9,
5
  "format": "portal-deploy/1",
6
- "index": "75e29bbc75408ca6b9a6097bd5227763860bd43a"
7
  }
 
1
  {
2
+ "commit": "2da961963825eb361a239d6c04fb0a0b48af73b1",
3
+ "digest": "4e64fccdd34a52a2ad12f75da2950a09c332c6acc32c3fe2706eb47f22ff476e",
4
+ "files": 12,
5
  "format": "portal-deploy/1",
6
+ "index": "02b52f1fb8ec44ec91c6593f759faff0be42b200"
7
  }
style.css CHANGED
@@ -36,6 +36,8 @@
36
  --shadow: 0 1px 2px rgba(17, 37, 50, 0.06), 0 8px 24px -12px rgba(17, 37, 50, 0.18);
37
  --shadow-lift: 0 2px 4px rgba(17, 37, 50, 0.08), 0 18px 40px -16px rgba(17, 37, 50, 0.28);
38
  --sans: "Inter", "Noto Sans SC", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
 
 
39
  --mono: "IBM Plex Mono", ui-monospace, "Cascadia Mono", Consolas, monospace;
40
  /* type */
41
  --fs-small: 13px;
@@ -53,6 +55,7 @@
53
  --t: 160ms;
54
  }
55
  @media (prefers-reduced-motion: reduce) { :root { --t: 0ms; } }
 
56
 
57
  * { box-sizing: border-box; }
58
  [hidden] { display: none !important; }
@@ -121,6 +124,7 @@ button.link { border: 0; background: none; color: var(--link); text-decoration:
121
  .nav-links { display: flex; align-items: center; gap: var(--s2); font-size: 14px; }
122
  .nav-links a:not(.btn) { display: inline-flex; align-items: center; gap: 6px; color: var(--ink-2); text-decoration: none; }
123
  .nav-links a:not(.btn):hover { color: var(--link); text-decoration: underline; }
 
124
 
125
  /* ---- skeletons: shapes, never words ---- */
126
  .sk { display: block; border-radius: 8px; background: linear-gradient(90deg, #e3eaef 0%, #f3f7f9 50%, #e3eaef 100%); background-size: 200% 100%; }
@@ -531,8 +535,9 @@ details.long[open] > summary { margin-bottom: var(--s1); }
531
  @media (max-width: 720px) {
532
  :root { --fs-small: 12px; --fs-body: 15px; --fs-sub: 18px; --fs-h2: 26px; --fs-h1: 28px; --nav-h: 56px; --r: 14px; }
533
  .nav-row { grid-template-columns: auto minmax(0, 1fr); gap: var(--s2); }
534
- .brand, .nav-links { display: none; }
535
- .nav-row { grid-template-columns: minmax(0, 1fr); }
 
536
  .progress { gap: 6px; }
537
  .progress a { font-size: 11px; column-gap: 3px; }
538
  .hero-grid { padding-top: var(--s5); gap: var(--s4); }
 
36
  --shadow: 0 1px 2px rgba(17, 37, 50, 0.06), 0 8px 24px -12px rgba(17, 37, 50, 0.18);
37
  --shadow-lift: 0 2px 4px rgba(17, 37, 50, 0.08), 0 18px 40px -16px rgba(17, 37, 50, 0.28);
38
  --sans: "Inter", "Noto Sans SC", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
39
+ /* Chinese text (the page in Chinese, or the language switch while it names Chinese): Noto Sans SC first */
40
+ --sans-zh: "Noto Sans SC", "Inter", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
41
  --mono: "IBM Plex Mono", ui-monospace, "Cascadia Mono", Consolas, monospace;
42
  /* type */
43
  --fs-small: 13px;
 
55
  --t: 160ms;
56
  }
57
  @media (prefers-reduced-motion: reduce) { :root { --t: 0ms; } }
58
+ html:lang(zh) { --sans: var(--sans-zh); }
59
 
60
  * { box-sizing: border-box; }
61
  [hidden] { display: none !important; }
 
124
  .nav-links { display: flex; align-items: center; gap: var(--s2); font-size: 14px; }
125
  .nav-links a:not(.btn) { display: inline-flex; align-items: center; gap: 6px; color: var(--ink-2); text-decoration: none; }
126
  .nav-links a:not(.btn):hover { color: var(--link); text-decoration: underline; }
127
+ .lang-btn:lang(zh) { font-family: var(--sans-zh); }
128
 
129
  /* ---- skeletons: shapes, never words ---- */
130
  .sk { display: block; border-radius: 8px; background: linear-gradient(90deg, #e3eaef 0%, #f3f7f9 50%, #e3eaef 100%); background-size: 200% 100%; }
 
535
  @media (max-width: 720px) {
536
  :root { --fs-small: 12px; --fs-body: 15px; --fs-sub: 18px; --fs-h2: 26px; --fs-h1: 28px; --nav-h: 56px; --r: 14px; }
537
  .nav-row { grid-template-columns: auto minmax(0, 1fr); gap: var(--s2); }
538
+ /* on a phone the nav keeps the progress bar and the language switch */
539
+ .brand, .nav-links > :not(.lang-btn) { display: none; }
540
+ .nav-row { grid-template-columns: minmax(0, 1fr) auto; }
541
  .progress { gap: 6px; }
542
  .progress a { font-size: 11px; column-gap: 3px; }
543
  .hero-grid { padding-top: var(--s5); gap: var(--s4); }