data-line commited on
Commit
975e4ee
·
1 Parent(s): bec7df7

portal: catalog page of repository commit ae5043d3f133

Browse files

Files of catalog/portal/ (18), digest 5defae89560cdd5321dbce69717fc621ee175790ca7b87b2fa198a863ea3b875.

Files changed (14) hide show
  1. README.md +59 -17
  2. app.js +73 -7
  3. data.js +6 -0
  4. first.js +6 -3
  5. index.html +22 -8
  6. llms.txt +1 -1
  7. locales/en.json +41 -16
  8. locales/zh.json +57 -33
  9. manual.js +59 -0
  10. manual/manual.en.md +142 -0
  11. manual/manual.zh.md +142 -0
  12. manual/shots.yml +42 -0
  13. portal.json +3 -3
  14. style.css +23 -2
README.md CHANGED
@@ -16,7 +16,7 @@ datasets:
16
  A static page into the public pointer index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index)
17
  for people who build tasks for a terminal agent, and for their agent: task material (material sets in their
18
  collections) and corpora as two layers, one card per collection, the largest material sets, the most requested
19
- categories, the domain × form map with the bars by domain and by form for one layer at a time (linked: a click on one
20
  filters the others and the list), the material sets or corpora with their fit and a preview of their rows, and one
21
  command per layer to take a sample.
22
  There is no backend; the page reads the dataset in the browser. Agents read [llms.txt](llms.txt).
@@ -47,6 +47,10 @@ The index's dataset card and `docs/sources.md` keep the two layers in separate t
47
 
48
  One pair of names is used everywhere: "corpus" and "corpora", "material set" and "material sets"
49
  (in Chinese the two names that `locales/zh.json` has for them, `kind.corpus` and `kind.material_set`).
 
 
 
 
50
  No number and no heading counts "sources" over both layers.
51
 
52
  Every number says which layer it counts: task material in collections, material sets, units and units with an answer file;
@@ -56,12 +60,12 @@ When `meta/catalog.parquet` has the columns `units` (distinct units; a corpus co
56
  an answer file) and `files` (files of every role), the page reads them as they are.
57
  Until then a material set's units are its answer files when it has any (answers kept apart), else its unit rows (tests
58
  and worked examples hold their expected results inside).
59
- A category card leads with the units of its material sets ("none yet" at 0) and says in a second line how many corpus
60
- rows the category also has to search;
61
  the two are never added.
62
- "Shippable" is the share of rows whose redistribution is yes
63
- or conditional (`license_rows.redistribution_ok`), weighted by the units of material sets ("Shippable, material sets"),
64
- or by the rows of corpora when a category has no material set ("Shippable, corpora").
65
  A collection card shows the units of its material sets, its material sets, its units with an answer file and its main
66
  domains;
67
  a count of 0 is left out, and the card carries no collection code (#1197: codes stay where someone types them, in the
@@ -90,7 +94,10 @@ the recorded size (`bytes`, the sum of `size_bytes` over the rows that have one)
90
  the deduplicated size (`distinct_bytes`, the recorded size minus the sizes of the rows that repeat another row's
91
  content);
92
  the check (`size_check`: of the rows read from `meta/sizes.parquet` that compared a size or hash, `compared`, the
93
- share that match their recorded size, "Size check: 98.8% match (3,413 rows compared)", #1225);
 
 
 
94
  and, for the rows without a recorded size (`unsized_rows`), their estimate (`unsized_estimate`), marked as an estimate,
95
  with its 95% range.
96
  Above a source the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider
@@ -132,7 +139,8 @@ The first screen (`first.js`) and the reader of the index (`data.js`) share that
132
 
133
  | File | Read | Fields used |
134
  |---|---|---|
135
- | `meta/catalog.parquet` | at load; then loaded whole into DuckDB-WASM as the table `catalog` | `source`, `title`, `summary`, `description`, `status`, `status_reasons`, `kind` (when published), `collection` (when published), `domain`, `form` (when published), `license` (`spdx`, `tier`, `note`, `ship`), `contamination` (`level`, `benchmark`, `scope`; level `not scanned` reads as not known), `pii` (`level`, `evidence`), `files`, `license_counts` (when published: per material set `kind`, `upstream` or `index` for the files the index wrote under `campaign_files/`, `spdx`, `training_ok`, `redistribution_ok` and `files`, counted by the projection over all its rows), `bytes`, `distinct_bytes` (when published), `unsized_rows`, `unit`, `top_types`, `needs_auth` |
 
136
  | `meta/size_figures.json` | at load, when the commit has it | per level (`index`, `layers`, `collections`, `sources`): `bytes`, `distinct_bytes`, `unsized_rows`, `check` (`rows`, `compared`, `match`, `unreachable`, `gone`, `needs_auth`, `no_length`, `error`), `estimate` (`bytes`, `low`, `high`, `readings`, `rows`) |
137
  | `meta/sources.json` | at load | per source: `license_rows` (training and redistribution, rows per answer), `license` (`name`, `statement`, `attribution`, `url`), `by_ext`, `size_quantiles_upper_bytes` |
138
  | `meta/coverage.json` | at load | `by_license.training_ok` (rows per answer) |
@@ -201,7 +209,7 @@ the other filters (leaf, form, status, training, redistribution, size, file type
201
  `details` element closed by default and open when one of them is set (#1197).
202
  The panel has a `max-height` with its own scroll, so it never runs past the viewport.
203
  The filters are the clauses of one crossfilter selection (Mosaic, `linked.js`), whatever sets them: the filter panel, a
204
- click on a bar of the domain or form chart, a click on a map cell (domain and form), a category card, a collection card
205
  or the layer switch.
206
  Each chart is filtered by every clause but its own (a click on a domain bar filters the map, the form bars and the
207
  source list, and the domain bars stay whole); the source list and "Get the data" read every clause.
@@ -217,6 +225,9 @@ there), most rows first, as CKAN's Formats facet and the Hub's Format filter do;
217
  the keys in brackets (`(none)`, `(other)`) name no extension and are not listed.
218
  Selecting types keeps the sources with rows of any of them; the command adds one `--ext <type>` per selected type, the
219
  SQL tab `WHERE ext IN (...)`, and the CSV keeps only the rows of those types.
 
 
 
220
 
221
  ## Languages
222
 
@@ -248,7 +259,9 @@ first, `:lang(zh)` in `style.css`).
248
  The detector reads the query string first (`?lng=zh`), then localStorage, and keeps the choice in localStorage
249
  (`i18nextLng`), so the page opens in English unless the reader chose Chinese;
250
  the switch drops a `lng` from the address, so the kept choice holds on the next visit.
251
- `llms.txt` and this README stay in English.
 
 
252
 
253
  ## Files
254
 
@@ -271,6 +284,15 @@ the switch drops a `lng` from the address, so the kept choice holds on the next
271
  - `i18n.js`: the page's languages (i18next with its browser language detector, the words from `locales/en.json` and
272
  `locales/zh.json`), the words of `index.html` (`data-i18n` sets an element's text, `data-i18n-attr` its attributes)
273
  and the language switch; the other modules draw their text again on i18next's `languageChanged`.
 
 
 
 
 
 
 
 
 
274
  - `linked.js`: the linked views of the browse section, by Mosaic vgplot: DuckDB-WASM through `wasmConnector()`, the
275
  tables `catalog`, `page` and `src` and the view `dom`, the domain × form map and the bars by domain and by form, of
276
  the layer the clause `kind` selects, in its color (material sets `#4F6D80`, corpora `#8C8C8C`): the bars with the
@@ -301,7 +323,7 @@ the switch drops a `lng` from the address, so the kept choice holds on the next
301
  material sets and its units with an answer file side by side at one size, named as on the first screen, its units in
302
  one small line under them; its main domains wrap to at most two lines, with an ellipsis and the full text as the
303
  title), the largest material sets as
304
- bars (the rest in one line under the chart and its table, never as a bar), the most requested categories (the constant
305
  `MOST_REQUESTED`, ten: one card each, in two rows of five from 1,100 px; those with material by size, then the
306
  wanted ones with a request; `index.html` has ten skeleton cards to match), the filters and their URL state (`F`), the
307
  chips of the selection, the sections, the fit of a source, the command, the SQL and the CSV.
@@ -315,7 +337,11 @@ the switch drops a `lng` from the address, so the kept choice holds on the next
315
  #1302: an address in `uri` is cut in its middle, its host at the start and its file name at the end, with the whole
316
  address on hover, and the table's caption is in the page's language only;
317
  each fit dot names its check and its state on hover (`title` and `aria-label`), and the header of the fit column
318
- names the seven checks with an icon and a short label (`fitShort.*`), each above its dot;
 
 
 
 
319
  the caret with the title, the kind mark (only when the list holds both layers) and the id start at the title's left
320
  edge;
321
  in a collection whose material sets all have one form, the form cells are left out (#1216).
@@ -340,8 +366,10 @@ the switch drops a `lng` from the address, so the kept choice holds on the next
340
  a picture of what its output folder will hold (`commandArgs`, `layerArgs`, `selectionArgs` and `folderTree`).
341
  Beside it, two downloads keep the selection: the selection file (`agentic-env-selection-<commit>.json`: the ids of
342
  the sources, the filters with the layer and the same release tag as its revision), which `fetch.py --selection`
343
- reads, with the command that reads it pinned to the same tag; and the attribution file (Markdown, in English: per source its title, license, license URL and
344
- attribution statement from `meta/sources.json`).
 
 
345
  #1286: each row of the list, in either layer, has a "Pick" toggle that reads "Picked" once on;
346
  the picks are kept in the address as `pick=`, as the filters are.
347
  While at least one source is picked, "Get the data" says how many and shows, for exactly them, the same "Look first"
@@ -394,8 +422,9 @@ Fonts: Inter (400 to 800) for text and numbers, Noto Sans SC for CJK, IBM Plex M
394
  Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
395
 
396
  Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
397
- are zstd-compressed), `i18next@26.4.5` and `i18next-browser-languagedetector@8.2.1`, and `@uwdata/vgplot@0.34.3`
398
- (Mosaic, an ES module, loaded after the first screen has painted; it brings DuckDB-WASM from jsDelivr).
 
399
  There is no build step. To look at it locally: `python -m http.server` in this folder (without a
400
  `first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
401
  writes one).
@@ -408,4 +437,17 @@ Space's `portal.json` records, after the push has passed its outbound checks. It
408
  numbers, per layer and never added together (task material: collections, material sets, units, units with an answer file;
409
  corpora: corpora and rows), and the size figures of the whole index and each layer (`sizes`, from
410
  `meta/size_figures.json`, format `portal-first/3`), from the commit main of the index points to,
411
- computed with the same rule as `data.js` (the catalog column `kind` when the commit has it).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  A static page into the public pointer index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index)
17
  for people who build tasks for a terminal agent, and for their agent: task material (material sets in their
18
  collections) and corpora as two layers, one card per collection, the largest material sets, the most requested
19
+ domains, the domain × form map with the bars by domain and by form for one layer at a time (linked: a click on one
20
  filters the others and the list), the material sets or corpora with their fit and a preview of their rows, and one
21
  command per layer to take a sample.
22
  There is no backend; the page reads the dataset in the browser. Agents read [llms.txt](llms.txt).
 
47
 
48
  One pair of names is used everywhere: "corpus" and "corpora", "material set" and "material sets"
49
  (in Chinese the two names that `locales/zh.json` has for them, `kind.corpus` and `kind.material_set`).
50
+ #1577: one word per concept in each language, the same as in the index's docs: a source is "source" (in Chinese the
51
+ word of `col.title`), its card the source card, a domain "domain" (in Chinese the word of `filter.domain`; never
52
+ "category"), and the redistribution permit "redistribution" (in Chinese the word of `chip.redistribution`; never
53
+ "shipping", which names the card field `license.ship`).
54
  No number and no heading counts "sources" over both layers.
55
 
56
  Every number says which layer it counts: task material in collections, material sets, units and units with an answer file;
 
60
  an answer file) and `files` (files of every role), the page reads them as they are.
61
  Until then a material set's units are its answer files when it has any (answers kept apart), else its unit rows (tests
62
  and worked examples hold their expected results inside).
63
+ A domain card leads with the units of its material sets ("none yet" at 0) and says in a second line how many corpus
64
+ rows the domain also has to search;
65
  the two are never added.
66
+ "Redistributable" is the share of rows whose redistribution is yes
67
+ or conditional (`license_rows.redistribution_ok`), weighted by the units of material sets ("Redistributable, material
68
+ sets"), or by the rows of corpora when a domain has no material set ("Redistributable, corpora").
69
  A collection card shows the units of its material sets, its material sets, its units with an answer file and its main
70
  domains;
71
  a count of 0 is left out, and the card carries no collection code (#1197: codes stay where someone types them, in the
 
94
  the deduplicated size (`distinct_bytes`, the recorded size minus the sizes of the rows that repeat another row's
95
  content);
96
  the check (`size_check`: of the rows read from `meta/sizes.parquet` that compared a size or hash, `compared`, the
97
+ share that match their recorded size, "Size check: 98.8% match (3,413 rows compared)", #1225; #1577: a level with
98
+ rows read but none compared shows its own reading, the rows read and what kept each from its file, "Size check: none
99
+ of the 9 rows read could be compared (address unreachable 9)", never "no row compared yet", which is left for a level
100
+ none of whose rows is read);
101
  and, for the rows without a recorded size (`unsized_rows`), their estimate (`unsized_estimate`), marked as an estimate,
102
  with its 95% range.
103
  Above a source the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider
 
139
 
140
  | File | Read | Fields used |
141
  |---|---|---|
142
+ | `meta/catalog.parquet` | at load; then loaded whole into DuckDB-WASM as the table `catalog` | `source`, `title`, `summary`, `description`, `status`, `status_reasons`, `kind` (when published), `collection` (when published), `domain`, `form` (when published), `license` (`spdx`, `tier`, `note`, `ship`), `contamination` (`level`, `benchmark`, `scope`; level `not scanned` reads as not known), `pii` (`level`, `evidence`), `files`, `license_counts` (when published: per material set `kind`, `upstream` or `index` for the files the index wrote under `campaign_files/`, `spdx`, `training_ok`, `redistribution_ok` and `files`, counted by the projection over all its rows), `bytes`, `distinct_bytes` (when published), `unsized_rows`, `fetch` (when published, #1577: `level`, `rows`, `fetched`, `read_at`), `unit`, `top_types`, `needs_auth` |
143
+ | `meta/attribution.parquet` | when the attribution file is made, when the commit has it (#1577) | `source`, `publisher`, `license`, `license_url`, `files` |
144
  | `meta/size_figures.json` | at load, when the commit has it | per level (`index`, `layers`, `collections`, `sources`): `bytes`, `distinct_bytes`, `unsized_rows`, `check` (`rows`, `compared`, `match`, `unreachable`, `gone`, `needs_auth`, `no_length`, `error`), `estimate` (`bytes`, `low`, `high`, `readings`, `rows`) |
145
  | `meta/sources.json` | at load | per source: `license_rows` (training and redistribution, rows per answer), `license` (`name`, `statement`, `attribution`, `url`), `by_ext`, `size_quantiles_upper_bytes` |
146
  | `meta/coverage.json` | at load | `by_license.training_ok` (rows per answer) |
 
209
  `details` element closed by default and open when one of them is set (#1197).
210
  The panel has a `max-height` with its own scroll, so it never runs past the viewport.
211
  The filters are the clauses of one crossfilter selection (Mosaic, `linked.js`), whatever sets them: the filter panel, a
212
+ click on a bar of the domain or form chart, a click on a map cell (domain and form), a domain card, a collection card
213
  or the layer switch.
214
  Each chart is filtered by every clause but its own (a click on a domain bar filters the map, the form bars and the
215
  source list, and the domain bars stay whole); the source list and "Get the data" read every clause.
 
225
  the keys in brackets (`(none)`, `(other)`) name no extension and are not listed.
226
  Selecting types keeps the sources with rows of any of them; the command adds one `--ext <type>` per selected type, the
227
  SQL tab `WHERE ext IN (...)`, and the CSV keeps only the rows of those types.
228
+ #1577: the SQL tab's query reads the rows as published and applies no rule of the default profile of `tools/fetch.py`
229
+ (no deduplication, and none of the benchmark, license, language, personal data and size rules);
230
+ a comment line in the query says so (`sql.noProfile`), so a copied query carries it.
231
 
232
  ## Languages
233
 
 
259
  The detector reads the query string first (`?lng=zh`), then localStorage, and keeps the choice in localStorage
260
  (`i18nextLng`), so the page opens in English unless the reader chose Chinese;
261
  the switch drops a `lng` from the address, so the kept choice holds on the next visit.
262
+ `llms.txt` and this README stay in English;
263
+ the manual has a text of its own per language (`manual/manual.en.md`, `manual/manual.zh.md`), and the view shows the one of the
264
+ page's language.
265
 
266
  ## Files
267
 
 
284
  - `i18n.js`: the page's languages (i18next with its browser language detector, the words from `locales/en.json` and
285
  `locales/zh.json`), the words of `index.html` (`data-i18n` sets an element's text, `data-i18n-attr` its attributes)
286
  and the language switch; the other modules draw their text again on i18next's `languageChanged`.
287
+ - `manual/`, `manual.js` (#1617): the user manual for people who build data from the index, a view of its own at the
288
+ address `#manual` (the links "Manual" in the header, under the first screen and in the footer; `app.js` `showView`
289
+ shows it in place of the page, and any other address shows the page again).
290
+ Its master copy is here: `manual/manual.en.md` and `manual/manual.zh.md`, one text per language in the four parts of Diátaxis
291
+ (tutorial, how-to guides, reference, explanation), and `manual/shots.yml`, its screenshots in shot-scraper's `multi`
292
+ format (per shot `url`, `output`, `width`, `selector`, `javascript`, `wait_for`).
293
+ `manual.js` renders the text of the page's language with markdown-it (CommonMark), again when the language changes;
294
+ an image names its shot's `output` and is read from `manual/shots/`, and a shot with no image there yet shows its
295
+ name in its place.
296
  - `linked.js`: the linked views of the browse section, by Mosaic vgplot: DuckDB-WASM through `wasmConnector()`, the
297
  tables `catalog`, `page` and `src` and the view `dom`, the domain × form map and the bars by domain and by form, of
298
  the layer the clause `kind` selects, in its color (material sets `#4F6D80`, corpora `#8C8C8C`): the bars with the
 
323
  material sets and its units with an answer file side by side at one size, named as on the first screen, its units in
324
  one small line under them; its main domains wrap to at most two lines, with an ellipsis and the full text as the
325
  title), the largest material sets as
326
+ bars (the rest in one line under the chart and its table, never as a bar), the most requested domains (the constant
327
  `MOST_REQUESTED`, ten: one card each, in two rows of five from 1,100 px; those with material by size, then the
328
  wanted ones with a request; `index.html` has ten skeleton cards to match), the filters and their URL state (`F`), the
329
  chips of the selection, the sections, the fit of a source, the command, the SQL and the CSV.
 
337
  #1302: an address in `uri` is cut in its middle, its host at the start and its file name at the end, with the whole
338
  address on hover, and the table's caption is in the page's language only;
339
  each fit dot names its check and its state on hover (`title` and `aria-label`), and the header of the fit column
340
+ names the eight checks with an icon and a short label (`fitShort.*`), each above its dot;
341
+ #1577: the eighth, "Upstream serves the files", reads the catalog column `fetch`, the projection's reading of the
342
+ source's sampled rows in `meta/sizes.parquet` (`all` fits, `some` partly, `none` does not, `not read` is not known),
343
+ and the detail says it in a line of its own, "Fetch check", with the rows read, those that fetched and matched their
344
+ recorded size and hash, and the day read;
345
  the caret with the title, the kind mark (only when the list holds both layers) and the id start at the title's left
346
  edge;
347
  in a collection whose material sets all have one form, the form cells are left out (#1216).
 
366
  a picture of what its output folder will hold (`commandArgs`, `layerArgs`, `selectionArgs` and `folderTree`).
367
  Beside it, two downloads keep the selection: the selection file (`agentic-env-selection-<commit>.json`: the ids of
368
  the sources, the filters with the layer and the same release tag as its revision), which `fetch.py --selection`
369
+ reads, with the command that reads it pinned to the same tag; and the attribution file (Markdown, in English: per source its title, license and
370
+ attribution statement from `meta/sources.json`; #1577: and its files by publisher, license and license URL from
371
+ `meta/attribution.parquet`, each file's own values, its source's license and license URL where its row names none;
372
+ a source without rows there, or a commit without the table, gives its source-level license URL).
373
  #1286: each row of the list, in either layer, has a "Pick" toggle that reads "Picked" once on;
374
  the picks are kept in the address as `pick=`, as the filters are.
375
  While at least one source is picked, "Get the data" says how many and shows, for exactly them, the same "Look first"
 
422
  Motion (update highlights, sections fading in, cards lifting, view transitions) is off under `prefers-reduced-motion`.
423
 
424
  Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
425
+ are zstd-compressed), `i18next@26.4.5` and `i18next-browser-languagedetector@8.2.1`, `@uwdata/vgplot@0.34.3`
426
+ (Mosaic, an ES module, loaded after the first screen has painted; it brings DuckDB-WASM from jsDelivr), and
427
+ `markdown-it@15.0.3` (the manual's Markdown, loaded with `manual.js` the first time the manual is shown).
428
  There is no build step. To look at it locally: `python -m http.server` in this folder (without a
429
  `first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
430
  writes one).
 
437
  numbers, per layer and never added together (task material: collections, material sets, units, units with an answer file;
438
  corpora: corpora and rows), and the size figures of the whole index and each layer (`sizes`, from
439
  `meta/size_figures.json`, format `portal-first/3`), from the commit main of the index points to,
440
+ computed with the same rule as `data.js` (the catalog column `kind` when the commit has it).
441
+
442
+ #1617: the manual's texts go to the Space with their markers filled.
443
+ `{first:<key>}` is a number of `first.json` (the same computation; a count with thousands separators), and
444
+ `{first:index}` the index commit the deploy reads.
445
+ A line `{run:<command>}` becomes the command after `$ ` with its output below it: the deploy runs the command (with
446
+ `{first:index}` filled in) against that commit, within a time limit.
447
+ A command that fails or runs out of time makes the deploy step red, and the Space keeps its previous manual.
448
+ After the push, a step of the same publish run (`03_data_line/steps/shots.py`) waits until the live page serves the
449
+ deploy and takes every shot of `manual/shots.yml` on it with `shot-scraper multi` (Playwright, Chromium), each shot
450
+ waiting for its element with `wait_for`;
451
+ the images go to the Space under `manual/shots/` in a commit of their own after the deploy's, through the same
452
+ outbound gate.
453
+ A shot that fails keeps its previous image and is named in that step's summary, and the step is red.
app.js CHANGED
@@ -10,7 +10,7 @@ import {
10
  } from './data.js';
11
  import {
12
  clear, columns, compactParts, el, figureWithToggle, fmtAnd, fmtBytes, fmtCompact, fmtDate, fmtInt, fmtOr, fmtPct,
13
- fmtShort, hbars, hideTip, meter, segmented, stack, table, tableShown, vt, withTip,
14
  } from './charts.js';
15
  import { decorate, drawHero, markFromIndex, setNumber, sizeLine, sizeParts, sizeShort } from './first.js';
16
  import { lang, onLanguage, setUpSwitch, t, tEn, translatePage } from './i18n.js';
@@ -34,6 +34,8 @@ const MAX_FILE = 20e6;
34
  const MAX_LISTED = 30;
35
  // the phone layout of style.css
36
  const PHONE = '(max-width: 720px)';
 
 
37
  const $ = (id) => document.getElementById(id);
38
 
39
  // ---- plain names (codes only inside the SQL and the commands); the words are in the locale files, by code:
@@ -173,9 +175,19 @@ const FIT = [
173
  { k: 'alone', icon: 'doc' },
174
  { k: 'answers', icon: 'check' },
175
  { k: 'login', icon: 'link' },
 
 
176
  ];
177
  const fitLabel = (k) => t(`fit.${k}`);
178
  const fitWord = (state) => t(`fitWord.${state}`);
 
 
 
 
 
 
 
 
179
  function fitOf(s) {
180
  const q = s.info.size_quantiles_upper_bytes || {};
181
  const c = s.contamination?.level;
@@ -190,6 +202,7 @@ function fitOf(s) {
190
  alone: kinds.every((k) => ['unit', 'file', 'API response'].includes(k)) ? 'yes' : 'part',
191
  answers: s.isTask ? 'yes' : 'no',
192
  login: s.needsAuth === 'no' ? 'yes' : s.needsAuth === 'mixed' ? 'part' : 'no',
 
193
  };
194
  }
195
  function fitDot(state, label) {
@@ -1142,6 +1155,9 @@ function detail(s) {
1142
  dl.append(item('size', t('detail.size'), el('span', { class: 'lic' }, el('b', {}, recorded), el('span', { class: 'small' }, amounts),
1143
  rest.length ? el('span', { class: 'small size-parts' }, rest.join(' \u00b7 ')) : null),
1144
  q.p50 ? sizeStrip(q) : null));
 
 
 
1145
  // 9 one example file (#1197): the first unit file of its preview rows (meta/preview.parquet, where a material set's
1146
  // unit files come first); a source with none there has no example
1147
  const pv = previewOf(s);
@@ -1209,6 +1225,8 @@ function selectionSql(sel) {
1209
  return [
1210
  `-- ${head}`,
1211
  `-- ${chips.length ? chips.map(([k, v]) => t('sql.chip', { k, v })).join('; ') : t('sql.all')}`,
 
 
1212
  `SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
1213
  'FROM read_parquet([',
1214
  paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
@@ -1274,24 +1292,43 @@ function download(name, text, type) {
1274
  }
1275
  // ---- the selection kept as files: the selection file (the ids, the filters, the release tag of the commit, which the
1276
  // command beside it names too; tools/fetch.py --selection reads it, so the same file gives the same files later) and
1277
- // the attribution file (per source its title, licence, licence URL and attribution statement from meta/sources.json,
 
1278
  // in English, the language of the dataset)
1279
  const selectionName = () => `agentic-env-selection-${IX.sha.slice(0, 7)}.json`;
 
 
1280
  function selectionFile(sel, picks) {
1281
  return `${JSON.stringify({
1282
  format: 'agentic-env-selection/1', dataset: REPO, revision: IX.pin,
1283
  filters: picks ? {} : Object.fromEntries([...activeFilters(), 'kind'].map((k) => [k, F[k]])), sources: sel.map((s) => s.id).sort(),
1284
  }, null, 1)}\n`;
1285
  }
1286
- function attributionFile(sel) {
 
 
 
 
 
 
 
 
 
1287
  const lines = [`# ${tEn('attr.title', { count: sel.length, n: fmtInt(sel.length), dataset: REPO })}`, '',
1288
  tEn('attr.commit', { sha: IX.sha, url: `${DATASET_URL}/tree/${IX.sha}` }), ''];
1289
  for (const s of [...sel].sort((a, b) => a.title.localeCompare(b.title))) {
1290
  const lic = s.info.license || {};
 
1291
  lines.push(`## ${s.title}`, '', `- ${tEn('attr.source')}: \`${s.id}\``,
1292
- `- ${tEn('attr.licence')}: ${lic.name || s.spdx || tEn('attr.notStated')}`,
1293
- `- ${tEn('attr.licenceUrl')}: ${lic.url || tEn('attr.notStated')}`,
1294
- `- ${tEn('attr.attribution')}: ${lic.attribution || lic.statement || tEn('attr.notStated')}`, '');
 
 
 
 
 
 
1295
  }
1296
  return lines.join('\n');
1297
  }
@@ -1307,7 +1344,11 @@ function keepCard(sel, picks = false) {
1307
  el('div', { class: 'keep-actions' },
1308
  el('button', { type: 'button', class: 'btn solid', onclick: () => download(selectionName(), selectionFile(sel, picks), 'application/json') },
1309
  icon('get', { size: 16 }), t('keep.selection')),
1310
- el('button', { type: 'button', class: 'btn', onclick: () => download(`agentic-env-attribution-${IX.sha.slice(0, 7)}.md`, attributionFile(sel), 'text/markdown') },
 
 
 
 
1311
  icon('doc', { size: 16 }), t('keep.attribution'))),
1312
  el('p', { class: 'small' }, el('code', {}, selectionName()), ' ', t('keep.selectionNote', { sources: plural(picks ? 'n.picked' : layerCount(), sel.length), sha: IX.sha.slice(0, 7) }))),
1313
  codeCard('run', t('keep.again'), selectionCommand(picks), 3));
@@ -1581,6 +1622,29 @@ function reveal() {
1581
  nodes.forEach((n) => io.observe(n));
1582
  }
1583
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1584
  /** Everything that shows the selection, drawn again when the selection answers with its ids. */
1585
  function renderScoped() {
1586
  hideTip();
@@ -1660,6 +1724,8 @@ async function main() {
1660
  onLanguage(renderLanguage);
1661
  decorate();
1662
  reveal();
 
 
1663
  let ticking = false;
1664
  // the tooltip is placed for where its mark was; a scroll moves the mark away, so the tooltip goes
1665
  const onScroll = () => { hideTip(); if (!ticking) { ticking = true; requestAnimationFrame(() => { ticking = false; progress(); }); } };
 
10
  } from './data.js';
11
  import {
12
  clear, columns, compactParts, el, figureWithToggle, fmtAnd, fmtBytes, fmtCompact, fmtDate, fmtInt, fmtOr, fmtPct,
13
+ fmtShare, fmtShort, hbars, hideTip, meter, segmented, stack, table, tableShown, vt, withTip,
14
  } from './charts.js';
15
  import { decorate, drawHero, markFromIndex, setNumber, sizeLine, sizeParts, sizeShort } from './first.js';
16
  import { lang, onLanguage, setUpSwitch, t, tEn, translatePage } from './i18n.js';
 
34
  const MAX_LISTED = 30;
35
  // the phone layout of style.css
36
  const PHONE = '(max-width: 720px)';
37
+ // #1617: the address of the user manual's view (#manual; showView)
38
+ const MANUAL_VIEW = 'manual';
39
  const $ = (id) => document.getElementById(id);
40
 
41
  // ---- plain names (codes only inside the SQL and the commands); the words are in the locale files, by code:
 
175
  { k: 'alone', icon: 'doc' },
176
  { k: 'answers', icon: 'check' },
177
  { k: 'login', icon: 'link' },
178
+ // #1577: whether the upstream still serves the files (catalog column fetch)
179
+ { k: 'fetch', icon: 'get' },
180
  ];
181
  const fitLabel = (k) => t(`fit.${k}`);
182
  const fitWord = (state) => t(`fitWord.${state}`);
183
+ // #1577: the fit of a fetch level (catalog column fetch, index_projection.FETCH_LEVELS)
184
+ const FETCH_FIT = { all: 'yes', some: 'part', none: 'no' };
185
+ /** #1577: a source's fetch check in words: how many of its sampled rows read fetched and matched their record, out of
186
+ * how many, and the day read; or that none is read yet. */
187
+ function fetchText(f) {
188
+ if (!f || !FETCH_FIT[f.level]) return t('fetch.notRead');
189
+ return t(`fetch.${f.level}`, { fetched: fmtInt(f.fetched), rows: fmtInt(f.rows), share: fmtShare(f.fetched, f.rows), date: f.readAt || t('common.unknown') });
190
+ }
191
  function fitOf(s) {
192
  const q = s.info.size_quantiles_upper_bytes || {};
193
  const c = s.contamination?.level;
 
202
  alone: kinds.every((k) => ['unit', 'file', 'API response'].includes(k)) ? 'yes' : 'part',
203
  answers: s.isTask ? 'yes' : 'no',
204
  login: s.needsAuth === 'no' ? 'yes' : s.needsAuth === 'mixed' ? 'part' : 'no',
205
+ fetch: FETCH_FIT[s.fetch?.level] || 'unknown',
206
  };
207
  }
208
  function fitDot(state, label) {
 
1155
  dl.append(item('size', t('detail.size'), el('span', { class: 'lic' }, el('b', {}, recorded), el('span', { class: 'small' }, amounts),
1156
  rest.length ? el('span', { class: 'small size-parts' }, rest.join(' \u00b7 ')) : null),
1157
  q.p50 ? sizeStrip(q) : null));
1158
+ // #1577: whether its upstream still serves the files, from the sampled rows read in meta/sizes.parquet (catalog
1159
+ // column fetch)
1160
+ dl.append(item('get', t('detail.fetch'), fetchText(s.fetch)));
1161
  // 9 one example file (#1197): the first unit file of its preview rows (meta/preview.parquet, where a material set's
1162
  // unit files come first); a source with none there has no example
1163
  const pv = previewOf(s);
 
1225
  return [
1226
  `-- ${head}`,
1227
  `-- ${chips.length ? chips.map(([k, v]) => t('sql.chip', { k, v })).join('; ') : t('sql.all')}`,
1228
+ // #1577: the query reads the rows as published; the default profile of the command is not applied, and it says so
1229
+ `-- ${t('sql.noProfile')}`,
1230
  `SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
1231
  'FROM read_parquet([',
1232
  paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
 
1292
  }
1293
  // ---- the selection kept as files: the selection file (the ids, the filters, the release tag of the commit, which the
1294
  // command beside it names too; tools/fetch.py --selection reads it, so the same file gives the same files later) and
1295
+ // the attribution file (per source its title, licence and attribution statement from meta/sources.json; #1577: and
1296
+ // its files by publisher, licence and licence URL from meta/attribution.parquet, each file's own, else its source's;
1297
  // in English, the language of the dataset)
1298
  const selectionName = () => `agentic-env-selection-${IX.sha.slice(0, 7)}.json`;
1299
+ const ATTRIBUTION = 'meta/attribution.parquet';
1300
+ const ATTRIBUTION_COLUMNS = ['source', 'publisher', 'license', 'license_url', 'files'];
1301
  function selectionFile(sel, picks) {
1302
  return `${JSON.stringify({
1303
  format: 'agentic-env-selection/1', dataset: REPO, revision: IX.pin,
1304
  filters: picks ? {} : Object.fromEntries([...activeFilters(), 'kind'].map((k) => [k, F[k]])), sources: sel.map((s) => s.id).sort(),
1305
  }, null, 1)}\n`;
1306
  }
1307
+ async function attributionFile(sel) {
1308
+ const ids = new Set(sel.map((s) => s.id));
1309
+ const files = new Map();
1310
+ if (IX.hasAttribution) {
1311
+ for (const r of (await readColumns(fileUrl(IX.sha, ATTRIBUTION), ATTRIBUTION_COLUMNS)).rows) {
1312
+ if (!ids.has(r.source)) continue;
1313
+ if (!files.has(r.source)) files.set(r.source, []);
1314
+ files.get(r.source).push(r);
1315
+ }
1316
+ }
1317
  const lines = [`# ${tEn('attr.title', { count: sel.length, n: fmtInt(sel.length), dataset: REPO })}`, '',
1318
  tEn('attr.commit', { sha: IX.sha, url: `${DATASET_URL}/tree/${IX.sha}` }), ''];
1319
  for (const s of [...sel].sort((a, b) => a.title.localeCompare(b.title))) {
1320
  const lic = s.info.license || {};
1321
+ const own = files.get(s.id) || [];
1322
  lines.push(`## ${s.title}`, '', `- ${tEn('attr.source')}: \`${s.id}\``,
1323
+ `- ${tEn('attr.licence')}: ${lic.name || s.spdx || tEn('attr.notStated')}`);
1324
+ if (own.length) {
1325
+ lines.push(`- ${tEn('attr.files')}:`, ...own.map((r) => {
1326
+ const n = num(r.files);
1327
+ return ` - ${tEn('attr.fileLine', { count: n, n: fmtInt(n), publisher: r.publisher || tEn('attr.noPublisher'),
1328
+ licence: r.license || tEn('attr.notStated'), url: r.license_url || tEn('attr.notStated') })}`;
1329
+ }));
1330
+ } else lines.push(`- ${tEn('attr.licenceUrl')}: ${lic.url || tEn('attr.notStated')}`);
1331
+ lines.push(`- ${tEn('attr.attribution')}: ${lic.attribution || lic.statement || tEn('attr.notStated')}`, '');
1332
  }
1333
  return lines.join('\n');
1334
  }
 
1344
  el('div', { class: 'keep-actions' },
1345
  el('button', { type: 'button', class: 'btn solid', onclick: () => download(selectionName(), selectionFile(sel, picks), 'application/json') },
1346
  icon('get', { size: 16 }), t('keep.selection')),
1347
+ el('button', { type: 'button', class: 'btn', onclick: async (ev) => {
1348
+ const btn = ev.currentTarget;
1349
+ btn.disabled = true;
1350
+ try { download(`agentic-env-attribution-${IX.sha.slice(0, 7)}.md`, await attributionFile(sel), 'text/markdown'); } finally { btn.disabled = false; }
1351
+ } },
1352
  icon('doc', { size: 16 }), t('keep.attribution'))),
1353
  el('p', { class: 'small' }, el('code', {}, selectionName()), ' ', t('keep.selectionNote', { sources: plural(picks ? 'n.picked' : layerCount(), sel.length), sha: IX.sha.slice(0, 7) }))),
1354
  codeCard('run', t('keep.again'), selectionCommand(picks), 3));
 
1622
  nodes.forEach((n) => io.observe(n));
1623
  }
1624
 
1625
+ /** #1617: the user manual is a view of its own: the address #manual shows it in place of the page (manual.js, loaded
1626
+ * the first time it is shown, draws it in the page's language); any other address shows the page, at the block it
1627
+ * names. */
1628
+ function showView() {
1629
+ const on = location.hash === `#${MANUAL_VIEW}`;
1630
+ const back = !on && $('page').hidden;
1631
+ $('manual').hidden = !on;
1632
+ $('page').hidden = on;
1633
+ document.body.classList.toggle('manual-on', on);
1634
+ if (on) {
1635
+ scrollTo(0, 0);
1636
+ import('./manual.js').then((m) => m.drawManual($('manual-body')), (e) => {
1637
+ clear($('manual-body')).append(el('p', { class: 'small problem-text' }, t('manual.missing', { error: e.message })));
1638
+ console.error(e);
1639
+ });
1640
+ } else if (back) {
1641
+ // the charts were drawn, if at all, while the page was hidden
1642
+ redrawOnResize();
1643
+ progress();
1644
+ if (location.hash) document.getElementById(decodeURIComponent(location.hash.slice(1)))?.scrollIntoView();
1645
+ }
1646
+ }
1647
+
1648
  /** Everything that shows the selection, drawn again when the selection answers with its ids. */
1649
  function renderScoped() {
1650
  hideTip();
 
1724
  onLanguage(renderLanguage);
1725
  decorate();
1726
  reveal();
1727
+ showView();
1728
+ addEventListener('hashchange', showView);
1729
  let ticking = false;
1730
  // the tooltip is placed for where its mark was; a scroll moves the mark away, so the tooltip goes
1731
  const onScroll = () => { hideTip(); if (!ticking) { ticking = true; requestAnimationFrame(() => { ticking = false; progress(); }); } };
data.js CHANGED
@@ -394,6 +394,9 @@ export async function loadIndex(params, onNote) {
394
  distinct: has('distinct_bytes') ? num(r.distinct_bytes) : null,
395
  // #1158: the source's size figures from meta/size_figures.json (recorded, deduplicated, check, estimate)
396
  size: figures?.sources?.[r.source] || null,
 
 
 
397
  topTypes: Array.isArray(r.top_types) ? r.top_types : r.top_types?.toArray?.() || [],
398
  needsAuth: r.needs_auth || null,
399
  train: permission(info.license_rows?.training_ok),
@@ -414,6 +417,9 @@ export async function loadIndex(params, onNote) {
414
  date: rev.date,
415
  overrides: { catalog: params.get('catalog'), vocabulary: params.get('vocabulary') },
416
  hasVocabulary: Boolean(vocabJson),
 
 
 
417
  hasKind,
418
  vocab,
419
  sources,
 
394
  distinct: has('distinct_bytes') ? num(r.distinct_bytes) : null,
395
  // #1158: the source's size figures from meta/size_figures.json (recorded, deduplicated, check, estimate)
396
  size: figures?.sources?.[r.source] || null,
397
+ // #1577: whether its upstream still serves its files (catalog column fetch: level, rows, fetched, read_at); null
398
+ // for a catalog without the column
399
+ fetch: r.fetch && r.fetch.level ? { level: String(r.fetch.level), rows: num(r.fetch.rows) || 0, fetched: num(r.fetch.fetched) || 0, readAt: r.fetch.read_at || null } : null,
400
  topTypes: Array.isArray(r.top_types) ? r.top_types : r.top_types?.toArray?.() || [],
401
  needsAuth: r.needs_auth || null,
402
  train: permission(info.license_rows?.training_ok),
 
417
  date: rev.date,
418
  overrides: { catalog: params.get('catalog'), vocabulary: params.get('vocabulary') },
419
  hasVocabulary: Boolean(vocabJson),
420
+ // #1577: who published each file of a material set and under which license, read when the attribution list is
421
+ // made; a commit before it has none
422
+ hasAttribution: has('meta/attribution.parquet'),
423
  hasKind,
424
  vocab,
425
  sources,
first.js CHANGED
@@ -37,9 +37,12 @@ export function setNumber(node, key, value, fmt) {
37
  }
38
 
39
  /** #1225: the size check of a level (check of meta/size_figures.json): the share of the rows compared that match,
40
- * with their number. */
 
 
 
41
  const checkText = (c) => (c.compared ? t('size.check', { share: fmtShare(c.match, c.compared), n: fmtInt(c.compared) })
42
- : t('size.checkNone'));
43
 
44
  /** #1158: one level's size figures (a level of meta/size_figures.json, read as they are and never summed here), each
45
  * saying what it is: the recorded size, the deduplicated size, the check (checkText) and, when the level
@@ -74,7 +77,7 @@ export function sizeShort(g, summed) {
74
  const recorded = t('size.recordedShort', { size: fmtBytes(g.bytes) });
75
  const rest = [t('size.distinctShort', { size: fmtBytes(g.distinct_bytes) })];
76
  const tip = [c.compared ? t('size.checkSentence', { share: fmtShare(c.match, c.compared), n: fmtInt(c.compared) })
77
- : t('size.checkNoneSentence')];
78
  if (g.unsized_rows) {
79
  rest.push(e ? t('size.unsizedEst', { size: fmtBytes(e.bytes) }) : t('size.unsizedNone', { rows: fmtCompact(g.unsized_rows) }));
80
  tip.push(sizeParts(g, summed).at(-1));
 
37
  }
38
 
39
  /** #1225: the size check of a level (check of meta/size_figures.json): the share of the rows compared that match,
40
+ * with their number; #1577: a level with rows read but none compared says so with its own reading, the rows read and
41
+ * what kept each from its file (the same counts as the md card's size line), never that nothing was compared yet. */
42
+ const NOT_COMPARED = ['unreachable', 'gone', 'needs_auth', 'no_length', 'error'];
43
+ const whyNot = (c) => NOT_COMPARED.filter((k) => c[k]).map((k) => t(`size.status.${k}`, { n: fmtInt(c[k]) })).join(t('size.listSep'));
44
  const checkText = (c) => (c.compared ? t('size.check', { share: fmtShare(c.match, c.compared), n: fmtInt(c.compared) })
45
+ : c.rows ? t('size.checkNotCompared', { read: fmtInt(c.rows), why: whyNot(c) }) : t('size.checkNone'));
46
 
47
  /** #1158: one level's size figures (a level of meta/size_figures.json, read as they are and never summed here), each
48
  * saying what it is: the recorded size, the deduplicated size, the check (checkText) and, when the level
 
77
  const recorded = t('size.recordedShort', { size: fmtBytes(g.bytes) });
78
  const rest = [t('size.distinctShort', { size: fmtBytes(g.distinct_bytes) })];
79
  const tip = [c.compared ? t('size.checkSentence', { share: fmtShare(c.match, c.compared), n: fmtInt(c.compared) })
80
+ : c.rows ? t('size.checkNotComparedSentence', { read: fmtInt(c.rows), why: whyNot(c) }) : t('size.checkNoneSentence')];
81
  if (g.unsized_rows) {
82
  rest.push(e ? t('size.unsizedEst', { size: fmtBytes(e.bytes) }) : t('size.unsizedNone', { rows: fmtCompact(g.unsized_rows) }));
83
  tip.push(sizeParts(g, summed).at(-1));
index.html CHANGED
@@ -6,7 +6,7 @@
6
  <title data-i18n="page.title">Agentic Env Index</title>
7
  <link rel="icon" href="data:,">
8
  <link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents">
9
- <meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it." data-i18n-attr="content:page.description">
10
  <meta name="theme-color" content="#112532">
11
  <link rel="preconnect" href="https://fonts.googleapis.com">
12
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
@@ -31,6 +31,7 @@
31
  <li><a href="#releases" data-part="releases" data-icon="releases"><span class="pl pl-long" data-i18n="nav.releases">Releases</span><span class="pl pl-short" data-i18n="nav.releasesShort">Releases</span><span class="track"><span class="fill"></span></span></a></li>
32
  </ol>
33
  <div class="nav-links">
 
34
  <a href="llms.txt" data-icon="robot" data-size="16"><span data-i18n="nav.agents">For agents</span></a>
35
  <a class="btn solid" href="https://huggingface.co/datasets/Battam/agentic-env-index/discussions/new" data-i18n="nav.request">Request data</a>
36
  <!-- the language switch: the other language's name, in that language (i18n.js) -->
@@ -39,11 +40,11 @@
39
  </div>
40
  </nav>
41
 
42
- <main>
43
  <header class="hero" id="top" data-part="overview">
44
  <div class="wrap hero-grid">
45
  <div class="hero-main">
46
- <p class="eyebrow" data-icon="sources" data-size="18"><span data-i18n="hero.eyebrow">The Agentic Env Index, by category</span></p>
47
  <h1 data-i18n="hero.title">Real material for agent tasks</h1>
48
  <!-- #1256: under the title, the sentence on how the two layers relate (meta/vocabulary.json, app.js renderLayers)
49
  and how the sizes are made, in the index's docs/sources.md (the size note under the list links there too);
@@ -51,7 +52,7 @@
51
  <p class="hero-relation" id="hero-relation"><span class="sk sk-line"></span></p>
52
  <p class="hero-how"><a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.md" data-i18n-attr="href:url.docsSources"><span data-i18n="size.how">How sizes are measured</span></a></p>
53
  <div class="hero-cta">
54
- <a class="btn hero-btn" href="#material" data-icon="browse" data-size="18"><span data-i18n="hero.browse">Browse categories</span></a>
55
  <a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18"><span data-i18n="nav.get">Get the data</span></a>
56
  </div>
57
  </div>
@@ -86,6 +87,7 @@
86
  <span class="chip chip-dark" id="preview-chip" hidden></span>
87
  <a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a>
88
  <a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Catalog fields</a>
 
89
  <span class="notice" id="notice" hidden></span>
90
  </div>
91
  </header>
@@ -103,8 +105,8 @@
103
  <div class="where-grid one">
104
  <figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
105
  </div>
106
- <div class="band-head sub-head reveal"><h3 id="cats-h" data-i18n="material.cats">Most requested categories</h3></div>
107
- <!-- one card per most requested category (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
108
  <ul class="cat-grid" id="req-cards">
109
  <li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
110
  </ul>
@@ -139,7 +141,7 @@
139
  <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
140
  <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
141
  </select></label>
142
- <label><span data-i18n="filter.shipping">Shipping</span><select id="f-redist">
143
  <option value="" data-i18n="filter.any">Any</option>
144
  <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
145
  <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
@@ -238,13 +240,24 @@
238
  </section>
239
  </main>
240
 
 
 
 
 
 
 
 
 
 
 
 
241
  <footer class="site-foot" id="request" data-part="releases">
242
  <div class="wrap foot-grid">
243
  <div class="foot-cta">
244
  <span class="foot-ico" data-icon="request" data-size="32"></span>
245
  <div>
246
  <h2 data-i18n="foot.missing">Missing material?</h2>
247
- <p data-i18n="foot.ask">Ask for a category, a form or a source.</p>
248
  </div>
249
  <a class="btn on-dark-solid" href="https://huggingface.co/datasets/Battam/agentic-env-index/discussions/new" data-i18n="nav.request">Request data</a>
250
  </div>
@@ -252,6 +265,7 @@
252
  <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a></li>
253
  <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Catalog fields</a></li>
254
  <li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li>
 
255
  <li><a href="#top" data-i18n="foot.top">Back to top</a></li>
256
  </ul>
257
  <p class="foot-pin mono" id="foot-pin"></p>
 
6
  <title data-i18n="page.title">Agentic Env Index</title>
7
  <link rel="icon" href="data:,">
8
  <link rel="alternate" type="text/plain" href="llms.txt" title="For agents" data-i18n-attr="title:nav.agents">
9
+ <meta name="description" content="Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by domain, fit at a glance, one command to take it." data-i18n-attr="content:page.description">
10
  <meta name="theme-color" content="#112532">
11
  <link rel="preconnect" href="https://fonts.googleapis.com">
12
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 
31
  <li><a href="#releases" data-part="releases" data-icon="releases"><span class="pl pl-long" data-i18n="nav.releases">Releases</span><span class="pl pl-short" data-i18n="nav.releasesShort">Releases</span><span class="track"><span class="fill"></span></span></a></li>
32
  </ol>
33
  <div class="nav-links">
34
+ <a href="#manual" data-icon="doc" data-size="16"><span data-i18n="nav.manual">Manual</span></a>
35
  <a href="llms.txt" data-icon="robot" data-size="16"><span data-i18n="nav.agents">For agents</span></a>
36
  <a class="btn solid" href="https://huggingface.co/datasets/Battam/agentic-env-index/discussions/new" data-i18n="nav.request">Request data</a>
37
  <!-- the language switch: the other language's name, in that language (i18n.js) -->
 
40
  </div>
41
  </nav>
42
 
43
+ <main id="page">
44
  <header class="hero" id="top" data-part="overview">
45
  <div class="wrap hero-grid">
46
  <div class="hero-main">
47
+ <p class="eyebrow" data-icon="sources" data-size="18"><span data-i18n="hero.eyebrow">The Agentic Env Index, by domain</span></p>
48
  <h1 data-i18n="hero.title">Real material for agent tasks</h1>
49
  <!-- #1256: under the title, the sentence on how the two layers relate (meta/vocabulary.json, app.js renderLayers)
50
  and how the sizes are made, in the index's docs/sources.md (the size note under the list links there too);
 
52
  <p class="hero-relation" id="hero-relation"><span class="sk sk-line"></span></p>
53
  <p class="hero-how"><a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.md" data-i18n-attr="href:url.docsSources"><span data-i18n="size.how">How sizes are measured</span></a></p>
54
  <div class="hero-cta">
55
+ <a class="btn hero-btn" href="#material" data-icon="browse" data-size="18"><span data-i18n="hero.browse">Browse domains</span></a>
56
  <a class="btn hero-btn ghost" href="#get" data-icon="get" data-size="18"><span data-i18n="nav.get">Get the data</span></a>
57
  </div>
58
  </div>
 
87
  <span class="chip chip-dark" id="preview-chip" hidden></span>
88
  <a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a>
89
  <a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Catalog fields</a>
90
+ <a href="#manual" data-i18n="nav.manual">Manual</a>
91
  <span class="notice" id="notice" hidden></span>
92
  </div>
93
  </header>
 
105
  <div class="where-grid one">
106
  <figure class="card chart-card where-card" id="where-src"><div class="sk sk-block"></div></figure>
107
  </div>
108
+ <div class="band-head sub-head reveal"><h3 id="cats-h" data-i18n="material.cats">Most requested domains</h3></div>
109
+ <!-- one card per most requested domain (MOST_REQUESTED in app.js, ten): two rows of five from 1,100 px -->
110
  <ul class="cat-grid" id="req-cards">
111
  <li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
112
  </ul>
 
141
  <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
142
  <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
143
  </select></label>
144
+ <label><span data-i18n="filter.shipping">Redistribution</span><select id="f-redist">
145
  <option value="" data-i18n="filter.any">Any</option>
146
  <option value="yes" data-i18n="filter.permit.yes">Allowed, all rows</option>
147
  <option value="cond" data-i18n="filter.permit.cond">Allowed or conditional</option>
 
240
  </section>
241
  </main>
242
 
243
+ <!-- #1617: the user manual, a view of its own at the address #manual (app.js showView, manual.js), in place of the page
244
+ above: manual/manual.<language>.md of this folder, its numbers and command outputs filled at deploy
245
+ (03_data_line/steps/portal.py), its images manual/shots/<output>, taken on this page after each deploy
246
+ (03_data_line/steps/shots.py) -->
247
+ <main class="manual-view" id="manual" hidden aria-label="User manual" data-i18n-attr="aria-label:manual.title">
248
+ <div class="wrap manual-wrap">
249
+ <p class="manual-back"><a href="#top" data-i18n="manual.back">Back to the page</a></p>
250
+ <article class="card manual-body" id="manual-body"><div class="sk sk-block tall"></div></article>
251
+ </div>
252
+ </main>
253
+
254
  <footer class="site-foot" id="request" data-part="releases">
255
  <div class="wrap foot-grid">
256
  <div class="foot-cta">
257
  <span class="foot-ico" data-icon="request" data-size="32"></span>
258
  <div>
259
  <h2 data-i18n="foot.missing">Missing material?</h2>
260
+ <p data-i18n="foot.ask">Ask for a domain, a form or a source.</p>
261
  </div>
262
  <a class="btn on-dark-solid" href="https://huggingface.co/datasets/Battam/agentic-env-index/discussions/new" data-i18n="nav.request">Request data</a>
263
  </div>
 
265
  <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index" data-i18n="link.dataset">Dataset</a></li>
266
  <li><a href="https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/fields.md" data-i18n="link.fields" data-i18n-attr="href:url.docsFields">Catalog fields</a></li>
267
  <li><a href="llms.txt" data-i18n="nav.agents">For agents</a></li>
268
+ <li><a href="#manual" data-i18n="nav.manual">Manual</a></li>
269
  <li><a href="#top" data-i18n="foot.top">Back to top</a></li>
270
  </ul>
271
  <p class="foot-pin mono" id="foot-pin"></p>
llms.txt CHANGED
@@ -1,7 +1,7 @@
1
  # Agentic Env Index: for agents
2
 
3
  - The index holds two layers over the same public web, kept apart and never added together. A material set is a batch of files picked from one source (a site, a repository or a dataset), for building tasks; material sets are grouped into collections by task family; corpora are whole upstreams read in bulk; material sets are not taken from the corpora. Task material comes first: material sets, grouped into collections, one per task family. A material set (task-ready) is a batch of files picked from one source (a site, a repository or a dataset), for building tasks: its files are grouped into units, its rows carry the roles unit, companion and answer (the catalog column `kind` is `material_set`, and `collection` names its collection), and each unit comes with its expected results. Corpora come second, to search when no material set fits: a corpus (raw; plural corpora) is one whole upstream that a collector keeps reading, such as Common Crawl's office documents, PyPI's release files, Mendeley Data or NVD: its rows are pointers to files or API answers, with no units and no answers, and it is mined one unit per row (`kind` is `corpus`).
4
- - Pick material sets or corpora from `meta/catalog.parquet` (one row per material set or corpus, about 50 KB; do not download `meta/` whole). Columns: `source`; `kind`, `material_set` or `corpus` (select one layer: units and rows are never added together); `collection`, the code of a material set's collection (A, B, C, G, GT, HW, M, MD, ML, OP, S, SC, SW; empty for a corpus); `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a corpus has one unit per row), `answers` (units with an answer file, the expected results kept apart) and `files` (files of every role); `publishers` (a material set's distinct publisher names over all its rows; empty when its rows name none); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (by the rule of the default profile of `tools/fetch.py` over `meta/bench_overlap.parquet` and `meta/bench_checked.parquet`: `none` when every data file is compared and kept and no row is removed, the hit class with the most removed rows when some are removed, `not scanned` otherwise); `license.ship`; `pii.level`; `distinct_bytes` (deduplicated size) and `bytes` (recorded size, the sum over rows); `size_check` (the rows read in `meta/sizes.parquet`, `compared` those that compared a size or hash, and how many have each status, `match` when the measured size agrees; the check's share is `match` over `compared`), `unsized_rows` (rows without a recorded size) and `unsized_estimate` (their estimated size, with its 95% range); the same figures for the whole index, each layer and each collection are in `meta/size_figures.json`. The vocabulary is `meta/vocabulary.json`: every domain code with its name (and `zh`, its Chinese name), what it covers and its parent, every form with its Chinese name and what it means, every category of `meta/not_included.parquet` with its requirement in English and in Chinese, the two layers with their relation, and every collection with its name, what it holds and its campaigns (take a collection with `--tag campaign:<name>` for each of its campaigns).
5
  - Size of this release: Whole index: Recorded size 4.50 PB; deduplicated size 4.09 PB; size check: 99.6% match (48,396 of the 49,407 rows read compared); not compared: address unreachable 477, file gone 131, login needed 331, no size from the server 68, other error 4; estimated size of 309,522,457 of the 309,787,645 rows without a recorded size: 657 TB (an estimate; 95% range 512 TB to 1.05 PB, the sum of its sources' ranges, wider than a joint interval; from 17,054 rows read), the other 265,188 with too few rows read for an estimate. Task material: Recorded size 2.90 GB; deduplicated size 2.82 GB; size check: 99.7% match (9,111 of the 9,135 rows read compared); not compared: address unreachable 11, file gone 3, login needed 8, other error 2. Corpora: Recorded size 4.50 PB; deduplicated size 4.09 PB; size check: 99.6% match (39,285 of the 40,272 rows read compared); not compared: address unreachable 466, file gone 128, login needed 323, no size from the server 68, other error 2; estimated size of 309,522,457 of the 309,787,645 rows without a recorded size: 657 TB (an estimate; 95% range 512 TB to 1.05 PB, the sum of its sources' ranges, wider than a joint interval; from 17,054 rows read), the other 265,188 with too few rows read for an estimate. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it. The recorded size is the sum of `size_bytes` over the rows that have one (catalog column `bytes`). The check reads a sample of rows of every data file (`meta/sizes.parquet`): 20 of its rows with a recorded size, and of its rows without one its share of its source's sample, sized for their mean size within 25% at 95% confidence by the spread of the sizes read so far (at most 400 rows per source per round, 20 for a source with none read yet); its share is the rows that match their recorded size out of the rows it compared (status `match`, `size_mismatch` or `hash_mismatch`), and the rows it could not compare, which never reached their file, are counted beside it by status (`unreachable`, `gone`, `needs_auth`, `no_length`, `error`; catalog column `size_check`). The estimate for a source's rows without a recorded size (catalog columns `unsized_rows` and `unsized_estimate`) is their number times the mean measured size of such rows read, with a 95% BCa bootstrap range (`scipy.stats.bootstrap`, its default resample count); a source with fewer than 10 such rows read has none. Above one source, the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider than a joint interval. The figures of every level are in `meta/size_figures.json`.
6
  - Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--kind` (`material_set` or `corpus`: one layer), `--domain`, `--form`, `--tag`, `--ext`, or `--selection <file>` (a selection file the data page saves: the ids of its sources and its commit); `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
7
  - It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
 
1
  # Agentic Env Index: for agents
2
 
3
  - The index holds two layers over the same public web, kept apart and never added together. A material set is a batch of files picked from one source (a site, a repository or a dataset), for building tasks; material sets are grouped into collections by task family; corpora are whole upstreams read in bulk; material sets are not taken from the corpora. Task material comes first: material sets, grouped into collections, one per task family. A material set (task-ready) is a batch of files picked from one source (a site, a repository or a dataset), for building tasks: its files are grouped into units, its rows carry the roles unit, companion and answer (the catalog column `kind` is `material_set`, and `collection` names its collection), and each unit comes with its expected results. Corpora come second, to search when no material set fits: a corpus (raw; plural corpora) is one whole upstream that a collector keeps reading, such as Common Crawl's office documents, PyPI's release files, Mendeley Data or NVD: its rows are pointers to files or API answers, with no units and no answers, and it is mined one unit per row (`kind` is `corpus`).
4
+ - Pick material sets or corpora from `meta/catalog.parquet` (one row per material set or corpus, about 50 KB; do not download `meta/` whole). Columns: `source`; `kind`, `material_set` or `corpus` (select one layer: units and rows are never added together); `collection`, the code of a material set's collection (A, B, C, G, GT, HW, M, MD, ML, OP, S, SC, SW; empty for a corpus); `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a corpus has one unit per row), `answers` (units with an answer file, the expected results kept apart) and `files` (files of every role); `publishers` (a material set's distinct publisher names over all its rows; empty when its rows name none); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (by the rule of the default profile of `tools/fetch.py` over `meta/bench_overlap.parquet` and `meta/bench_checked.parquet`: `none` when every data file is compared and kept and no row is removed, the hit class with the most removed rows when some are removed, `not scanned` otherwise); `license.ship`; `pii.level`; `distinct_bytes` (deduplicated size) and `bytes` (recorded size, the sum over rows); `size_check` (the rows read in `meta/sizes.parquet`, `compared` those that compared a size or hash, and how many have each status, `match` when the measured size agrees; the check's share is `match` over `compared`), `unsized_rows` (rows without a recorded size) and `unsized_estimate` (their estimated size, with its 95% range); `fetch` (whether the upstream still serves the files: `rows`, the sampled rows read in `meta/sizes.parquet`; `fetched`, those that fetched and matched their recorded size and hash; `read_at`, the day read; `level`, `all`, `some`, `none` or `not read`); the same figures for the whole index, each layer and each collection are in `meta/size_figures.json`. Who published each file of a material set, and under which license and license URL, is in `meta/attribution.parquet` (one row per source, publisher, license and license URL, with its number of files). The vocabulary is `meta/vocabulary.json`: every domain code with its name (and `zh`, its Chinese name), what it covers and its parent, every form with its Chinese name and what it means, every category of `meta/not_included.parquet` with its requirement in English and in Chinese, the two layers with their relation, and every collection with its name, what it holds and its campaigns (take a collection with `--tag campaign:<name>` for each of its campaigns).
5
  - Size of this release: Whole index: Recorded size 4.50 PB; deduplicated size 4.09 PB; size check: 99.6% match (48,396 of the 49,407 rows read compared); not compared: address unreachable 477, file gone 131, login needed 331, no size from the server 68, other error 4; estimated size of 309,522,457 of the 309,787,645 rows without a recorded size: 657 TB (an estimate; 95% range 512 TB to 1.05 PB, the sum of its sources' ranges, wider than a joint interval; from 17,054 rows read), the other 265,188 with too few rows read for an estimate. Task material: Recorded size 2.90 GB; deduplicated size 2.82 GB; size check: 99.7% match (9,111 of the 9,135 rows read compared); not compared: address unreachable 11, file gone 3, login needed 8, other error 2. Corpora: Recorded size 4.50 PB; deduplicated size 4.09 PB; size check: 99.6% match (39,285 of the 40,272 rows read compared); not compared: address unreachable 466, file gone 128, login needed 323, no size from the server 68, other error 2; estimated size of 309,522,457 of the 309,787,645 rows without a recorded size: 657 TB (an estimate; 95% range 512 TB to 1.05 PB, the sum of its sources' ranges, wider than a joint interval; from 17,054 rows read), the other 265,188 with too few rows read for an estimate. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted at every release, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it. The recorded size is the sum of `size_bytes` over the rows that have one (catalog column `bytes`). The check reads a sample of rows of every data file (`meta/sizes.parquet`): 20 of its rows with a recorded size, and of its rows without one its share of its source's sample, sized for their mean size within 25% at 95% confidence by the spread of the sizes read so far (at most 400 rows per source per round, 20 for a source with none read yet); its share is the rows that match their recorded size out of the rows it compared (status `match`, `size_mismatch` or `hash_mismatch`), and the rows it could not compare, which never reached their file, are counted beside it by status (`unreachable`, `gone`, `needs_auth`, `no_length`, `error`; catalog column `size_check`). The estimate for a source's rows without a recorded size (catalog columns `unsized_rows` and `unsized_estimate`) is their number times the mean measured size of such rows read, with a 95% BCa bootstrap range (`scipy.stats.bootstrap`, its default resample count); a source with fewer than 10 such rows read has none. Above one source, the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider than a joint interval. The figures of every level are in `meta/size_figures.json`.
6
  - Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--kind` (`material_set` or `corpus`: one layer), `--domain`, `--form`, `--tag`, `--ext`, or `--selection <file>` (a selection file the data page saves: the ids of its sources and its commit); `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts.
7
  - It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
locales/en.json CHANGED
@@ -3,7 +3,7 @@
3
  "lang.switch": "Show this page in Chinese",
4
 
5
  "page.title": "Agentic Env Index",
6
- "page.description": "Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by category, fit at a glance, one command to take it.",
7
 
8
  "nav.sections": "Page sections",
9
  "nav.overview": "Overview",
@@ -15,11 +15,12 @@
15
  "nav.releases": "Releases",
16
  "nav.releasesShort": "Releases",
17
  "nav.agents": "For agents",
 
18
  "nav.request": "Request data",
19
 
20
- "hero.eyebrow": "The Agentic Env Index, by category",
21
  "hero.title": "Real material for agent tasks",
22
- "hero.browse": "Browse categories",
23
  "hero.facts": "Facts",
24
  "hero.indexOf": "Index of",
25
  "hero.commit": "Commit {{sha}}",
@@ -38,7 +39,7 @@
38
  "url.docsSources": "https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.md",
39
 
40
  "foot.missing": "Missing material?",
41
- "foot.ask": "Ask for a category, a form or a source.",
42
  "foot.top": "Back to top",
43
  "foot.index": "Index {{sha}} · {{date}}",
44
 
@@ -126,7 +127,7 @@
126
  "permit.yes": "Allowed",
127
  "permit.no": "Not allowed",
128
  "permit.conditional": "Conditional",
129
- "permit.conditional.sameLicence": "Conditional (ship the same license)",
130
  "permit.conditional.nonCommercial": "Conditional (non-commercial or research use only)",
131
  "permit.unknown": "Not stated",
132
  "unitKind.file": "a file",
@@ -152,21 +153,23 @@
152
  "batch.collecting": "Collecting",
153
  "batch.complete": "Complete",
154
 
155
- "fit.ship": "Shippable license",
156
  "fit.bench": "No benchmark overlap",
157
  "fit.pii": "No personal data",
158
  "fit.size": "Files up to 20 MB",
159
  "fit.alone": "Unit stands alone",
160
  "fit.answers": "Answers or expected results",
161
  "fit.login": "No login",
 
162
  "fit.dot": "{{label}}: {{state}}",
163
- "fitShort.ship": "Ship",
164
  "fitShort.bench": "Bench",
165
  "fitShort.pii": "PII",
166
  "fitShort.size": "Size",
167
  "fitShort.alone": "Alone",
168
  "fitShort.answers": "Answer",
169
  "fitShort.login": "Login",
 
170
  "fitWord.yes": "Fits",
171
  "fitWord.part": "Partly",
172
  "fitWord.no": "Does not fit",
@@ -174,7 +177,7 @@
174
 
175
  "material.before": "Where the",
176
  "material.after": "units are",
177
- "material.cats": "Most requested categories",
178
  "kind.material_set": "Material sets",
179
  "kind.corpus": "Corpora",
180
  "kindOne.material_set": "Material set",
@@ -192,10 +195,10 @@
192
  "coll.unitsAll_other": "{{n}} units in all",
193
  "coll.open": "Open its material sets",
194
  "coll.showing": "Showing its material sets",
195
- "cat.shippable": "Shippable, material sets",
196
- "cat.shippableRaw": "Shippable, corpora",
197
- "cat.shipTipReady": "of the units may be shipped",
198
- "cat.shipTipRaw": "of the rows in corpora may be shipped",
199
  "cat.sample": "Take a sample",
200
 
201
  "stack.otherForms_one": "Other {{n}} form",
@@ -233,7 +236,7 @@
233
  "filter.status": "Status",
234
  "filter.status.flagged": "Flagged",
235
  "filter.training": "Training",
236
- "filter.shipping": "Shipping",
237
  "filter.permit.yes": "Allowed, all rows",
238
  "filter.permit.cond": "Allowed or conditional",
239
  "filter.tier": "License tier",
@@ -277,7 +280,7 @@
277
  "map.title": "{{metric}}: domain × form",
278
 
279
  "col.title": "Source",
280
- "col.domain": "Category",
281
  "col.form": "Form",
282
  "col.units": "Units",
283
  "col.rows": "Rows",
@@ -309,6 +312,13 @@
309
  "size.distinct": "{{size}} deduplicated size",
310
  "size.check": "Size check: {{share}} match ({{n}} rows compared)",
311
  "size.checkNone": "Size check: no row compared yet",
 
 
 
 
 
 
 
312
  "size.estimate": "estimate for the {{rows}} rows without a size: {{size}} (95% range {{low}} to {{high}}{{summed}}; from {{readings}} rows read)",
313
  "size.estimatePart": "estimate for {{rows}} of the {{all}} rows without a size: {{size}} (95% range {{low}} to {{high}}{{summed}}; from {{readings}} rows read; {{rest}} rows have too few rows read for one)",
314
  "size.estimateNone": "{{rows}} rows without a size, too few read for an estimate",
@@ -319,6 +329,7 @@
319
  "size.how": "How sizes are measured",
320
  "size.checkSentence": "Size check: {{n}} rows were read again from their source and their size and hash compared with those the index records; {{share}} of them match, that is, the file is still the one the index describes.",
321
  "size.checkNoneSentence": "Size check: no row has been read again from its source and compared with the size and hash the index records yet.",
 
322
  "size.unsizedEst": "≈ {{size}} more without a recorded size (est.)",
323
  "size.unsizedNone": "{{rows}} rows without a recorded size",
324
 
@@ -347,7 +358,7 @@
347
  "detail.anyEditor": "Any editor",
348
  "detail.licence": "License",
349
  "detail.training": "Training",
350
- "detail.shipping": "Shipping",
351
  "detail.shareOfSet": "of the files of this material set ({{licences}})",
352
  "detail.shareOfSetOnly": "of the files of this material set",
353
  "detail.shareOfCorpus": "of the rows of this corpus ({{licences}})",
@@ -364,6 +375,11 @@
364
  "detail.size": "Size",
365
  "detail.median": "median {{size}}",
366
  "detail.stripAria": "median {{median}}, 99% under {{p99}}",
 
 
 
 
 
367
  "detail.example": "Example",
368
  "detail.take": "Take it",
369
  "detail.more": "Batches and first rows",
@@ -391,6 +407,7 @@
391
  "sql.headSets": "Agentic Env Index @ {{sha}}: {{sets}} ({{units}})",
392
  "sql.headCorpora": "Agentic Env Index @ {{sha}}: {{corpora}} ({{rows}})",
393
  "sql.all": "no filter",
 
394
  "sql.chip": "{{k}}: {{v}}",
395
  "csv.progress_one": "{{done}}/{{n}} file",
396
  "csv.progress_other": "{{done}}/{{n}} files",
@@ -413,6 +430,10 @@
413
  "attr.source": "Source",
414
  "attr.licence": "License",
415
  "attr.licenceUrl": "License URL",
 
 
 
 
416
  "attr.attribution": "Attribution",
417
  "attr.notStated": "not stated",
418
  "tree.title": "What it writes",
@@ -424,7 +445,7 @@
424
  "tree.leftOut": "Left out by default",
425
  "profile.duplicates": "Duplicates",
426
  "profile.benchmark": "Benchmark copies",
427
- "profile.licence": "No shipping license",
428
  "profile.english": "Not English",
429
  "profile.personal": "Personal data",
430
  "profile.large": "Over 20 MB",
@@ -433,6 +454,10 @@
433
  "video.title": "Usage example video",
434
  "video.text": "A usage example video, in Chinese and in English with subtitles, shows the index in use from this page. It plays in the page's language; media/ of the dataset holds both videos and their subtitle files.",
435
  "video.subtitles": "English",
 
 
 
 
436
 
437
  "history.title": "What each release added",
438
  "history.start": "start",
 
3
  "lang.switch": "Show this page in Chinese",
4
 
5
  "page.title": "Agentic Env Index",
6
+ "page.description": "Real material for terminal-agent tasks from the public pointer index Battam/agentic-env-index: by domain, fit at a glance, one command to take it.",
7
 
8
  "nav.sections": "Page sections",
9
  "nav.overview": "Overview",
 
15
  "nav.releases": "Releases",
16
  "nav.releasesShort": "Releases",
17
  "nav.agents": "For agents",
18
+ "nav.manual": "Manual",
19
  "nav.request": "Request data",
20
 
21
+ "hero.eyebrow": "The Agentic Env Index, by domain",
22
  "hero.title": "Real material for agent tasks",
23
+ "hero.browse": "Browse domains",
24
  "hero.facts": "Facts",
25
  "hero.indexOf": "Index of",
26
  "hero.commit": "Commit {{sha}}",
 
39
  "url.docsSources": "https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.md",
40
 
41
  "foot.missing": "Missing material?",
42
+ "foot.ask": "Ask for a domain, a form or a source.",
43
  "foot.top": "Back to top",
44
  "foot.index": "Index {{sha}} · {{date}}",
45
 
 
127
  "permit.yes": "Allowed",
128
  "permit.no": "Not allowed",
129
  "permit.conditional": "Conditional",
130
+ "permit.conditional.sameLicence": "Conditional (redistribute under the same license)",
131
  "permit.conditional.nonCommercial": "Conditional (non-commercial or research use only)",
132
  "permit.unknown": "Not stated",
133
  "unitKind.file": "a file",
 
153
  "batch.collecting": "Collecting",
154
  "batch.complete": "Complete",
155
 
156
+ "fit.ship": "Redistributable license",
157
  "fit.bench": "No benchmark overlap",
158
  "fit.pii": "No personal data",
159
  "fit.size": "Files up to 20 MB",
160
  "fit.alone": "Unit stands alone",
161
  "fit.answers": "Answers or expected results",
162
  "fit.login": "No login",
163
+ "fit.fetch": "Upstream serves the files",
164
  "fit.dot": "{{label}}: {{state}}",
165
+ "fitShort.ship": "Redist.",
166
  "fitShort.bench": "Bench",
167
  "fitShort.pii": "PII",
168
  "fitShort.size": "Size",
169
  "fitShort.alone": "Alone",
170
  "fitShort.answers": "Answer",
171
  "fitShort.login": "Login",
172
+ "fitShort.fetch": "Fetch",
173
  "fitWord.yes": "Fits",
174
  "fitWord.part": "Partly",
175
  "fitWord.no": "Does not fit",
 
177
 
178
  "material.before": "Where the",
179
  "material.after": "units are",
180
+ "material.cats": "Most requested domains",
181
  "kind.material_set": "Material sets",
182
  "kind.corpus": "Corpora",
183
  "kindOne.material_set": "Material set",
 
195
  "coll.unitsAll_other": "{{n}} units in all",
196
  "coll.open": "Open its material sets",
197
  "coll.showing": "Showing its material sets",
198
+ "cat.shippable": "Redistributable, material sets",
199
+ "cat.shippableRaw": "Redistributable, corpora",
200
+ "cat.shipTipReady": "of the units may be redistributed",
201
+ "cat.shipTipRaw": "of the rows in corpora may be redistributed",
202
  "cat.sample": "Take a sample",
203
 
204
  "stack.otherForms_one": "Other {{n}} form",
 
236
  "filter.status": "Status",
237
  "filter.status.flagged": "Flagged",
238
  "filter.training": "Training",
239
+ "filter.shipping": "Redistribution",
240
  "filter.permit.yes": "Allowed, all rows",
241
  "filter.permit.cond": "Allowed or conditional",
242
  "filter.tier": "License tier",
 
280
  "map.title": "{{metric}}: domain × form",
281
 
282
  "col.title": "Source",
283
+ "col.domain": "Domain",
284
  "col.form": "Form",
285
  "col.units": "Units",
286
  "col.rows": "Rows",
 
312
  "size.distinct": "{{size}} deduplicated size",
313
  "size.check": "Size check: {{share}} match ({{n}} rows compared)",
314
  "size.checkNone": "Size check: no row compared yet",
315
+ "size.checkNotCompared": "Size check: none of the {{read}} rows read could be compared ({{why}})",
316
+ "size.listSep": ", ",
317
+ "size.status.unreachable": "address unreachable {{n}}",
318
+ "size.status.gone": "file gone {{n}}",
319
+ "size.status.needs_auth": "login needed {{n}}",
320
+ "size.status.no_length": "no size from the server {{n}}",
321
+ "size.status.error": "other error {{n}}",
322
  "size.estimate": "estimate for the {{rows}} rows without a size: {{size}} (95% range {{low}} to {{high}}{{summed}}; from {{readings}} rows read)",
323
  "size.estimatePart": "estimate for {{rows}} of the {{all}} rows without a size: {{size}} (95% range {{low}} to {{high}}{{summed}}; from {{readings}} rows read; {{rest}} rows have too few rows read for one)",
324
  "size.estimateNone": "{{rows}} rows without a size, too few read for an estimate",
 
329
  "size.how": "How sizes are measured",
330
  "size.checkSentence": "Size check: {{n}} rows were read again from their source and their size and hash compared with those the index records; {{share}} of them match, that is, the file is still the one the index describes.",
331
  "size.checkNoneSentence": "Size check: no row has been read again from its source and compared with the size and hash the index records yet.",
332
+ "size.checkNotComparedSentence": "Size check: {{read}} rows were read again from their source, but none could be compared with the size and hash the index records ({{why}}).",
333
  "size.unsizedEst": "≈ {{size}} more without a recorded size (est.)",
334
  "size.unsizedNone": "{{rows}} rows without a recorded size",
335
 
 
358
  "detail.anyEditor": "Any editor",
359
  "detail.licence": "License",
360
  "detail.training": "Training",
361
+ "detail.shipping": "Redistribution",
362
  "detail.shareOfSet": "of the files of this material set ({{licences}})",
363
  "detail.shareOfSetOnly": "of the files of this material set",
364
  "detail.shareOfCorpus": "of the rows of this corpus ({{licences}})",
 
375
  "detail.size": "Size",
376
  "detail.median": "median {{size}}",
377
  "detail.stripAria": "median {{median}}, 99% under {{p99}}",
378
+ "detail.fetch": "Fetch check",
379
+ "fetch.all": "All {{rows}} sampled rows read fetched and matched their recorded size and hash (read {{date}}).",
380
+ "fetch.some": "{{fetched}} of the {{rows}} sampled rows read ({{share}}) fetched and matched their recorded size and hash (read {{date}}); the others did not.",
381
+ "fetch.none": "None of the {{rows}} sampled rows read fetched and matched their recorded size and hash (read {{date}}): the upstream does not serve these files now.",
382
+ "fetch.notRead": "No sampled row read yet.",
383
  "detail.example": "Example",
384
  "detail.take": "Take it",
385
  "detail.more": "Batches and first rows",
 
407
  "sql.headSets": "Agentic Env Index @ {{sha}}: {{sets}} ({{units}})",
408
  "sql.headCorpora": "Agentic Env Index @ {{sha}}: {{corpora}} ({{rows}})",
409
  "sql.all": "no filter",
410
+ "sql.noProfile": "Rows as published: this query applies no rule of the default profile of tools/fetch.py (no deduplication; benchmark copies, rows not compared with the benchmark list, licenses that do not allow redistribution, languages other than English, personal data and files over 20 MB all stay in); the command applies them.",
411
  "sql.chip": "{{k}}: {{v}}",
412
  "csv.progress_one": "{{done}}/{{n}} file",
413
  "csv.progress_other": "{{done}}/{{n}} files",
 
430
  "attr.source": "Source",
431
  "attr.licence": "License",
432
  "attr.licenceUrl": "License URL",
433
+ "attr.files": "Files by publisher, license and license URL (each file's own; its source's where the file names none)",
434
+ "attr.fileLine_one": "{{publisher}}: {{licence}}, {{url}} ({{n}} file)",
435
+ "attr.fileLine_other": "{{publisher}}: {{licence}}, {{url}} ({{n}} files)",
436
+ "attr.noPublisher": "publisher not stated",
437
  "attr.attribution": "Attribution",
438
  "attr.notStated": "not stated",
439
  "tree.title": "What it writes",
 
445
  "tree.leftOut": "Left out by default",
446
  "profile.duplicates": "Duplicates",
447
  "profile.benchmark": "Benchmark copies",
448
+ "profile.licence": "No redistribution license",
449
  "profile.english": "Not English",
450
  "profile.personal": "Personal data",
451
  "profile.large": "Over 20 MB",
 
454
  "video.title": "Usage example video",
455
  "video.text": "A usage example video, in Chinese and in English with subtitles, shows the index in use from this page. It plays in the page's language; media/ of the dataset holds both videos and their subtitle files.",
456
  "video.subtitles": "English",
457
+ "manual.title": "User manual",
458
+ "manual.back": "Back to the page",
459
+ "manual.loading": "Loading the manual…",
460
+ "manual.missing": "The manual is not on this page yet: {{error}}",
461
 
462
  "history.title": "What each release added",
463
  "history.start": "start",
locales/zh.json CHANGED
@@ -3,7 +3,7 @@
3
  "lang.switch": "以英文显示本页",
4
 
5
  "page.title": "Agentic Env 索引",
6
- "page.description": "来自公开指针索引 Battam/agentic-env-index 的终端智能体任务真实素材:按类别浏览,一眼看出是否适配,一条命令即可获取。",
7
 
8
  "nav.sections": "页面分区",
9
  "nav.overview": "概览",
@@ -15,11 +15,12 @@
15
  "nav.releases": "发布",
16
  "nav.releasesShort": "发布",
17
  "nav.agents": "给智能体",
 
18
  "nav.request": "申请数据",
19
 
20
- "hero.eyebrow": "Agentic Env 索引,按类别",
21
  "hero.title": "面向智能体任务的真实素材",
22
- "hero.browse": "浏览类别",
23
  "hero.facts": "概况",
24
  "hero.indexOf": "索引版本",
25
  "hero.commit": "提交 {{sha}}",
@@ -38,13 +39,13 @@
38
  "url.docsSources": "https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.zh.md",
39
 
40
  "foot.missing": "缺少素材?",
41
- "foot.ask": "可以申请一个类别、一种形态或一个数据源。",
42
  "foot.top": "回到顶部",
43
  "foot.index": "索引 {{sha}} · {{date}}",
44
 
45
  "notice.vocabulary": "无法读取词表({{error}});代码按原样显示。",
46
  "notice.failed": "无法读取索引:{{error}}",
47
- "notice.linked": "联动图表和数据源列表无法启动:{{error}}",
48
  "error.noCommit": "Hub 未返回提交 ID",
49
 
50
  "num.unknown": "未知",
@@ -65,7 +66,7 @@
65
 
66
  "n.materialSet_other": "{{n}} 个材料集",
67
  "n.corpus_other": "{{n}} 个语料库",
68
- "n.picked_other": "{{n}} 个已挑选的数据源",
69
  "n.domain_other": "{{n}} 个领域",
70
  "n.unit_other": "{{n}} 个单元",
71
  "n.unitFile_other": "{{n}} 个单元文件",
@@ -113,7 +114,7 @@
113
  "permit.yes": "允许",
114
  "permit.no": "不允许",
115
  "permit.conditional": "有条件",
116
- "permit.conditional.sameLicence": "有条件(要带上同样的许可证)",
117
  "permit.conditional.nonCommercial": "有条件(仅限非商业或研究用途)",
118
  "permit.unknown": "未注明",
119
  "unitKind.file": "一个文件",
@@ -133,21 +134,23 @@
133
  "batch.collecting": "收集中",
134
  "batch.complete": "已完成",
135
 
136
- "fit.ship": "许可证允许分发",
137
  "fit.bench": "与基准无重合",
138
  "fit.pii": "无个人数据",
139
  "fit.size": "文件不超过 20 MB",
140
  "fit.alone": "单元可独立使用",
141
  "fit.answers": "有答案或预期结果",
142
  "fit.login": "无需登录",
 
143
  "fit.dot": "{{label}}:{{state}}",
144
- "fitShort.ship": "分发",
145
  "fitShort.bench": "基准",
146
  "fitShort.pii": "隐私",
147
  "fitShort.size": "大小",
148
  "fitShort.alone": "独立",
149
  "fitShort.answers": "答案",
150
  "fitShort.login": "登录",
 
151
  "fitWord.yes": "适配",
152
  "fitWord.part": "部分适配",
153
  "fitWord.no": "不适配",
@@ -155,7 +158,7 @@
155
 
156
  "material.before": "这",
157
  "material.after": "个单元的分布",
158
- "material.cats": "需求最多的类别",
159
  "kind.material_set": "材料集",
160
  "kind.corpus": "语料库",
161
  "kindOne.material_set": "材料集",
@@ -169,10 +172,10 @@
169
  "coll.unitsAll_other": "共 {{n}} 个单元",
170
  "coll.open": "打开它的材料集",
171
  "coll.showing": "正在显示它的材料集",
172
- "cat.shippable": "可分发(材料集)",
173
- "cat.shippableRaw": "可分发(语料库)",
174
- "cat.shipTipReady": "的单元可以分发",
175
- "cat.shipTipRaw": "的语料库数据行可以分发",
176
  "cat.sample": "取样",
177
 
178
  "stack.otherForms_other": "其他 {{n}} 种形态",
@@ -206,11 +209,11 @@
206
  "filter.status": "状态",
207
  "filter.status.flagged": "已标记",
208
  "filter.training": "训练",
209
- "filter.shipping": "分发",
210
  "filter.permit.yes": "允许,所有行",
211
  "filter.permit.cond": "允许或有条件",
212
  "filter.tier": "许可证等级",
213
- "filter.size": "数据源大小",
214
  "filter.size.lt1g": "小于 1 GB",
215
  "filter.size.1g1t": "1 GB 到 1 TB",
216
  "filter.size.ge1t": "1 TB 及以上",
@@ -247,8 +250,8 @@
247
  "map.head": "{{amount}},{{domains}}",
248
  "map.title": "{{metric}}:领域 × 形态",
249
 
250
- "col.title": "数据源",
251
- "col.domain": "类别",
252
  "col.form": "形态",
253
  "col.units": "单元数",
254
  "col.rows": "行数",
@@ -257,7 +260,7 @@
257
  "col.fit": "适配",
258
  "sources.headSets": "{{sets}}",
259
  "sources.headCorpora": "{{corpora}}",
260
- "sources.none": "没有匹配的数据源",
261
  "sources.unitsLabel": "单元",
262
  "sources.rowsLabel": "语料库行数",
263
  "sources.unitWord_other": " 个单元",
@@ -269,7 +272,7 @@
269
  "preview.hide": "收起行",
270
  "preview.title": "前 {{n}} 行,单元文件在前(meta/preview.parquet)",
271
  "preview.loading": "正在读取前几行…",
272
- "preview.none": "本次发布还没有这个数据源的预览行。",
273
  "preview.failed": "无法读取前几行:{{error}}",
274
  "size.note": "{{layer}}:{{line}}。",
275
  "size.index": "整个索引:",
@@ -278,16 +281,24 @@
278
  "size.distinct": "去重后大小 {{size}}",
279
  "size.check": "大小核对:{{share}} 一致(比对了 {{n}} 行)",
280
  "size.checkNone": "大小核对:还没有比对任何行",
 
 
 
 
 
 
 
281
  "size.estimate": "没有大小的 {{rows}} 行的估计值:{{size}}(95% 范围 {{low}} 至 {{high}}{{summed}};依据读过的 {{readings}} 行)",
282
  "size.estimatePart": "没有大小的 {{all}} 行中 {{rows}} 行的估计值:{{size}}(95% 范围 {{low}} 至 {{high}}{{summed}};依据读过的 {{readings}} 行;其余 {{rest}} 行读过的太少,没有估计值)",
283
  "size.estimateNone": "{{rows}} 行没有大小,读过的太少,没有估计值",
284
- "size.summed": ",为各数据源范围之和,比联合区间更宽",
285
  "size.none": "本次发布尚未发布大小数字",
286
  "size.recordedShort": "记录 {{size}}",
287
  "size.distinctShort": "(去重后 {{size}})",
288
  "size.how": "大小如何测量",
289
- "size.checkSentence": "大小核对:从来源重新读取了 {{n}} 行,将其大小和哈希与索引记录的比对;其中 {{share}} 一致,即文件仍是索引所描述的那个文件。",
290
- "size.checkNoneSentence": "大小核对:还没有任何行从来源重新读取并与索引记录的大小和哈希比对。",
 
291
  "size.unsizedEst": "另有约 {{size}} 没有记录大小(估)",
292
  "size.unsizedNone": "{{rows}} 行没有记录大小",
293
 
@@ -298,7 +309,7 @@
298
  "detail.countsByFirst": "{{what}};前几行由 {{publishers}} 发布。",
299
  "detail.oneUnit": "一个单元",
300
  "detail.files": "文件",
301
- "detail.from": "来源",
302
  "detail.publishers_other": "{{n}} 个发布方;主要网站",
303
  "detail.notRecorded": "未记录",
304
  "detail.alone": "可独立使用",
@@ -315,7 +326,7 @@
315
  "detail.anyEditor": "任意编辑器",
316
  "detail.licence": "许可证",
317
  "detail.training": "训练",
318
- "detail.shipping": "分发",
319
  "detail.shareOfSet": "(占本材料集的文件;{{licences}})",
320
  "detail.shareOfSetOnly": "(占本材料集的文件)",
321
  "detail.shareOfCorpus": "(占本语料库的行;{{licences}})",
@@ -330,6 +341,11 @@
330
  "detail.size": "大小",
331
  "detail.median": "中位数 {{size}}",
332
  "detail.stripAria": "中位数 {{median}},99% 小于 {{p99}}",
 
 
 
 
 
333
  "detail.example": "示例",
334
  "detail.take": "获取",
335
  "detail.more": "批次与前几行",
@@ -353,6 +369,7 @@
353
  "sql.headSets": "Agentic Env 索引 @ {{sha}}:{{sets}}({{units}})",
354
  "sql.headCorpora": "Agentic Env 索引 @ {{sha}}:{{corpora}}({{rows}})",
355
  "sql.all": "无筛选",
 
356
  "sql.chip": "{{k}}:{{v}}",
357
  "csv.progress_other": "{{done}}/{{n}} 个文件",
358
  "csv.button": "CSV · {{rows}}",
@@ -364,26 +381,29 @@
364
  "keep.again": "按文件再取一次",
365
  "pick.add": "挑选",
366
  "pick.picked": "已挑选",
367
- "pick.word_other": "个数据源已挑选",
368
  "pick.clear": "清除挑选",
369
- "pick.note": "这条命令只取已挑选的数据源(两层都可挑选),不受筛选影响;清除挑选后改取筛选所选的内容。",
370
- "attr.title_other": "署名:{{dataset}} 的 {{n}} 个数据源",
371
  "attr.commit": "数据集提交 {{sha}}:{{url}}",
372
- "attr.source": "数据源",
373
- "attr.licence": "许可",
374
- "attr.licenceUrl": "许可链接",
 
 
 
375
  "attr.attribution": "署名",
376
  "attr.notStated": "未注明",
377
  "tree.title": "写入内容",
378
  "tree.units": "单元、README、LICENSE",
379
  "tree.answers": "答案,单独存放",
380
  "tree.manifest": "每个文件一行,已校验 sha256",
381
- "tree.sizes": "每个数据源一行",
382
  "tree.domainNote": ":例如 D3_data 或 D3",
383
  "tree.leftOut": "默认排除",
384
  "profile.duplicates": "重复内容",
385
  "profile.benchmark": "基准副本",
386
- "profile.licence": "许可证不允许分发",
387
  "profile.english": "非英文",
388
  "profile.personal": "个人数据",
389
  "profile.large": "超过 20 MB",
@@ -392,6 +412,10 @@
392
  "video.title": "使用示例视频",
393
  "video.text": "一段使用示例视频有中文版和英文版,各带字幕,演示了如何从本页使用这份索引。这里按页面语言播放;两版视频和字幕文件都在数据集的 media/ 下。",
394
  "video.subtitles": "中文",
 
 
 
 
395
 
396
  "history.title": "每次发布新增了什么",
397
  "history.start": "起点",
 
3
  "lang.switch": "以英文显示本页",
4
 
5
  "page.title": "Agentic Env 索引",
6
+ "page.description": "来自公开指针索引 Battam/agentic-env-index 的终端智能体任务真实素材:按领域浏览,一眼看出是否适配,一条命令即可获取。",
7
 
8
  "nav.sections": "页面分区",
9
  "nav.overview": "概览",
 
15
  "nav.releases": "发布",
16
  "nav.releasesShort": "发布",
17
  "nav.agents": "给智能体",
18
+ "nav.manual": "手册",
19
  "nav.request": "申请数据",
20
 
21
+ "hero.eyebrow": "Agentic Env 索引,按领域",
22
  "hero.title": "面向智能体任务的真实素材",
23
+ "hero.browse": "浏览领域",
24
  "hero.facts": "概况",
25
  "hero.indexOf": "索引版本",
26
  "hero.commit": "提交 {{sha}}",
 
39
  "url.docsSources": "https://huggingface.co/datasets/Battam/agentic-env-index/blob/main/docs/sources.zh.md",
40
 
41
  "foot.missing": "缺少素材?",
42
+ "foot.ask": "可以申请一个领域、一种形态或一个来源。",
43
  "foot.top": "回到顶部",
44
  "foot.index": "索引 {{sha}} · {{date}}",
45
 
46
  "notice.vocabulary": "无法读取词表({{error}});代码按原样显示。",
47
  "notice.failed": "无法读取索引:{{error}}",
48
+ "notice.linked": "联动图表和来源列表无法启动:{{error}}",
49
  "error.noCommit": "Hub 未返回提交 ID",
50
 
51
  "num.unknown": "未知",
 
66
 
67
  "n.materialSet_other": "{{n}} 个材料集",
68
  "n.corpus_other": "{{n}} 个语料库",
69
+ "n.picked_other": "{{n}} 个已挑选的来源",
70
  "n.domain_other": "{{n}} 个领域",
71
  "n.unit_other": "{{n}} 个单元",
72
  "n.unitFile_other": "{{n}} 个单元文件",
 
114
  "permit.yes": "允许",
115
  "permit.no": "不允许",
116
  "permit.conditional": "有条件",
117
+ "permit.conditional.sameLicence": "有条件(以同样的许可证再分发)",
118
  "permit.conditional.nonCommercial": "有条件(仅限非商业或研究用途)",
119
  "permit.unknown": "未注明",
120
  "unitKind.file": "一个文件",
 
134
  "batch.collecting": "收集中",
135
  "batch.complete": "已完成",
136
 
137
+ "fit.ship": "许可证允许再分发",
138
  "fit.bench": "与基准无重合",
139
  "fit.pii": "无个人数据",
140
  "fit.size": "文件不超过 20 MB",
141
  "fit.alone": "单元可独立使用",
142
  "fit.answers": "有答案或预期结果",
143
  "fit.login": "无需登录",
144
+ "fit.fetch": "上游提供文件",
145
  "fit.dot": "{{label}}:{{state}}",
146
+ "fitShort.ship": "再分发",
147
  "fitShort.bench": "基准",
148
  "fitShort.pii": "隐私",
149
  "fitShort.size": "大小",
150
  "fitShort.alone": "独立",
151
  "fitShort.answers": "答案",
152
  "fitShort.login": "登录",
153
+ "fitShort.fetch": "取回",
154
  "fitWord.yes": "适配",
155
  "fitWord.part": "部分适配",
156
  "fitWord.no": "不适配",
 
158
 
159
  "material.before": "这",
160
  "material.after": "个单元的分布",
161
+ "material.cats": "需求最多的领域",
162
  "kind.material_set": "材料集",
163
  "kind.corpus": "语料库",
164
  "kindOne.material_set": "材料集",
 
172
  "coll.unitsAll_other": "共 {{n}} 个单元",
173
  "coll.open": "打开它的材料集",
174
  "coll.showing": "正在显示它的材料集",
175
+ "cat.shippable": "可再分发(材料集)",
176
+ "cat.shippableRaw": "可再分发(语料库)",
177
+ "cat.shipTipReady": "的单元可以再分发",
178
+ "cat.shipTipRaw": "的语料库数据行可以再分发",
179
  "cat.sample": "取样",
180
 
181
  "stack.otherForms_other": "其他 {{n}} 种形态",
 
209
  "filter.status": "状态",
210
  "filter.status.flagged": "已标记",
211
  "filter.training": "训练",
212
+ "filter.shipping": "再分发",
213
  "filter.permit.yes": "允许,所有行",
214
  "filter.permit.cond": "允许或有条件",
215
  "filter.tier": "许可证等级",
216
+ "filter.size": "来源大小",
217
  "filter.size.lt1g": "小于 1 GB",
218
  "filter.size.1g1t": "1 GB 到 1 TB",
219
  "filter.size.ge1t": "1 TB 及以上",
 
250
  "map.head": "{{amount}},{{domains}}",
251
  "map.title": "{{metric}}:领域 × 形态",
252
 
253
+ "col.title": "来源",
254
+ "col.domain": "领域",
255
  "col.form": "形态",
256
  "col.units": "单元数",
257
  "col.rows": "行数",
 
260
  "col.fit": "适配",
261
  "sources.headSets": "{{sets}}",
262
  "sources.headCorpora": "{{corpora}}",
263
+ "sources.none": "没有匹配的来源",
264
  "sources.unitsLabel": "单元",
265
  "sources.rowsLabel": "语料库行数",
266
  "sources.unitWord_other": " 个单元",
 
272
  "preview.hide": "收起行",
273
  "preview.title": "前 {{n}} 行,单元文件在前(meta/preview.parquet)",
274
  "preview.loading": "正在读取前几行…",
275
+ "preview.none": "本次发布还没有这个来源的预览行。",
276
  "preview.failed": "无法读取前几行:{{error}}",
277
  "size.note": "{{layer}}:{{line}}。",
278
  "size.index": "整个索引:",
 
281
  "size.distinct": "去重后大小 {{size}}",
282
  "size.check": "大小核对:{{share}} 一致(比对了 {{n}} 行)",
283
  "size.checkNone": "大小核对:还没有比对任何行",
284
+ "size.checkNotCompared": "大小核对:读过的 {{read}} 行都没能比对({{why}})",
285
+ "size.listSep": "、",
286
+ "size.status.unreachable": "地址无法访问 {{n}} 行",
287
+ "size.status.gone": "文件已不存在 {{n}} 行",
288
+ "size.status.needs_auth": "需要登录 {{n}} 行",
289
+ "size.status.no_length": "服务器未给出大小 {{n}} 行",
290
+ "size.status.error": "其他错误 {{n}} 行",
291
  "size.estimate": "没有大小的 {{rows}} 行的估计值:{{size}}(95% 范围 {{low}} 至 {{high}}{{summed}};依据读过的 {{readings}} 行)",
292
  "size.estimatePart": "没有大小的 {{all}} 行中 {{rows}} 行的估计值:{{size}}(95% 范围 {{low}} 至 {{high}}{{summed}};依据读过的 {{readings}} 行;其余 {{rest}} 行读过的太少,没有估计值)",
293
  "size.estimateNone": "{{rows}} 行没有大小,读过的太少,没有估计值",
294
+ "size.summed": ",为各来源范围之和,比联合区间更宽",
295
  "size.none": "本次发布尚未发布大小数字",
296
  "size.recordedShort": "记录 {{size}}",
297
  "size.distinctShort": "(去重后 {{size}})",
298
  "size.how": "大小如何测量",
299
+ "size.checkSentence": "大小核对:从上游重新读取了 {{n}} 行,将其大小和哈希与索引记录的比对;其中 {{share}} 一致,即文件仍是索引所描述的那个文件。",
300
+ "size.checkNoneSentence": "大小核对:还没有任何行从上游重新读取并与索引记录的大小和哈希比对。",
301
+ "size.checkNotComparedSentence": "大小核对:从上游重新读取了 {{read}} 行,但都没能与索引记录的大小和哈希比对({{why}})。",
302
  "size.unsizedEst": "另有约 {{size}} 没有记录大小(估)",
303
  "size.unsizedNone": "{{rows}} 行没有记录大小",
304
 
 
309
  "detail.countsByFirst": "{{what}};前几行由 {{publishers}} 发布。",
310
  "detail.oneUnit": "一个单元",
311
  "detail.files": "文件",
312
+ "detail.from": "出自",
313
  "detail.publishers_other": "{{n}} 个发布方;主要网站",
314
  "detail.notRecorded": "未记录",
315
  "detail.alone": "可独立使用",
 
326
  "detail.anyEditor": "任意编辑器",
327
  "detail.licence": "许可证",
328
  "detail.training": "训练",
329
+ "detail.shipping": "再分发",
330
  "detail.shareOfSet": "(占本材料集的文件;{{licences}})",
331
  "detail.shareOfSetOnly": "(占本材料集的文件)",
332
  "detail.shareOfCorpus": "(占本语料库的行;{{licences}})",
 
341
  "detail.size": "大小",
342
  "detail.median": "中位数 {{size}}",
343
  "detail.stripAria": "中位数 {{median}},99% 小于 {{p99}}",
344
+ "detail.fetch": "取回核对",
345
+ "fetch.all": "读过的 {{rows}} 个抽样行全部取到,大小和哈希与记录一致(读取于 {{date}})。",
346
+ "fetch.some": "读过的 {{rows}} 个抽样行中有 {{fetched}} 个({{share}})取到,大小和哈希与记录一致(读取于 {{date}});其余没有。",
347
+ "fetch.none": "读过的 {{rows}} 个抽样行没有一个取到大小和哈希与记录一致的文件(读取于 {{date}}):上游现在不提供这些文件。",
348
+ "fetch.notRead": "还没有读过抽样行。",
349
  "detail.example": "示例",
350
  "detail.take": "获取",
351
  "detail.more": "批次与前几行",
 
369
  "sql.headSets": "Agentic Env 索引 @ {{sha}}:{{sets}}({{units}})",
370
  "sql.headCorpora": "Agentic Env 索引 @ {{sha}}:{{corpora}}({{rows}})",
371
  "sql.all": "无筛选",
372
+ "sql.noProfile": "按发布原样读取各行:这条查询不应用 tools/fetch.py 默认配置的任何规则(不去重;基准副本、没有与基准测试清单比对过的行、不允许再分发的许可证、非英文、个人数据和超过 20 MB 的文件都保留);命令会应用这些规则。",
373
  "sql.chip": "{{k}}:{{v}}",
374
  "csv.progress_other": "{{done}}/{{n}} 个文件",
375
  "csv.button": "CSV · {{rows}}",
 
381
  "keep.again": "按文件再取一次",
382
  "pick.add": "挑选",
383
  "pick.picked": "已挑选",
384
+ "pick.word_other": "个来源已挑选",
385
  "pick.clear": "清除挑选",
386
+ "pick.note": "这条命令只取已挑选的来源(两层都可挑选),不受筛选影响;清除挑选后改取筛选所选的内容。",
387
+ "attr.title_other": "署名:{{dataset}} 的 {{n}} 个来源",
388
  "attr.commit": "数据集提交 {{sha}}:{{url}}",
389
+ "attr.source": "来源",
390
+ "attr.licence": "许可证",
391
+ "attr.licenceUrl": "许可证链接",
392
+ "attr.files": "按发布方、许可证和许可证链接分的文件(各文件自己的;文件没有写明时取其来源的)",
393
+ "attr.fileLine_other": "{{publisher}}:{{licence}},{{url}}({{n}} 个文件)",
394
+ "attr.noPublisher": "未注明发布方",
395
  "attr.attribution": "署名",
396
  "attr.notStated": "未注明",
397
  "tree.title": "写入内容",
398
  "tree.units": "单元、README、LICENSE",
399
  "tree.answers": "答案,单独存放",
400
  "tree.manifest": "每个文件一行,已校验 sha256",
401
+ "tree.sizes": "每个来源一行",
402
  "tree.domainNote": ":例如 D3_data 或 D3",
403
  "tree.leftOut": "默认排除",
404
  "profile.duplicates": "重复内容",
405
  "profile.benchmark": "基准副本",
406
+ "profile.licence": "许可证不允许再分发",
407
  "profile.english": "非英文",
408
  "profile.personal": "个人数据",
409
  "profile.large": "超过 20 MB",
 
412
  "video.title": "使用示例视频",
413
  "video.text": "一段使用示例视频有中文版和英文版,各带字幕,演示了如何从本页使用这份索引。这里按页面语言播放;两版视频和字幕文件都在数据集的 media/ 下。",
414
  "video.subtitles": "中文",
415
+ "manual.title": "使用手册",
416
+ "manual.back": "返回页面",
417
+ "manual.loading": "正在加载手册…",
418
+ "manual.missing": "本页还没有手册:{{error}}",
419
 
420
  "history.title": "每次发布新增了什么",
421
  "history.start": "起点",
manual.js ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // #1617: the user manual (manual/manual.en.md and manual/manual.zh.md of this folder) in the page's language, as the
2
+ // view #manual shows it (app.js showView). Its numbers and command outputs were filled at deploy (03_data_line/steps/portal.py).
3
+ // An image names its shot's output in manual/shots.yml and is read from manual/shots/, where the screenshots taken on
4
+ // this page after each deploy are (03_data_line/steps/shots.py); a shot with no image there shows its name in its place.
5
+ // Standard tool: Markdown rendered to HTML by markdown-it (CommonMark).
6
+ import MarkdownIt from 'https://cdn.jsdelivr.net/npm/markdown-it@15.0.3/+esm';
7
+ import { clear, el } from './charts.js';
8
+ import { lang, onLanguage, t } from './i18n.js';
9
+
10
+ const SHOTS = 'manual/shots/';
11
+ const md = new MarkdownIt('commonmark');
12
+ const image = md.renderer.rules.image;
13
+ md.renderer.rules.image = (tokens, i, options, env, self) => {
14
+ const shot = tokens[i].attrGet('src');
15
+ tokens[i].attrSet('src', SHOTS + shot);
16
+ tokens[i].attrSet('data-shot', shot);
17
+ return image(tokens, i, options, env, self);
18
+ };
19
+
20
+ // per language, the text once read
21
+ const texts = new Map();
22
+ // the view's article, once drawn: the language switch draws it again while it is shown
23
+ let target = null;
24
+
25
+ function read(l) {
26
+ if (!texts.has(l)) {
27
+ const p = fetch(new URL(`manual/manual.${l}.md`, document.baseURI)).then((r) => {
28
+ if (!r.ok) throw new Error(`manual/manual.${l}.md: HTTP ${r.status}`);
29
+ return r.text();
30
+ });
31
+ p.catch(() => texts.delete(l));
32
+ texts.set(l, p);
33
+ }
34
+ return texts.get(l);
35
+ }
36
+
37
+ /** The manual of the page's language into node (the view's article). */
38
+ export async function drawManual(node) {
39
+ target = node;
40
+ const l = lang();
41
+ if (!node.querySelector('.manual-text')) clear(node).append(el('p', { class: 'small' }, t('manual.loading')));
42
+ let text;
43
+ try {
44
+ text = await read(l);
45
+ } catch (e) {
46
+ clear(node).append(el('p', { class: 'small problem-text' }, t('manual.missing', { error: e.message })));
47
+ return;
48
+ }
49
+ if (l !== lang()) return;
50
+ const body = el('div', { class: 'manual-text', lang: l === 'zh' ? 'zh-CN' : 'en' });
51
+ // the manual is this folder's own Markdown, rendered by markdown-it; nothing read from the dataset goes in
52
+ body.innerHTML = md.render(text);
53
+ for (const img of body.querySelectorAll('img[data-shot]')) {
54
+ img.addEventListener('error', () => img.replaceWith(el('span', { class: 'shot-missing' }, img.dataset.shot)), { once: true });
55
+ }
56
+ clear(node).append(body);
57
+ }
58
+
59
+ onLanguage(() => { if (target && !target.closest('[hidden]')) drawManual(target); });
manual/manual.en.md ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # User manual
2
+
3
+ This manual is for people who build data from the public index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index).
4
+ This is a placeholder;
5
+ its texts come later, in the four parts of [Diátaxis](https://diataxis.fr/): a tutorial, how-to guides, a reference and an explanation.
6
+ The numbers below and the output of the command were filled in when this page was deployed, from the index commit `f4b175ed19709225382f0ae5a13fd4c18530b94b`.
7
+
8
+ ## Tutorial
9
+
10
+ The index holds 1,799 material sets in 13 collections, with 4,183 units, 787 of them with an answer file.
11
+ Beside them it holds 133 corpora with 1,227,714,425 rows.
12
+
13
+ ![The first screen of the page](first-screen-en-1920.png)
14
+
15
+ ## How-to guides
16
+
17
+ The index's command `tools/fetch.py` takes a selection of the index into a folder;
18
+ these are its options:
19
+
20
+ ```console
21
+ $ uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/f4b175ed19709225382f0ae5a13fd4c18530b94b/tools/fetch.py --help
22
+ usage: fetchHygMeO.py [-h] [--repo REPO] [--data DATA] [--revision REVISION]
23
+ [--tag TAG] [--domain DOMAIN] [--form FORM]
24
+ [--kind KIND] [--ext EXT] [--source SOURCE]
25
+ [--selection FILE] [--tier TIER] [--min-size MIN_SIZE]
26
+ [--max-size MAX_SIZE]
27
+ [--max-size-filter MAX_SIZE_FILTER] [--where WHERE]
28
+ [--limit LIMIT] [--sample SAMPLE] [--all]
29
+ [--keep-duplicates] [--allow-benchmark]
30
+ [--allow-not-compared] [--any-license] [--any-language]
31
+ [--allow-personal-data] [--any-size] [--out OUT]
32
+ [--dry-run] [--local-root LOCAL_ROOT]
33
+ [--staging STAGING] [--print-sql]
34
+ [--memory-limit MEMORY_LIMIT] [--threads THREADS]
35
+ [--temp-dir TEMP_DIR] [--decompress] [--check-content]
36
+ [--proxy PROXY] [--contact CONTACT] [--quiet] [--json]
37
+ [--timeout TIMEOUT]
38
+
39
+ Select rows of the Agentic Env Index, download the files they point to and check them against their rows.
40
+
41
+ options:
42
+ -h, --help show this help message and exit
43
+ --repo REPO dataset repository (default Battam/agentic-env-index)
44
+ --data DATA read the tables from this local copy or URL prefix instead of the repository
45
+ --revision REVISION read the repository at this commit (40 hex characters) or tag (pinned to the commit the
46
+ tag points to); default: the release this script was published in (release-2026-11-05.3),
47
+ else the commit the main branch points to when the script starts.
48
+ Record it to repeat a selection later. With a local --data copy it only records the
49
+ commit that copy holds (not checked)
50
+ --tag TAG collection tag campaign:<name> (repeatable; see meta/tags.parquet)
51
+ --domain DOMAIN domain code from meta/catalog.parquet (repeatable, any of them): D1 to D10 or a leaf
52
+ such as D1.6; a top-level code also matches its leaves (D3 matches D3.5), a leaf only
53
+ itself; multi and unclassified match only themselves. Codes: catalog/domains.yaml of
54
+ the pipeline repository, or the README of the index
55
+ --form FORM form id from meta/catalog.parquet, such as test-suites (repeatable, any of them);
56
+ with --domain a source must match both
57
+ --kind KIND layer from the column kind of meta/catalog.parquet: material_set or corpus (repeatable,
58
+ any of them); with --domain or --form a source must match those too
59
+ --ext EXT file extension without the dot (repeatable, any of them): takes the units with a unit
60
+ file of that extension, each with its answers and its source's companion files
61
+ --source SOURCE source name (repeatable)
62
+ --selection FILE a selection file saved by the data page (JSON: sources, revision, filters): its sources
63
+ are read as --source and its revision as --revision
64
+ --tier TIER training license tiers, comma separated (default: all tiers)
65
+ --min-size MIN_SIZE smallest size in bytes
66
+ --max-size MAX_SIZE size cap in bytes used when fetching, for every row (replaces the overall cap); it
67
+ does not remove rows from the selection
68
+ --max-size-filter MAX_SIZE_FILTER
69
+ leave out the rows whose published size is larger than this many bytes (rows of
70
+ unknown size stay; --max-size or the overall cap still bound what is downloaded)
71
+ --where WHERE extra SQL condition on the row table p (and the batch table b)
72
+ --limit LIMIT at most this many units over the whole selection (a fixed order)
73
+ --sample SAMPLE this many units per selected source, the same ones every run (a fixed order); with
74
+ their answers and the source's companion files
75
+ --all turn the default profile off: apply only the conditions given here
76
+ --keep-duplicates profile: keep duplicate copies
77
+ --allow-benchmark profile: keep rows that matched a benchmark (copies of benchmark data, collections
78
+ awaiting a ruling) and rows not compared with the benchmark list yet
79
+ --allow-not-compared profile: keep rows whose data file was not compared with the benchmark list yet (rows
80
+ that matched a benchmark stay out)
81
+ --any-license, --any-licence
82
+ profile: keep rows whose license does not let the files be shipped
83
+ --any-language profile: keep rows recorded in a language other than English
84
+ --allow-personal-data
85
+ profile: keep rows of sources or rows marked as carrying personal data
86
+ --any-size profile: keep rows above 20,000,000 bytes (the overall cap still bounds the
87
+ download)
88
+ --out OUT directory for the files and _meta/ (default ./fetched)
89
+ --dry-run print the counts and one line per source of what would be fetched (units, answers,
90
+ companion files, bytes) and a total; write _meta/fetch_manifest.jsonl with one line per
91
+ file of the selection (status selected, removed or not_taken), download nothing
92
+ --local-root LOCAL_ROOT
93
+ reuse a file under this directory whose size and hashes match a row instead of downloading
94
+ it (repeatable; default: the ARTPOOL_LOCAL_ROOTS variable, directories separated by
95
+ ':'); each manifest line records obtained: reused, staged or fetched
96
+ --staging STAGING directory where an outside fetch placed files; checked like a local root, recorded as
97
+ staged (default: the ARTPOOL_STAGING variable)
98
+ --print-sql print the query and exit
99
+ --memory-limit MEMORY_LIMIT
100
+ DuckDB memory limit, for example 8GB (default: the smaller of 4GB and a quarter of the
101
+ memory available, the cgroup limit when lower); what does not fit is written to the
102
+ spill directory
103
+ --threads THREADS DuckDB threads (default: one per GB of the memory limit, at most the number of CPUs)
104
+ --temp-dir TEMP_DIR DuckDB spill directory (default: .fetch_spill under --out, removed at the end)
105
+ --decompress after a verified download, decompress .gz, .bz2 and .zst files (within the row's cap)
106
+ and delete the compressed file; .zst needs pip install zstandard
107
+ --check-content after a verified download, flag scanned PDFs, error pages saved as files and extensions
108
+ that do not match the content (manifest field content_check); needs the file command
109
+ and poppler-utils (pdffonts, pdftotext), a missing tool skips its check
110
+ --proxy PROXY proxy for every request: http://host:port, or socks5h://host:port
111
+ (a proxy login goes before the host as user:password followed by an at sign).
112
+ (DuckDB and the byte-range reads of archive members cannot use a socks proxy,
113
+ see --data).
114
+ Default: HTTPS_PROXY, HTTP_PROXY or ALL_PROXY from the environment
115
+ --contact CONTACT your e-mail or URL, sent in the User-Agent so a site can reach you instead of blocking
116
+ --quiet no progress lines and no per-file or per-source lines on stderr
117
+ --json end with the summary as one JSON object on stdout (the default when stdout is not a
118
+ terminal); on a terminal it is otherwise a few lines in words
119
+ --timeout TIMEOUT seconds a download may wait to connect, or for its next bytes, before that address
120
+ fails (default 120); it does not bound a whole download. The reads of the index tables
121
+ have their own: 120 seconds a request, retried for up to 510 seconds
122
+
123
+ environment:
124
+ HF_TOKEN a Hugging Face access token; the index tables are read with it (a DuckDB huggingface
125
+ secret), so Hugging Face allows more requests per 5 minutes than to an anonymous address,
126
+ which runs into HTTP 429 sooner (each read then waits and retries). It overrides a token
127
+ stored on the machine. Without it the run says once that it reads anonymously. To set it,
128
+ create a read token at https://huggingface.co/settings/tokens, then
129
+ export HF_TOKEN=hf_... (PowerShell: $env:HF_TOKEN = 'hf_...')
130
+ ```
131
+
132
+ ## Reference
133
+
134
+ The first screen on a phone:
135
+
136
+ ![The first screen on a phone](first-screen-en-390.png)
137
+
138
+ ## Explanation
139
+
140
+ The whole page:
141
+
142
+ ![The whole page](page-en-1920.png)
manual/manual.zh.md ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 使用手册
2
+
3
+ 本手册写给从公开索引 [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index) 构建数据的人。
4
+ 这是占位版本;
5
+ 正文稍后补上,按 [Diátaxis](https://diataxis.fr/) 分为四部分:教程、操作指南、参考和解释。
6
+ 下面的数字和命令输出是本页部署时填入的,取自索引提交 `f4b175ed19709225382f0ae5a13fd4c18530b94b`。
7
+
8
+ ## 教程
9
+
10
+ 索引有 13 个合集中的 1,799 个材料集,共 4,183 个单元,其中 787 个带答案文件。
11
+ 此外还有 133 个语料库,共 1,227,714,425 行。
12
+
13
+ ![本页首屏](first-screen-zh-1920.png)
14
+
15
+ ## 操作指南
16
+
17
+ 索引的命令 `tools/fetch.py` 把索引中的一份选择取到一个文件夹;
18
+ 它的选项如下:
19
+
20
+ ```console
21
+ $ uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/f4b175ed19709225382f0ae5a13fd4c18530b94b/tools/fetch.py --help
22
+ usage: fetchtXYHj5.py [-h] [--repo REPO] [--data DATA] [--revision REVISION]
23
+ [--tag TAG] [--domain DOMAIN] [--form FORM]
24
+ [--kind KIND] [--ext EXT] [--source SOURCE]
25
+ [--selection FILE] [--tier TIER] [--min-size MIN_SIZE]
26
+ [--max-size MAX_SIZE]
27
+ [--max-size-filter MAX_SIZE_FILTER] [--where WHERE]
28
+ [--limit LIMIT] [--sample SAMPLE] [--all]
29
+ [--keep-duplicates] [--allow-benchmark]
30
+ [--allow-not-compared] [--any-license] [--any-language]
31
+ [--allow-personal-data] [--any-size] [--out OUT]
32
+ [--dry-run] [--local-root LOCAL_ROOT]
33
+ [--staging STAGING] [--print-sql]
34
+ [--memory-limit MEMORY_LIMIT] [--threads THREADS]
35
+ [--temp-dir TEMP_DIR] [--decompress] [--check-content]
36
+ [--proxy PROXY] [--contact CONTACT] [--quiet] [--json]
37
+ [--timeout TIMEOUT]
38
+
39
+ Select rows of the Agentic Env Index, download the files they point to and check them against their rows.
40
+
41
+ options:
42
+ -h, --help show this help message and exit
43
+ --repo REPO dataset repository (default Battam/agentic-env-index)
44
+ --data DATA read the tables from this local copy or URL prefix instead of the repository
45
+ --revision REVISION read the repository at this commit (40 hex characters) or tag (pinned to the commit the
46
+ tag points to); default: the release this script was published in (release-2026-11-05.3),
47
+ else the commit the main branch points to when the script starts.
48
+ Record it to repeat a selection later. With a local --data copy it only records the
49
+ commit that copy holds (not checked)
50
+ --tag TAG collection tag campaign:<name> (repeatable; see meta/tags.parquet)
51
+ --domain DOMAIN domain code from meta/catalog.parquet (repeatable, any of them): D1 to D10 or a leaf
52
+ such as D1.6; a top-level code also matches its leaves (D3 matches D3.5), a leaf only
53
+ itself; multi and unclassified match only themselves. Codes: catalog/domains.yaml of
54
+ the pipeline repository, or the README of the index
55
+ --form FORM form id from meta/catalog.parquet, such as test-suites (repeatable, any of them);
56
+ with --domain a source must match both
57
+ --kind KIND layer from the column kind of meta/catalog.parquet: material_set or corpus (repeatable,
58
+ any of them); with --domain or --form a source must match those too
59
+ --ext EXT file extension without the dot (repeatable, any of them): takes the units with a unit
60
+ file of that extension, each with its answers and its source's companion files
61
+ --source SOURCE source name (repeatable)
62
+ --selection FILE a selection file saved by the data page (JSON: sources, revision, filters): its sources
63
+ are read as --source and its revision as --revision
64
+ --tier TIER training license tiers, comma separated (default: all tiers)
65
+ --min-size MIN_SIZE smallest size in bytes
66
+ --max-size MAX_SIZE size cap in bytes used when fetching, for every row (replaces the overall cap); it
67
+ does not remove rows from the selection
68
+ --max-size-filter MAX_SIZE_FILTER
69
+ leave out the rows whose published size is larger than this many bytes (rows of
70
+ unknown size stay; --max-size or the overall cap still bound what is downloaded)
71
+ --where WHERE extra SQL condition on the row table p (and the batch table b)
72
+ --limit LIMIT at most this many units over the whole selection (a fixed order)
73
+ --sample SAMPLE this many units per selected source, the same ones every run (a fixed order); with
74
+ their answers and the source's companion files
75
+ --all turn the default profile off: apply only the conditions given here
76
+ --keep-duplicates profile: keep duplicate copies
77
+ --allow-benchmark profile: keep rows that matched a benchmark (copies of benchmark data, collections
78
+ awaiting a ruling) and rows not compared with the benchmark list yet
79
+ --allow-not-compared profile: keep rows whose data file was not compared with the benchmark list yet (rows
80
+ that matched a benchmark stay out)
81
+ --any-license, --any-licence
82
+ profile: keep rows whose license does not let the files be shipped
83
+ --any-language profile: keep rows recorded in a language other than English
84
+ --allow-personal-data
85
+ profile: keep rows of sources or rows marked as carrying personal data
86
+ --any-size profile: keep rows above 20,000,000 bytes (the overall cap still bounds the
87
+ download)
88
+ --out OUT directory for the files and _meta/ (default ./fetched)
89
+ --dry-run print the counts and one line per source of what would be fetched (units, answers,
90
+ companion files, bytes) and a total; write _meta/fetch_manifest.jsonl with one line per
91
+ file of the selection (status selected, removed or not_taken), download nothing
92
+ --local-root LOCAL_ROOT
93
+ reuse a file under this directory whose size and hashes match a row instead of downloading
94
+ it (repeatable; default: the ARTPOOL_LOCAL_ROOTS variable, directories separated by
95
+ ':'); each manifest line records obtained: reused, staged or fetched
96
+ --staging STAGING directory where an outside fetch placed files; checked like a local root, recorded as
97
+ staged (default: the ARTPOOL_STAGING variable)
98
+ --print-sql print the query and exit
99
+ --memory-limit MEMORY_LIMIT
100
+ DuckDB memory limit, for example 8GB (default: the smaller of 4GB and a quarter of the
101
+ memory available, the cgroup limit when lower); what does not fit is written to the
102
+ spill directory
103
+ --threads THREADS DuckDB threads (default: one per GB of the memory limit, at most the number of CPUs)
104
+ --temp-dir TEMP_DIR DuckDB spill directory (default: .fetch_spill under --out, removed at the end)
105
+ --decompress after a verified download, decompress .gz, .bz2 and .zst files (within the row's cap)
106
+ and delete the compressed file; .zst needs pip install zstandard
107
+ --check-content after a verified download, flag scanned PDFs, error pages saved as files and extensions
108
+ that do not match the content (manifest field content_check); needs the file command
109
+ and poppler-utils (pdffonts, pdftotext), a missing tool skips its check
110
+ --proxy PROXY proxy for every request: http://host:port, or socks5h://host:port
111
+ (a proxy login goes before the host as user:password followed by an at sign).
112
+ (DuckDB and the byte-range reads of archive members cannot use a socks proxy,
113
+ see --data).
114
+ Default: HTTPS_PROXY, HTTP_PROXY or ALL_PROXY from the environment
115
+ --contact CONTACT your e-mail or URL, sent in the User-Agent so a site can reach you instead of blocking
116
+ --quiet no progress lines and no per-file or per-source lines on stderr
117
+ --json end with the summary as one JSON object on stdout (the default when stdout is not a
118
+ terminal); on a terminal it is otherwise a few lines in words
119
+ --timeout TIMEOUT seconds a download may wait to connect, or for its next bytes, before that address
120
+ fails (default 120); it does not bound a whole download. The reads of the index tables
121
+ have their own: 120 seconds a request, retried for up to 510 seconds
122
+
123
+ environment:
124
+ HF_TOKEN a Hugging Face access token; the index tables are read with it (a DuckDB huggingface
125
+ secret), so Hugging Face allows more requests per 5 minutes than to an anonymous address,
126
+ which runs into HTTP 429 sooner (each read then waits and retries). It overrides a token
127
+ stored on the machine. Without it the run says once that it reads anonymously. To set it,
128
+ create a read token at https://huggingface.co/settings/tokens, then
129
+ export HF_TOKEN=hf_... (PowerShell: $env:HF_TOKEN = 'hf_...')
130
+ ```
131
+
132
+ ## 参考
133
+
134
+ 手机上的首屏:
135
+
136
+ ![手机上的首屏](first-screen-zh-390.png)
137
+
138
+ ## 解释
139
+
140
+ 整页:
141
+
142
+ ![整页](page-zh-1920.png)
manual/shots.yml ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # #1617: the screenshots of the user manual, in shot-scraper's multi format
2
+ # (https://shot-scraper.datasette.io/en/stable/multi.html). 03_data_line/steps/shots.py takes every shot on the live
3
+ # Space after each deploy, each url resolved against the live page, and writes each image to the Space as
4
+ # manual/shots/<output>; the manual's Markdown (manual.en.md, manual.zh.md) references an image by its output.
5
+ # url: relative to the live page; ?lng=en or ?lng=zh sets the page's language
6
+ # width: 1920 or 390, the two widths the page is checked at
7
+ # selector: the element to shoot; without one, the whole page
8
+ # javascript: run once the page has loaded, before the wait; before a full-page shot it shows the scroll-reveal
9
+ # elements (style.css starts .js-reveal .reveal at opacity 0 until it scrolls into view, and a full-page shot
10
+ # does not scroll)
11
+ # wait_for: a JavaScript expression, true once the shot's element is drawn; the page keeps loading modules from the
12
+ # CDN long after it is usable, so no shot waits for the network to be idle
13
+ - output: first-screen-en-1920.png
14
+ url: "?lng=en"
15
+ width: 1920
16
+ selector: "#top"
17
+ wait_for: "document.body.classList.contains('ready') && !document.querySelector('#top .sk')"
18
+ - output: first-screen-en-390.png
19
+ url: "?lng=en"
20
+ width: 390
21
+ selector: "#top"
22
+ wait_for: "document.body.classList.contains('ready') && !document.querySelector('#top .sk')"
23
+ - output: page-en-1920.png
24
+ url: "?lng=en"
25
+ width: 1920
26
+ javascript: "document.querySelectorAll('.reveal').forEach((e) => e.classList.add('in'))"
27
+ wait_for: "document.querySelector('#sources-body table') !== null"
28
+ - output: first-screen-zh-1920.png
29
+ url: "?lng=zh"
30
+ width: 1920
31
+ selector: "#top"
32
+ wait_for: "document.body.classList.contains('ready') && !document.querySelector('#top .sk')"
33
+ - output: first-screen-zh-390.png
34
+ url: "?lng=zh"
35
+ width: 390
36
+ selector: "#top"
37
+ wait_for: "document.body.classList.contains('ready') && !document.querySelector('#top .sk')"
38
+ - output: page-zh-1920.png
39
+ url: "?lng=zh"
40
+ width: 1920
41
+ javascript: "document.querySelectorAll('.reveal').forEach((e) => e.classList.add('in'))"
42
+ wait_for: "document.querySelector('#sources-body table') !== null"
portal.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
- "commit": "7c0d9d0aa627a10142b3cf47f34caadabcc2ed11",
3
- "digest": "8ca4ab7942da0beac737cfed53b73b945b14ab0efef64671b303dc4ea3ed23e0",
4
- "files": 14,
5
  "format": "portal-deploy/1",
6
  "index": "f4b175ed19709225382f0ae5a13fd4c18530b94b"
7
  }
 
1
  {
2
+ "commit": "ae5043d3f1334f1876bec9b2bd9ea384279deb75",
3
+ "digest": "5defae89560cdd5321dbce69717fc621ee175790ca7b87b2fa198a863ea3b875",
4
+ "files": 18,
5
  "format": "portal-deploy/1",
6
  "index": "f4b175ed19709225382f0ae5a13fd4c18530b94b"
7
  }
style.css CHANGED
@@ -410,8 +410,9 @@ button.bar-row:hover, button.bar-row:focus-visible { background: var(--ours-wash
410
  .fit-part { background: linear-gradient(90deg, var(--ours) 50%, #fff 50%); box-shadow: inset 0 0 0 1.5px var(--ours); }
411
  .fit-no { background: var(--problem); }
412
  .fit-unknown { background: #fff; box-shadow: inset 0 0 0 1.5px var(--other); }
413
- /* #1302: the dots and their names over the column share one grid of seven slots, so each name stands above its dot */
414
- .fit-strip, .fit-heads { display: inline-grid; grid-template-columns: repeat(7, 2.25rem); justify-items: center; align-items: center; }
 
415
  .fit-head { display: flex; flex-direction: column; align-items: center; gap: 2px; padding: 0 0 var(--s1); font-size: 10px; font-weight: 500; line-height: 1.1; color: var(--ink-2); text-align: center; white-space: nowrap; }
416
  .src-table { border-collapse: collapse; width: 100%; font-size: 15px; }
417
  .src-table th { font-size: var(--fs-small); font-weight: 600; color: var(--ink-2); text-align: left; padding: 0; border-bottom: 1px solid var(--line); }
@@ -584,6 +585,26 @@ details.long[open] > summary { margin-bottom: var(--s1); }
584
  .tip .k { display: block; color: var(--ink-2); }
585
  .tip .k::before { content: ""; display: inline-block; width: 10px; height: 2px; background: var(--ours); vertical-align: 3px; margin-right: 6px; }
586
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
587
  /* view transitions: brief, and none without motion */
588
  ::view-transition-old(root), ::view-transition-new(root) { animation-duration: 180ms; }
589
  @media (prefers-reduced-motion: reduce) {
 
410
  .fit-part { background: linear-gradient(90deg, var(--ours) 50%, #fff 50%); box-shadow: inset 0 0 0 1.5px var(--ours); }
411
  .fit-no { background: var(--problem); }
412
  .fit-unknown { background: #fff; box-shadow: inset 0 0 0 1.5px var(--other); }
413
+ /* #1302: the dots and their names over the column share one grid of a slot per fit (#1577: eight), so each name stands
414
+ above its dot */
415
+ .fit-strip, .fit-heads { display: inline-grid; grid-template-columns: repeat(8, 2.25rem); justify-items: center; align-items: center; }
416
  .fit-head { display: flex; flex-direction: column; align-items: center; gap: 2px; padding: 0 0 var(--s1); font-size: 10px; font-weight: 500; line-height: 1.1; color: var(--ink-2); text-align: center; white-space: nowrap; }
417
  .src-table { border-collapse: collapse; width: 100%; font-size: 15px; }
418
  .src-table th { font-size: var(--fs-small); font-weight: 600; color: var(--ink-2); text-align: left; padding: 0; border-bottom: 1px solid var(--line); }
 
585
  .tip .k { display: block; color: var(--ink-2); }
586
  .tip .k::before { content: ""; display: inline-block; width: 10px; height: 2px; background: var(--ours); vertical-align: 3px; margin-right: 6px; }
587
 
588
+ /* ---- #1617: the user manual, a view of its own (#manual): Markdown rendered by markdown-it in one card ---- */
589
+ .manual-on .progress { visibility: hidden; }
590
+ .manual-view { padding-block: var(--s5) var(--s10); min-height: calc(100vh - var(--nav-h)); }
591
+ .manual-wrap { max-width: calc(960px + 2 * var(--pad)); }
592
+ .manual-back { margin: 0 0 var(--s2); font-size: var(--fs-small); }
593
+ .manual-body { padding: var(--s4) var(--s5); line-height: 1.65; overflow-wrap: anywhere; }
594
+ .manual-body > :first-child, .manual-text > :first-child { margin-top: 0; }
595
+ .manual-body h1 { font-size: var(--fs-h1); line-height: 1.2; letter-spacing: -0.01em; }
596
+ .manual-body h2 { font-size: var(--fs-sub); margin-top: var(--s5); padding-top: var(--s3); border-top: 1px solid var(--line); }
597
+ .manual-body img { display: block; max-width: 100%; height: auto; margin: var(--s2) 0; border: 1px solid var(--line); border-radius: 10px; box-shadow: var(--shadow); }
598
+ .manual-body pre { background: var(--code); color: #dfe8ee; border-radius: 12px; padding: 16px 20px; font-size: 13.5px; line-height: 1.6;
599
+ overflow: auto; max-height: 28rem; }
600
+ .manual-body :not(pre) > code { background: var(--ours-wash); border-radius: 6px; padding: 1px 6px; font-size: 0.9em; }
601
+ .manual-body table { border-collapse: collapse; }
602
+ .manual-body th, .manual-body td { border: 1px solid var(--line); padding: 6px 10px; text-align: left; }
603
+ /* a shot with no image on the Space yet: its name in its place */
604
+ .shot-missing { display: block; margin: var(--s2) 0; padding: var(--s3); border: 1px dashed var(--other); border-radius: 10px;
605
+ font-family: var(--mono); font-size: var(--fs-small); color: var(--ink-muted); text-align: center; }
606
+ @media (max-width: 720px) { .manual-body { padding: var(--s3) var(--s2); } }
607
+
608
  /* view transitions: brief, and none without motion */
609
  ::view-transition-old(root), ::view-transition-new(root) { animation-duration: 180ms; }
610
  @media (prefers-reduced-motion: reduce) {