Spaces:
Running
Running
data-line commited on
Commit ยท
2ee2aa2
1
Parent(s): d20fe4d
portal: catalog page of repository commit a015777686bf
Browse filesFiles of catalog/portal/ (19), digest 63a115cc1a819d10fc6a572ea53d2b8ee4aa63e824ef957d062826b8892bf2ff.
- README.md +13 -3
- app.js +43 -1
- data.js +8 -1
- first.json +1 -1
- funnel.json +116 -0
- index.html +9 -0
- llms.txt +1 -1
- locales/en.json +12 -0
- locales/zh.json +12 -0
- manual/manual.en.md +33 -31
- manual/manual.zh.md +33 -31
- portal.json +4 -3
- stages.json +1 -1
README.md
CHANGED
|
@@ -16,7 +16,8 @@ datasets:
|
|
| 16 |
A static page into the public pointer index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index)
|
| 17 |
for people who build tasks for a terminal agent, and for their agent: task material (material sets in their
|
| 18 |
collections) and corpora as two layers, their counts and their sizes by layer and collection on the first screen, one
|
| 19 |
-
card per collection, the largest material sets, the most requested domains,
|
|
|
|
| 20 |
form map with the bars by domain and by form for one layer at a time (linked: a click on one filters the others and the
|
| 21 |
list), the material sets or corpora with their fit and a preview of their rows, and one command per layer to take a
|
| 22 |
sample.
|
|
@@ -161,6 +162,7 @@ The first screen (`first.js`) and the reader of the index (`data.js`) share that
|
|
| 161 |
| `meta/projection.json` | at load, when the commit has it | `tag`, the release tag the page's commands name |
|
| 162 |
| `stages.json` (#2006: beside the page, written by the deploy) | at load, when the deploy wrote one | `index` (the index commit it was computed for; read only when it is the commit the page reads), `counted` (whether that commit has the stage fields), `version` (of the stage table), `stages` (`id`, `name`, `zh`, `status`, `about`, `counted_by`, `counts`; #2066: the funnels' stages, whatever commit it names), `sources` (per source the ids of the stages one of its units passes) |
|
| 163 |
| `meta/stage_figures.json` (#2066) | at load, when the commit has it | `rules` (the rules every stage's material passes, in their order), per stage of `stages` (`id`, `counts`, each null before the release counted the stage fields) its `funnel` (#2065: per step `after`, null before the rules or the rule of `rules` it is counted after, and `counts`, by the measures the stage is counted by), the funnels |
|
|
|
|
| 164 |
| `meta/translations.parquet` (#2066: the table of Chinese names #2065 writes with the release) | at load, when the commit has it | `en` (an English text the index shows: a collection's name, a source's title or description as the catalog holds them) and `zh` (its Chinese text) |
|
| 165 |
|
| 166 |
`meta/tags.parquet` and `meta/not_included.parquet` are not read.
|
|
@@ -249,6 +251,10 @@ A selection by stage takes its sources by `--source`, under the default profile,
|
|
| 249 |
Each step is named as the dataset card's funnel tables name it (#2065: "After the benchmark rule", down to "After deduplication: the stage's count").
|
| 250 |
Until a release has counted its stage fields (no `meta/stage_figures.json`, or its counts empty), the section says so in full, as the filter's note does (`funnel.notCounted`);
|
| 251 |
a release that counts them without what each rule leaves says that.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 252 |
|
| 253 |
## Languages
|
| 254 |
|
|
@@ -384,7 +390,8 @@ page's language.
|
|
| 384 |
title), the largest material sets as
|
| 385 |
bars (#2066: `plots.js` `bars`, each named by its title, its id, form and sizes on hover, a click opening it in the list; the rest in one line under the chart and its table, never as a bar; the table with their recorded and deduplicated sizes), the most requested domains (the constant
|
| 386 |
`MOST_REQUESTED`, ten: one card each, in two rows of five from 1,100 px; those with material by size, then the
|
| 387 |
-
wanted ones with a request; `index.html` has ten skeleton cards to match), the
|
|
|
|
| 388 |
`renderStages`), the filters and their URL state (`F`), the
|
| 389 |
chips of the selection, the sections, the fit of a source, the command, the SQL and the CSV.
|
| 390 |
A source is named by its title wherever it is read (the largest material sets, the source table, the
|
|
@@ -507,7 +514,8 @@ page starts, since the first screen holds one of its charts: the builds vgplot i
|
|
| 507 |
`markdown-it@15.0.3` (the manual's Markdown, loaded with `manual.js` the first time the manual is shown).
|
| 508 |
There is no build step. To look at it locally: `python -m http.server` in this folder (without a
|
| 509 |
`first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
|
| 510 |
-
writes one; without a `stages.json` the filter by stage is off
|
|
|
|
| 511 |
|
| 512 |
## Deploy
|
| 513 |
|
|
@@ -524,6 +532,8 @@ a commit whose figures have no counts gets them from its catalog by the same rul
|
|
| 524 |
commit has it).
|
| 525 |
#2066: the page draws the counts of `first.json` at once; the first screen's sizes are its figure by layer and collection, drawn from the index's `meta/size_figures.json` once the page has read it.
|
| 526 |
#2006: it also adds `stages.json` (format `portal-stages/1`), the stage table of its code tree evaluated over the stage fields of the same index commit (section Query parameters), and records the sha256 of the table in `portal.json`, so a changed table is deployed even when the index has not changed.
|
|
|
|
|
|
|
| 527 |
|
| 528 |
#1617, #1633, #2060: the manual's texts are Jinja2 templates, the engine of the index's card and docs, and go to the
|
| 529 |
Space rendered, both with the same values of their markers, computed once per deploy from the index commit it reads.
|
|
|
|
| 16 |
A static page into the public pointer index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index)
|
| 17 |
for people who build tasks for a terminal agent, and for their agent: task material (material sets in their
|
| 18 |
collections) and corpora as two layers, their counts and their sizes by layer and collection on the first screen, one
|
| 19 |
+
card per collection, the largest material sets, the most requested domains, the funnel from the corpus to the training
|
| 20 |
+
stages, each training stage's funnel, the domain ร
|
| 21 |
form map with the bars by domain and by form for one layer at a time (linked: a click on one filters the others and the
|
| 22 |
list), the material sets or corpora with their fit and a preview of their rows, and one command per layer to take a
|
| 23 |
sample.
|
|
|
|
| 162 |
| `meta/projection.json` | at load, when the commit has it | `tag`, the release tag the page's commands name |
|
| 163 |
| `stages.json` (#2006: beside the page, written by the deploy) | at load, when the deploy wrote one | `index` (the index commit it was computed for; read only when it is the commit the page reads), `counted` (whether that commit has the stage fields), `version` (of the stage table), `stages` (`id`, `name`, `zh`, `status`, `about`, `counted_by`, `counts`; #2066: the funnels' stages, whatever commit it names), `sources` (per source the ids of the stages one of its units passes) |
|
| 164 |
| `meta/stage_figures.json` (#2066) | at load, when the commit has it | `rules` (the rules every stage's material passes, in their order), per stage of `stages` (`id`, `counts`, each null before the release counted the stage fields) its `funnel` (#2065: per step `after`, null before the rules or the rule of `rules` it is counted after, and `counts`, by the measures the stage is counted by), the funnels |
|
| 165 |
+
| `funnel.json` (#2287: beside the page, written by the deploy) | at load, when the deploy wrote one | `release` (the release tag it counts) and `index` (its merge commit), and `funnel`, the funnel's index half as the release counted it (the metadata of the release's asset `release_published`, which the task dashboard reads too): `layers` (`corpus`, its `rows`; `chosen` and `admitted`, their `units`; each with how many sources are not counted yet and `by_domain`, its count under each main domain; `admitted` also `promoted` and `search`) and `stages` (`id`, `name`, `zh`, `status`, `measure`, `before`, `kept` and `by_domain`), the figure from the corpus to the training stages |
|
| 166 |
| `meta/translations.parquet` (#2066: the table of Chinese names #2065 writes with the release) | at load, when the commit has it | `en` (an English text the index shows: a collection's name, a source's title or description as the catalog holds them) and `zh` (its Chinese text) |
|
| 167 |
|
| 168 |
`meta/tags.parquet` and `meta/not_included.parquet` are not read.
|
|
|
|
| 251 |
Each step is named as the dataset card's funnel tables name it (#2065: "After the benchmark rule", down to "After deduplication: the stage's count").
|
| 252 |
Until a release has counted its stage fields (no `meta/stage_figures.json`, or its counts empty), the section says so in full, as the filter's note does (`funnel.notCounted`);
|
| 253 |
a release that counts them without what each rule leaves says that.
|
| 254 |
+
#2287: before the stages' funnels, the funnel's index half is one figure (`app.js` `renderFlow`, `plots.js` `bars`), from `funnel.json` beside the page, whatever commit it names, its title naming the release it counts: the corpora's rows, the units selection files of corpus rows chose for admission (#1909), the units admitted into material sets (on hover, those promoted from the corpus and those from the search) and what each training stage keeps, in its units or rows (on hover, what is in its condition before the rules), one bar each on one log axis, in the color of the layer each counts.
|
| 255 |
+
Its Table view gives the same layers under each main domain: a source counted under the top-level domain of its first domain code.
|
| 256 |
+
These are the numbers the task dashboard joins its task half to (`06_tasks/orchestration/README.md`, "The funnel"), so the units admitted into material sets are one number in both;
|
| 257 |
+
without `funnel.json` the figure says that no release has counted it yet (`flow.none`).
|
| 258 |
|
| 259 |
## Languages
|
| 260 |
|
|
|
|
| 390 |
title), the largest material sets as
|
| 391 |
bars (#2066: `plots.js` `bars`, each named by its title, its id, form and sizes on hover, a click opening it in the list; the rest in one line under the chart and its table, never as a bar; the table with their recorded and deduplicated sizes), the most requested domains (the constant
|
| 392 |
`MOST_REQUESTED`, ten: one card each, in two rows of five from 1,100 px; those with material by size, then the
|
| 393 |
+
wanted ones with a request; `index.html` has ten skeleton cards to match), the funnel from the corpus to the training
|
| 394 |
+
stages (#2287, `renderFlow`), the training stages' funnels (#2066,
|
| 395 |
`renderStages`), the filters and their URL state (`F`), the
|
| 396 |
chips of the selection, the sections, the fit of a source, the command, the SQL and the CSV.
|
| 397 |
A source is named by its title wherever it is read (the largest material sets, the source table, the
|
|
|
|
| 514 |
`markdown-it@15.0.3` (the manual's Markdown, loaded with `manual.js` the first time the manual is shown).
|
| 515 |
There is no build step. To look at it locally: `python -m http.server` in this folder (without a
|
| 516 |
`first.json` the hero shows skeletons until the index is read; `03_data_line/steps/portal.py --write-first first.json`
|
| 517 |
+
writes one; without a `stages.json` the filter by stage is off; without a `funnel.json` the figure from the corpus to
|
| 518 |
+
the training stages says so).
|
| 519 |
|
| 520 |
## Deploy
|
| 521 |
|
|
|
|
| 532 |
commit has it).
|
| 533 |
#2066: the page draws the counts of `first.json` at once; the first screen's sizes are its figure by layer and collection, drawn from the index's `meta/size_figures.json` once the page has read it.
|
| 534 |
#2006: it also adds `stages.json` (format `portal-stages/1`), the stage table of its code tree evaluated over the stage fields of the same index commit (section Query parameters), and records the sha256 of the table in `portal.json`, so a changed table is deployed even when the index has not changed.
|
| 535 |
+
#2287: and `funnel.json` (format `portal-funnel/1`), the funnel's index half of the newest release that merged one, which the portal asset hands the step from that release's `release_published` metadata (`03_data_line/orchestration/definitions.py` `published_funnel`), and records that release's merge commit in `portal.json`, so a new funnel is deployed even when nothing else changed;
|
| 536 |
+
when the newest release merged nothing, the Space keeps the `funnel.json` it has.
|
| 537 |
|
| 538 |
#1617, #1633, #2060: the manual's texts are Jinja2 templates, the engine of the index's card and docs, and go to the
|
| 539 |
Space rendered, both with the same values of their markers, computed once per deploy from the index commit it reads.
|
app.js
CHANGED
|
@@ -1817,12 +1817,52 @@ function renderStages() {
|
|
| 1817 |
}
|
| 1818 |
}
|
| 1819 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1820 |
// ---- the Plot charts of the page (plots.js), drawn again once the module is loaded and when their width changes
|
| 1821 |
let plotWidths = '';
|
| 1822 |
-
const plotFigs = () => ['size-fig', 'where-src', 'history-fig', ...IX.stageList.map((st) => `funnel-${st.id}`)].filter((id) => $(id));
|
| 1823 |
function drawPlots() {
|
| 1824 |
renderSizes();
|
| 1825 |
renderWhere();
|
|
|
|
| 1826 |
renderStages();
|
| 1827 |
renderHistory();
|
| 1828 |
plotWidths = plotFigs().map(contentWidth).join(' ');
|
|
@@ -1951,6 +1991,7 @@ function renderLanguage() {
|
|
| 1951 |
renderHeader();
|
| 1952 |
renderCategories();
|
| 1953 |
renderCollections();
|
|
|
|
| 1954 |
renderStages();
|
| 1955 |
fillFilterOptions();
|
| 1956 |
drawCharts();
|
|
@@ -2035,6 +2076,7 @@ async function main() {
|
|
| 2035 |
renderHeader();
|
| 2036 |
renderCategories();
|
| 2037 |
renderCollections();
|
|
|
|
| 2038 |
renderStages();
|
| 2039 |
fillFilterRow();
|
| 2040 |
syncFilterRow();
|
|
|
|
| 1817 |
}
|
| 1818 |
}
|
| 1819 |
|
| 1820 |
+
// ---- #2287: the funnel's index half in one figure (funnel.json beside the page: the metadata of the release's asset
|
| 1821 |
+
// release_published, which the task dashboard reads too, so both give one number): the corpora's rows, the units
|
| 1822 |
+
// chosen for admission by selection files of corpus rows, the units admitted into material sets and what each
|
| 1823 |
+
// training stage keeps, one bar each on one log axis (they span thousands to billions), in the color of the layer
|
| 1824 |
+
// each counts; its Table view the same under each main domain. Without funnel.json the figure says so.
|
| 1825 |
+
const flowDomain = (code) => domainName(code === 'multi' ? MULTI : code === 'unclassified' ? NONE : code);
|
| 1826 |
+
function flowLayers(f) {
|
| 1827 |
+
const ly = f.funnel.layers;
|
| 1828 |
+
const unit = (st) => (st.measure === 'units' ? 'n.unit' : 'n.corpusRow');
|
| 1829 |
+
return [
|
| 1830 |
+
{ key: 'corpus', name: t('flow.corpus'), kind: CORPUS, n: ly.corpus.rows, noun: 'n.corpusRow', by: ly.corpus.by_domain,
|
| 1831 |
+
missing: ly.corpus.rows_not_counted },
|
| 1832 |
+
{ key: 'chosen', name: t('flow.chosen'), kind: CORPUS, n: ly.chosen.units, noun: 'n.unit', by: ly.chosen.by_domain,
|
| 1833 |
+
missing: ly.chosen.units_not_counted },
|
| 1834 |
+
{ key: 'admitted', name: t('flow.admitted'), kind: MATERIAL_SET, n: ly.admitted.units, noun: 'n.unit', by: ly.admitted.by_domain,
|
| 1835 |
+
missing: ly.admitted.units_not_counted,
|
| 1836 |
+
more: t('flow.split', { promoted: fmtInt(ly.admitted.promoted), search: fmtInt(ly.admitted.search) }) },
|
| 1837 |
+
...f.funnel.stages.map((st) => ({ key: `stage:${st.id}`, name: t('flow.kept', { stage: stageTitle(st) }),
|
| 1838 |
+
kind: st.measure === 'units' ? MATERIAL_SET : CORPUS, n: st.kept, noun: unit(st),
|
| 1839 |
+
by: Object.fromEntries(Object.entries(st.by_domain || {}).map(([d, x]) => [d, x.kept])),
|
| 1840 |
+
more: st.before == null ? null : t('flow.before', { what: plural(unit(st), st.before) }) })),
|
| 1841 |
+
];
|
| 1842 |
+
}
|
| 1843 |
+
function renderFlow() {
|
| 1844 |
+
const fig = $('flow-fig');
|
| 1845 |
+
const layers = IX.funnel ? flowLayers(IX.funnel) : [];
|
| 1846 |
+
const rows = layers.filter((x) => x.n > 0).map((x) => ({ key: x.key, name: x.name, value: x.n, text: fmtCompact(x.n),
|
| 1847 |
+
color: x.kind === CORPUS ? 'var(--other)' : 'var(--ours)', onBar: x.kind === CORPUS ? 'var(--ink)' : null,
|
| 1848 |
+
title: [x.name, plural(x.noun, x.n) + uncounted(x.missing), x.more].filter(Boolean).join('\n') }));
|
| 1849 |
+
if (!rows.length) { clear(fig).append(el('p', { class: 'small' }, t('flow.none'))); return; }
|
| 1850 |
+
const domains = [...new Set(layers.flatMap((x) => Object.keys(x.by || {})))]
|
| 1851 |
+
.sort((a, b) => (layers[0].by?.[b] || 0) - (layers[0].by?.[a] || 0) || a.localeCompare(b));
|
| 1852 |
+
figureWithToggle(fig, 'flow', t('flow.title', { release: IX.funnel.release || IX.funnel.index?.slice(0, 12) }), {
|
| 1853 |
+
chart: () => plotted(() => P.bars(rows, { width: contentWidth('flow-fig'), log: true, fmt: fmtCompact, label: t('flow.head') })),
|
| 1854 |
+
table: () => table([t('flow.domain'), ...layers.map((x) => ({ label: x.name, r: true }))],
|
| 1855 |
+
domains.map((d) => [flowDomain(d), ...layers.map((x) => (x.n == null ? '' : fmtInt(x.by?.[d] || 0)))])),
|
| 1856 |
+
}, () => el('p', { class: 'small fig-foot' }, tableShown('flow') ? t('flow.byDomain') : t('size.logScale')));
|
| 1857 |
+
}
|
| 1858 |
+
|
| 1859 |
// ---- the Plot charts of the page (plots.js), drawn again once the module is loaded and when their width changes
|
| 1860 |
let plotWidths = '';
|
| 1861 |
+
const plotFigs = () => ['size-fig', 'where-src', 'flow-fig', 'history-fig', ...IX.stageList.map((st) => `funnel-${st.id}`)].filter((id) => $(id));
|
| 1862 |
function drawPlots() {
|
| 1863 |
renderSizes();
|
| 1864 |
renderWhere();
|
| 1865 |
+
renderFlow();
|
| 1866 |
renderStages();
|
| 1867 |
renderHistory();
|
| 1868 |
plotWidths = plotFigs().map(contentWidth).join(' ');
|
|
|
|
| 1991 |
renderHeader();
|
| 1992 |
renderCategories();
|
| 1993 |
renderCollections();
|
| 1994 |
+
renderFlow();
|
| 1995 |
renderStages();
|
| 1996 |
fillFilterOptions();
|
| 1997 |
drawCharts();
|
|
|
|
| 2076 |
renderHeader();
|
| 2077 |
renderCategories();
|
| 2078 |
renderCollections();
|
| 2079 |
+
renderFlow();
|
| 2080 |
renderStages();
|
| 2081 |
fillFilterRow();
|
| 2082 |
syncFilterRow();
|
data.js
CHANGED
|
@@ -351,7 +351,7 @@ export async function loadIndex(params, onNote) {
|
|
| 351 |
const vocabFile = vocabUrl ? () => urlFile(vocabUrl) : has('meta/vocabulary.json') ? () => at('meta/vocabulary.json') : null;
|
| 352 |
|
| 353 |
const [cat, sourcesJson, coverage, batches, releases, changelog, vocabJson, figures, record, stagesJson, translations,
|
| 354 |
-
stageFigures] = await Promise.all([
|
| 355 |
catalogUrl ? readCatalog(urlFile(catalogUrl), /\.json(\?|$)/i.test(catalogUrl)) : readCatalog(at('meta/catalog.parquet'), false),
|
| 356 |
jsonOf(at('meta/sources.json')),
|
| 357 |
jsonOf(at('meta/coverage.json')),
|
|
@@ -371,6 +371,10 @@ export async function loadIndex(params, onNote) {
|
|
| 371 |
// #2066: the Chinese names of collections and sources (#2065), and the counts per stage with what each rule leaves
|
| 372 |
has(TRANSLATIONS) ? readParquet(at(TRANSLATIONS)) : [],
|
| 373 |
has(STAGE_FIGURES) ? jsonOf(at(STAGE_FIGURES)) : null,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 374 |
]);
|
| 375 |
// #2006: the stages describe the commit the page reads only when stages.json names it and was counted over its stage
|
| 376 |
// fields; otherwise no source has a stage and the filter by stage says why (#2066: in full, stagesWhy): no
|
|
@@ -498,6 +502,9 @@ export async function loadIndex(params, onNote) {
|
|
| 498 |
stagesWhy,
|
| 499 |
stageList: stagesJson?.stages || stageFigures?.stages || [],
|
| 500 |
stageFigures: parseStageFigures(stageFigures),
|
|
|
|
|
|
|
|
|
|
| 501 |
// #2066: the Chinese text of each English text the release shows (TRANSLATIONS, #2065): English -> Chinese, empty
|
| 502 |
// for a commit without the table
|
| 503 |
zh: new Map(translations.map((r) => [r.en, r.zh])),
|
|
|
|
| 351 |
const vocabFile = vocabUrl ? () => urlFile(vocabUrl) : has('meta/vocabulary.json') ? () => at('meta/vocabulary.json') : null;
|
| 352 |
|
| 353 |
const [cat, sourcesJson, coverage, batches, releases, changelog, vocabJson, figures, record, stagesJson, translations,
|
| 354 |
+
stageFigures, funnelJson] = await Promise.all([
|
| 355 |
catalogUrl ? readCatalog(urlFile(catalogUrl), /\.json(\?|$)/i.test(catalogUrl)) : readCatalog(at('meta/catalog.parquet'), false),
|
| 356 |
jsonOf(at('meta/sources.json')),
|
| 357 |
jsonOf(at('meta/coverage.json')),
|
|
|
|
| 371 |
// #2066: the Chinese names of collections and sources (#2065), and the counts per stage with what each rule leaves
|
| 372 |
has(TRANSLATIONS) ? readParquet(at(TRANSLATIONS)) : [],
|
| 373 |
has(STAGE_FIGURES) ? jsonOf(at(STAGE_FIGURES)) : null,
|
| 374 |
+
// #2287: the funnel's index half beside the page (03_data_line/steps/portal.py funnel_screen: the metadata of the
|
| 375 |
+
// release's asset release_published, which the task dashboard reads too), read past the browser's cache as
|
| 376 |
+
// stages.json is; a page without it draws no funnel figure
|
| 377 |
+
fetch('funnel.json', { cache: 'no-cache' }).then((r) => (r.ok ? r.json() : null)).catch(() => null),
|
| 378 |
]);
|
| 379 |
// #2006: the stages describe the commit the page reads only when stages.json names it and was counted over its stage
|
| 380 |
// fields; otherwise no source has a stage and the filter by stage says why (#2066: in full, stagesWhy): no
|
|
|
|
| 502 |
stagesWhy,
|
| 503 |
stageList: stagesJson?.stages || stageFigures?.stages || [],
|
| 504 |
stageFigures: parseStageFigures(stageFigures),
|
| 505 |
+
// #2287: the funnel's index half (funnel.json: release, the release tag it counts, index, its merge commit, and
|
| 506 |
+
// funnel, its layers and stages by domain), whatever commit it names; null without it
|
| 507 |
+
funnel: funnelJson?.funnel ? funnelJson : null,
|
| 508 |
// #2066: the Chinese text of each English text the release shows (TRANSLATIONS, #2065): English -> Chinese, empty
|
| 509 |
// for a commit without the table
|
| 510 |
zh: new Map(translations.map((r) => [r.en, r.zh])),
|
first.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
"collections": 14,
|
| 3 |
"date": "2026-11-10",
|
| 4 |
"format": "portal-first/5",
|
| 5 |
-
"index": "
|
| 6 |
"raw_rows": 1216362128,
|
| 7 |
"raw_rows_not_counted": 0,
|
| 8 |
"raw_sources": 139,
|
|
|
|
| 2 |
"collections": 14,
|
| 3 |
"date": "2026-11-10",
|
| 4 |
"format": "portal-first/5",
|
| 5 |
+
"index": "83dae072040e8688c08c84fb72fdc8690d0956e5",
|
| 6 |
"raw_rows": 1216362128,
|
| 7 |
"raw_rows_not_counted": 0,
|
| 8 |
"raw_sources": 139,
|
funnel.json
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"format": "portal-funnel/1",
|
| 3 |
+
"funnel": {
|
| 4 |
+
"counted_on": null,
|
| 5 |
+
"domain_names": {
|
| 6 |
+
"D1": "software-engineering",
|
| 7 |
+
"D10": "misc",
|
| 8 |
+
"D2": "system-administration",
|
| 9 |
+
"D3": "data-science",
|
| 10 |
+
"D4": "machine-learning",
|
| 11 |
+
"D5": "science",
|
| 12 |
+
"D6": "operations",
|
| 13 |
+
"D7": "hardware",
|
| 14 |
+
"D8": "security",
|
| 15 |
+
"D9": "media",
|
| 16 |
+
"multi": "many domains",
|
| 17 |
+
"unclassified": "unclassified"
|
| 18 |
+
},
|
| 19 |
+
"format": "index-funnel/1",
|
| 20 |
+
"layers": {
|
| 21 |
+
"admitted": {
|
| 22 |
+
"by_domain": {
|
| 23 |
+
"D1": 479,
|
| 24 |
+
"D10": 59,
|
| 25 |
+
"D2": 274,
|
| 26 |
+
"D3": 1039,
|
| 27 |
+
"D4": 502,
|
| 28 |
+
"D5": 388,
|
| 29 |
+
"D6": 426,
|
| 30 |
+
"D7": 296,
|
| 31 |
+
"D8": 329,
|
| 32 |
+
"D9": 391
|
| 33 |
+
},
|
| 34 |
+
"material_sets": 1799,
|
| 35 |
+
"promoted": 0,
|
| 36 |
+
"search": 4183,
|
| 37 |
+
"units": 4183,
|
| 38 |
+
"units_not_counted": 0
|
| 39 |
+
},
|
| 40 |
+
"chosen": {
|
| 41 |
+
"by_domain": {
|
| 42 |
+
"D8": 0
|
| 43 |
+
},
|
| 44 |
+
"material_sets": 1,
|
| 45 |
+
"units": 0,
|
| 46 |
+
"units_not_counted": 0
|
| 47 |
+
},
|
| 48 |
+
"corpus": {
|
| 49 |
+
"by_domain": {
|
| 50 |
+
"D1": 45267701,
|
| 51 |
+
"D10": 241125,
|
| 52 |
+
"D2": 185826217,
|
| 53 |
+
"D3": 750815212,
|
| 54 |
+
"D4": 9636889,
|
| 55 |
+
"D5": 44633675,
|
| 56 |
+
"D6": 39404642,
|
| 57 |
+
"D7": 8716520,
|
| 58 |
+
"D8": 4521631,
|
| 59 |
+
"D9": 12751612,
|
| 60 |
+
"multi": 109262742,
|
| 61 |
+
"unclassified": 5284162
|
| 62 |
+
},
|
| 63 |
+
"corpora": 139,
|
| 64 |
+
"rows": 1216362128,
|
| 65 |
+
"rows_not_counted": 0
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
"stages": [
|
| 69 |
+
{
|
| 70 |
+
"before": null,
|
| 71 |
+
"by_domain": {},
|
| 72 |
+
"bytes": null,
|
| 73 |
+
"domains": [],
|
| 74 |
+
"forms": [],
|
| 75 |
+
"id": "post-training",
|
| 76 |
+
"kept": null,
|
| 77 |
+
"kind": "material_set",
|
| 78 |
+
"measure": "units",
|
| 79 |
+
"name": "Post-training",
|
| 80 |
+
"status": "in use",
|
| 81 |
+
"zh": "ๅ่ฎญ็ป"
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"before": null,
|
| 85 |
+
"by_domain": {},
|
| 86 |
+
"bytes": null,
|
| 87 |
+
"domains": [],
|
| 88 |
+
"forms": [],
|
| 89 |
+
"id": "mid-training",
|
| 90 |
+
"kept": null,
|
| 91 |
+
"kind": "corpus",
|
| 92 |
+
"measure": "rows",
|
| 93 |
+
"name": "Mid-training",
|
| 94 |
+
"status": "provisional",
|
| 95 |
+
"zh": "ไธญ่ฎญ็ป"
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"before": null,
|
| 99 |
+
"by_domain": {},
|
| 100 |
+
"bytes": null,
|
| 101 |
+
"domains": [],
|
| 102 |
+
"forms": [],
|
| 103 |
+
"id": "pre-training",
|
| 104 |
+
"kept": null,
|
| 105 |
+
"kind": "corpus",
|
| 106 |
+
"measure": "rows",
|
| 107 |
+
"name": "Pre-training",
|
| 108 |
+
"status": "provisional",
|
| 109 |
+
"zh": "้ข่ฎญ็ป"
|
| 110 |
+
}
|
| 111 |
+
],
|
| 112 |
+
"table": 1
|
| 113 |
+
},
|
| 114 |
+
"index": "83dae072040e8688c08c84fb72fdc8690d0956e5",
|
| 115 |
+
"release": "release-2026-11-10.2"
|
| 116 |
+
}
|
index.html
CHANGED
|
@@ -111,6 +111,15 @@
|
|
| 111 |
<ul class="cat-grid" id="req-cards">
|
| 112 |
<li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
|
| 113 |
</ul>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
<!-- #2066: each training stage's funnel (app.js renderStages, plots.js bars): its units or rows through each rule, in
|
| 115 |
the order of meta/stage_figures.json, down to its count; until the release has counted them, why, in full -->
|
| 116 |
<div class="band-head sub-head reveal"><h3 id="stages-h" data-i18n="funnel.head">Training stages: what each rule leaves</h3>
|
|
|
|
| 111 |
<ul class="cat-grid" id="req-cards">
|
| 112 |
<li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li><li class="card cat sk-card"></li>
|
| 113 |
</ul>
|
| 114 |
+
<!-- #2287: the funnel's index half in one figure (app.js renderFlow, plots.js bars): the corpora's rows, the units
|
| 115 |
+
chosen for admission, the units admitted into material sets and what each training stage keeps, on one log
|
| 116 |
+
axis, its Table view by domain (funnel.json beside the page: the metadata of the release's asset
|
| 117 |
+
release_published, which the task dashboard reads too) -->
|
| 118 |
+
<div class="band-head sub-head reveal"><h3 id="flow-h" data-i18n="flow.head">From the corpus to the training stages</h3>
|
| 119 |
+
<p class="small band-note" data-i18n="flow.note">What the corpora hold, what selection files chose of them for admission, the units admitted into material sets and what each training stage keeps, as the newest release counted them.</p></div>
|
| 120 |
+
<div class="where-grid one">
|
| 121 |
+
<figure class="card chart-card where-card" id="flow-fig"><div class="sk sk-block"></div></figure>
|
| 122 |
+
</div>
|
| 123 |
<!-- #2066: each training stage's funnel (app.js renderStages, plots.js bars): its units or rows through each rule, in
|
| 124 |
the order of meta/stage_figures.json, down to its count; until the release has counted them, why, in full -->
|
| 125 |
<div class="band-head sub-head reveal"><h3 id="stages-h" data-i18n="funnel.head">Training stages: what each rule leaves</h3>
|
llms.txt
CHANGED
|
@@ -5,7 +5,7 @@
|
|
| 5 |
- Size of this release: Whole index: Recorded size 4.45 PB; deduplicated size 4.05 PB; size check: 99.7% match (37,927 of the 38,820 rows read compared); not compared: address unreachable 459, file gone 82, login needed 330, no size from the server 19, other error 3; estimated size of 215,326,513 of the 312,300,338 rows without a recorded size: 202 TB (an estimate; 95% range 171 TB to 245 TB, the sum of its sources' ranges, wider than a joint interval; from 14,161 rows read), the other 96,973,825 with too few rows read for an estimate. Task material: Recorded size 2.89 GB; deduplicated size 2.82 GB; size check: 99.8% match (7,204 of the 7,226 rows read compared); not compared: address unreachable 10, file gone 2, login needed 8, other error 2. Corpora: Recorded size 4.45 PB; deduplicated size 4.05 PB; size check: 99.6% match (30,723 of the 31,594 rows read compared); not compared: address unreachable 449, file gone 80, login needed 322, no size from the server 19, other error 1; estimated size of 215,326,513 of the 312,300,338 rows without a recorded size: 202 TB (an estimate; 95% range 171 TB to 245 TB, the sum of its sources' ranges, wider than a joint interval; from 14,161 rows read), the other 96,973,825 with too few rows read for an estimate. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted after each release over the data files it published, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it. The recorded size is the sum of `size_bytes` over the rows that have one (catalog column `bytes`). The check reads a sample of rows of every data file (`meta/sizes.parquet`): 20 of its rows with a recorded size, and of its rows without one its share of its source's sample, sized for their mean size within 25% at 95% confidence by the spread of the sizes read so far (at most 400 rows per source per round, 20 for a source with none read yet); its share is the rows that match their recorded size out of the rows it compared (status `match`, `size_mismatch` or `hash_mismatch`), and the rows it could not compare, which never reached their file, are counted beside it by status (`unreachable`, `gone`, `needs_auth`, `no_length`, `error`; catalog column `size_check`). The estimate for a source's rows without a recorded size (catalog columns `unsized_rows` and `unsized_estimate`) is their number times the mean measured size of such rows read, with a 95% BCa bootstrap range (`scipy.stats.bootstrap`, its default resample count); a source with fewer than 10 such rows read has none. Above one source, the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider than a joint interval. The figures of every level are in `meta/size_figures.json`.
|
| 6 |
- Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--kind` (`material_set` or `corpus`: one layer), `--domain`, `--form`, `--tag`, `--ext`, or `--selection <file>` (a selection file the data page saves: the ids of its sources and its commit); `--sample N` takes N units per source, `--limit N` takes N units over the whole selection, `--unit <unit_id>` takes the unit of that `unit_id` whole (with `--source`); before its first download the run prints the units, files and total size it selected.
|
| 7 |
- It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
|
| 8 |
-
- The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, rows of groupings awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet), and `--all` turns off every criterion except benchmark and not compared, so rows that matched a benchmark or were not compared yet come in only with their own options. Compared with the benchmark list (keys of 2026-11-
|
| 9 |
- Output: `./data/<domain>/<source>/<unit folder>/...` holds each unit whole, in a folder of its own named after its `unit_id`, its files at their paths within the unit; `./data/<domain>/<source>/` also holds the README.md and LICENSE of a material set; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, origin_kind, leaf, unit_id, verified); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
|
| 10 |
- The index holds material only: what a task checks is decided when the task is built from the material. Keep out of the environment the agent works in whatever the task checks its result against:
|
| 11 |
- Workbooks (such as the form `spreadsheets`): a published workbook holds the cached results of its formulas, so a workbook put in as fetched hands the agent the values a task may ask it to compute. Strip the cached values first: in .xlsx and .xlsm the `<v>` of every formula cell (`xl/worksheets/*.xml`), the chart caches (`c:numCache` and `c:strCache` in `xl/charts/*.xml`), the pivot caches (`xl/pivotCache/`, definitions and records) and the external link caches (`xl/externalLinks/`); in .ods the value attributes (`office:value` and the like) and the text of formula cells; in .xls the cached result of every FORMULA record and its STRING record.
|
|
|
|
| 5 |
- Size of this release: Whole index: Recorded size 4.45 PB; deduplicated size 4.05 PB; size check: 99.7% match (37,927 of the 38,820 rows read compared); not compared: address unreachable 459, file gone 82, login needed 330, no size from the server 19, other error 3; estimated size of 215,326,513 of the 312,300,338 rows without a recorded size: 202 TB (an estimate; 95% range 171 TB to 245 TB, the sum of its sources' ranges, wider than a joint interval; from 14,161 rows read), the other 96,973,825 with too few rows read for an estimate. Task material: Recorded size 2.89 GB; deduplicated size 2.82 GB; size check: 99.8% match (7,204 of the 7,226 rows read compared); not compared: address unreachable 10, file gone 2, login needed 8, other error 2. Corpora: Recorded size 4.45 PB; deduplicated size 4.05 PB; size check: 99.6% match (30,723 of the 31,594 rows read compared); not compared: address unreachable 449, file gone 80, login needed 322, no size from the server 19, other error 1; estimated size of 215,326,513 of the 312,300,338 rows without a recorded size: 202 TB (an estimate; 95% range 171 TB to 245 TB, the sum of its sources' ranges, wider than a joint interval; from 14,161 rows read), the other 96,973,825 with too few rows read for an estimate. Distinct content is the sum of `size_bytes` over the rows minus the sizes of the rows that repeat another row's content (the same sha256, else sha1, else md5 and size, within the same hash scope; one copy of each content is kept, in the first source by registry order), counted after each release over the data files it published, per source in the catalog column `distinct_bytes`; archive members whose archive the same source also lists whole have other hashes, so they are not subtracted. `tools/fetch.py` removes duplicates over your own selection when it reads it. The recorded size is the sum of `size_bytes` over the rows that have one (catalog column `bytes`). The check reads a sample of rows of every data file (`meta/sizes.parquet`): 20 of its rows with a recorded size, and of its rows without one its share of its source's sample, sized for their mean size within 25% at 95% confidence by the spread of the sizes read so far (at most 400 rows per source per round, 20 for a source with none read yet); its share is the rows that match their recorded size out of the rows it compared (status `match`, `size_mismatch` or `hash_mismatch`), and the rows it could not compare, which never reached their file, are counted beside it by status (`unreachable`, `gone`, `needs_auth`, `no_length`, `error`; catalog column `size_check`). The estimate for a source's rows without a recorded size (catalog columns `unsized_rows` and `unsized_estimate`) is their number times the mean measured size of such rows read, with a 95% BCa bootstrap range (`scipy.stats.bootstrap`, its default resample count); a source with fewer than 10 such rows read has none. Above one source, the estimate is the sum of its sources' estimates and its range the sum of their ranges, which is wider than a joint interval. The figures of every level are in `meta/size_figures.json`.
|
| 6 |
- Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--kind` (`material_set` or `corpus`: one layer), `--domain`, `--form`, `--tag`, `--ext`, or `--selection <file>` (a selection file the data page saves: the ids of its sources and its commit); `--sample N` takes N units per source, `--limit N` takes N units over the whole selection, `--unit <unit_id>` takes the unit of that `unit_id` whole (with `--source`); before its first download the run prints the units, files and total size it selected.
|
| 7 |
- It pins the commit it reads and prints it; pass `--revision <commit>` to get the same files later. A name that does not exist stops with exit code 2 and the closest names.
|
| 8 |
+
- The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, rows of groupings awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet), and `--all` turns off every criterion except benchmark and not compared, so rows that matched a benchmark or were not compared yet come in only with their own options. Compared with the benchmark list (keys of 2026-11-08): 1,799 of 1,799 material sets; 139 of 139 corpora. The profile also removes every row whose data file was not compared (`--allow-benchmark` keeps them, unchecked).
|
| 9 |
- Output: `./data/<domain>/<source>/<unit folder>/...` holds each unit whole, in a folder of its own named after its `unit_id`, its files at their paths within the unit; `./data/<domain>/<source>/` also holds the README.md and LICENSE of a material set; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, origin_kind, leaf, unit_id, verified); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`.
|
| 10 |
- The index holds material only: what a task checks is decided when the task is built from the material. Keep out of the environment the agent works in whatever the task checks its result against:
|
| 11 |
- Workbooks (such as the form `spreadsheets`): a published workbook holds the cached results of its formulas, so a workbook put in as fetched hands the agent the values a task may ask it to compute. Strip the cached values first: in .xlsx and .xlsm the `<v>` of every formula cell (`xl/worksheets/*.xml`), the chart caches (`c:numCache` and `c:strCache` in `xl/charts/*.xml`), the pivot caches (`xl/pivotCache/`, definitions and records) and the external link caches (`xl/externalLinks/`); in .ods the value attributes (`office:value` and the like) and the text of formula cells; in .xls the cached result of every FORMULA record and its STRING record.
|
locales/en.json
CHANGED
|
@@ -403,6 +403,18 @@
|
|
| 403 |
"rule.not compared": "After the rule on rows not compared with the benchmark list",
|
| 404 |
"rule.personal data": "After the personal-data rule",
|
| 405 |
"rule.duplicate": "After deduplication: the stage's count",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 406 |
"status.flagged": "Flagged, not usable as it is",
|
| 407 |
"fetch.all": "All {{rows}} sampled rows read fetched and matched their recorded size and hash (read {{date}}).",
|
| 408 |
"fetch.some": "{{fetched}} of the {{rows}} sampled rows read ({{share}}) fetched and matched their recorded size and hash (read {{date}}); the others did not.",
|
|
|
|
| 403 |
"rule.not compared": "After the rule on rows not compared with the benchmark list",
|
| 404 |
"rule.personal data": "After the personal-data rule",
|
| 405 |
"rule.duplicate": "After deduplication: the stage's count",
|
| 406 |
+
"flow.head": "From the corpus to the training stages",
|
| 407 |
+
"flow.note": "What the corpora hold, what selection files chose of them for admission, the units admitted into material sets and what each training stage keeps, as the newest release counted them.",
|
| 408 |
+
"flow.title": "Release {{release}}: from the corpus to the training stages",
|
| 409 |
+
"flow.corpus": "Corpus rows",
|
| 410 |
+
"flow.chosen": "Chosen for admission by selection files",
|
| 411 |
+
"flow.admitted": "Admitted into material sets",
|
| 412 |
+
"flow.kept": "{{stage}} keeps",
|
| 413 |
+
"flow.split": "{{promoted}} promoted from the corpus, {{search}} from the search",
|
| 414 |
+
"flow.before": "{{what}} in the stage's condition before the rules",
|
| 415 |
+
"flow.domain": "Domain",
|
| 416 |
+
"flow.byDomain": "Each source is counted under the top-level domain of its first domain code.",
|
| 417 |
+
"flow.none": "This page was deployed without the funnel (no funnel.json beside it): it shows once a release has counted it.",
|
| 418 |
"status.flagged": "Flagged, not usable as it is",
|
| 419 |
"fetch.all": "All {{rows}} sampled rows read fetched and matched their recorded size and hash (read {{date}}).",
|
| 420 |
"fetch.some": "{{fetched}} of the {{rows}} sampled rows read ({{share}}) fetched and matched their recorded size and hash (read {{date}}); the others did not.",
|
locales/zh.json
CHANGED
|
@@ -373,6 +373,18 @@
|
|
| 373 |
"rule.not compared": "ๆชไธๅบๅๆต่ฏๆธ
ๅๆฏๅฏน็่ก็่งๅไนๅ",
|
| 374 |
"rule.personal data": "ไธชไบบๆฐๆฎ่งๅไนๅ",
|
| 375 |
"rule.duplicate": "ๅป้ไนๅ๏ผ่ฏฅ้ถๆฎต็่ฎกๆฐ",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 376 |
"status.flagged": "ๅทฒๆ ่ฎฐ๏ผไธ่ฝ็ดๆฅไฝฟ็จ",
|
| 377 |
"fetch.all": "่ฏป่ฟ็ {{rows}} ไธชๆฝๆ ท่กๅ
จ้จๅๅฐ๏ผๅคงๅฐๅๅๅธไธ่ฎฐๅฝไธ่ด๏ผ่ฏปๅไบ {{date}}๏ผใ",
|
| 378 |
"fetch.some": "่ฏป่ฟ็ {{rows}} ไธชๆฝๆ ท่กไธญๆ {{fetched}} ไธช๏ผ{{share}}๏ผๅๅฐ๏ผๅคงๅฐๅๅๅธไธ่ฎฐๅฝไธ่ด๏ผ่ฏปๅไบ {{date}}๏ผ๏ผๅ
ถไฝๆฒกๆใ",
|
|
|
|
| 373 |
"rule.not compared": "ๆชไธๅบๅๆต่ฏๆธ
ๅๆฏๅฏน็่ก็่งๅไนๅ",
|
| 374 |
"rule.personal data": "ไธชไบบๆฐๆฎ่งๅไนๅ",
|
| 375 |
"rule.duplicate": "ๅป้ไนๅ๏ผ่ฏฅ้ถๆฎต็่ฎกๆฐ",
|
| 376 |
+
"flow.head": "ไป่ฏญๆๅบๅฐๅ่ฎญ็ป้ถๆฎต",
|
| 377 |
+
"flow.note": "่ฏญๆๅบ้ๆๅคๅฐใ้ๆฉๆไปถไปไธญๆๅบๅคๅฐ้ๅปๆถๅฝใๆถๅฝ่ฟๆๆ้็ๅๅ
๏ผไปฅๅๆฏไธช่ฎญ็ป้ถๆฎต็ไธๅคๅฐ๏ผๅๆๆๆฐไธๆฌกๅๅธ็่ฎกๆฐใ",
|
| 378 |
+
"flow.title": "ๅๅธ {{release}}๏ผไป่ฏญๆๅบๅฐๅ่ฎญ็ป้ถๆฎต",
|
| 379 |
+
"flow.corpus": "่ฏญๆๅบๆฐๆฎ่ก",
|
| 380 |
+
"flow.chosen": "้ๆฉๆไปถๆๅบ้ๅปๆถๅฝ็",
|
| 381 |
+
"flow.admitted": "ๆถๅฝ่ฟๆๆ้็",
|
| 382 |
+
"flow.kept": "{{stage}}็ไธ็",
|
| 383 |
+
"flow.split": "{{promoted}} ไธชไป่ฏญๆๅบๆๅ่ๆฅ๏ผ{{search}} ไธชๆฅ่ชๆ็ดข",
|
| 384 |
+
"flow.before": "็ฌฆๅ่ฏฅ้ถๆฎต็ๆกไปถใๅจๅๆก่งๅไนๅ๏ผ{{what}}",
|
| 385 |
+
"flow.domain": "้ขๅ",
|
| 386 |
+
"flow.byDomain": "ๆฏไธชๆฅๆบ่ฎกๅจๅฎ็ฌฌไธไธช้ขๅไปฃ็ ๆๅฑ็้กถๅฑ้ขๅไธใ",
|
| 387 |
+
"flow.none": "ๆฌ้กต้จ็ฝฒๆถๆฒกๆๅธฆไธๆผๆ๏ผๆ่พนๆฒกๆ funnel.json๏ผ๏ผ็ญๆๆฌกๅๅธ่ฎกๆฐไนๅๆไผๆพ็คบใ",
|
| 388 |
"status.flagged": "ๅทฒๆ ่ฎฐ๏ผไธ่ฝ็ดๆฅไฝฟ็จ",
|
| 389 |
"fetch.all": "่ฏป่ฟ็ {{rows}} ไธชๆฝๆ ท่กๅ
จ้จๅๅฐ๏ผๅคงๅฐๅๅๅธไธ่ฎฐๅฝไธ่ด๏ผ่ฏปๅไบ {{date}}๏ผใ",
|
| 390 |
"fetch.some": "่ฏป่ฟ็ {{rows}} ไธชๆฝๆ ท่กไธญๆ {{fetched}} ไธช๏ผ{{share}}๏ผๅๅฐ๏ผๅคงๅฐๅๅๅธไธ่ฎฐๅฝไธ่ด๏ผ่ฏปๅไบ {{date}}๏ผ๏ผๅ
ถไฝๆฒกๆใ",
|
manual/manual.en.md
CHANGED
|
@@ -8,7 +8,7 @@ It is neither a task list nor a copy of the files.
|
|
| 8 |
Entry points: the data page https://huggingface.co/spaces/Battam/agentic-env-catalog (to share a state, use the page itself, https://battam-agentic-env-catalog.static.hf.space/index.html );
|
| 9 |
the dataset https://huggingface.co/datasets/Battam/agentic-env-index .
|
| 10 |
|
| 11 |
-
The numbers and command outputs below were filled in at deploy time, from the index commit `
|
| 12 |
|
| 13 |
## Tutorial
|
| 14 |
|
|
@@ -87,7 +87,7 @@ or add `&open=zenodo-workbooks-cc-by` to the address.
|
|
| 87 |
### 5. Go to "Get the data" and keep the selection
|
| 88 |
|
| 89 |
**Do**: click "Get the data" at the top;
|
| 90 |
-
under "Keep this selection" click "Download the selection list (JSON)" and put `agentic-env-selection-
|
| 91 |
|
| 92 |
**See**: the heading "Take 3 units from each of 52 material sets", then "Units per source", the card "1 Take", "What it writes", the six kinds "Left out by default", and "2 Take it again from the file", which reads the list.
|
| 93 |
The list records release, filters and sources:
|
|
@@ -96,7 +96,7 @@ The list records release, filters and sources:
|
|
| 96 |
{
|
| 97 |
"format": "agentic-env-selection/1",
|
| 98 |
"dataset": "Battam/agentic-env-index",
|
| 99 |
-
"revision": "release-2026-11-10.
|
| 100 |
"filters": {
|
| 101 |
"collection": "A",
|
| 102 |
"train": "yes",
|
|
@@ -120,9 +120,9 @@ for 200 in all, use `--limit`.
|
|
| 120 |
**Do**: in the list's folder, run the command of "Take it again from the file" with `--sample 3` replaced by `--limit 220` (220 units in all, in a fixed order: a margin over 200, since some sites fail) and `--out ./data` by `--out ./take`:
|
| 121 |
|
| 122 |
```sh
|
| 123 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 124 |
-
--selection agentic-env-selection-
|
| 125 |
-
--revision release-2026-11-10.
|
| 126 |
--ext xlsx \
|
| 127 |
--ext xlsm \
|
| 128 |
--ext xls \
|
|
@@ -147,10 +147,10 @@ ok abhishekumarr00-supply-chain-mastery workbooks/abhishekumarr00-supply-chain-m
|
|
| 147 |
ok austender-ict-procurement-statistics workbooks/201314austenderictcontractstatistcs.xlsx: 26.3kB
|
| 148 |
ok austender-ict-procurement-statistics workbooks/2016-17-austender-ict-contract-data-category-sme-and-small-participation.xlsx: 32.9kB
|
| 149 |
...
|
| 150 |
-
|
| 151 |
-
files of units:
|
| 152 |
-
run_id
|
| 153 |
-
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('take/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '
|
| 154 |
```
|
| 155 |
|
| 156 |
Before the first download, `removed by` gives the units the default profile removed and the option that keeps them, counted in the data files read as far as `--limit` needed;
|
|
@@ -165,8 +165,8 @@ failed western-australia-government-workbooks workbooks/wa-gross-regional-produc
|
|
| 165 |
incomplete: 1 unit miss a file (the files fetched are kept; _meta/fetch_manifest.jsonl says why): ...
|
| 166 |
5 files fetched and checked (1.35MB) into probe; details in probe/_meta/fetch_run.json
|
| 167 |
files of units: 1 failed, 1 ok; files of the sources themselves: 4 ok
|
| 168 |
-
run_id
|
| 169 |
-
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('probe/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '
|
| 170 |
```
|
| 171 |
|
| 172 |
**Get**: every `ok` file matched its recorded size and hash.
|
|
@@ -206,18 +206,20 @@ GROUP BY ALL ORDER BY source;
|
|
| 206 |
โ excel_units โ
|
| 207 |
โ int64 โ
|
| 208 |
โโโโโโโโโโโโโโโค
|
| 209 |
-
โ
|
| 210 |
โโโโโโโโโโโโโโโ
|
| 211 |
โโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโโ
|
| 212 |
โ verified โ files โ bytes โ
|
| 213 |
โ boolean โ int64 โ int128 โ
|
| 214 |
โโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโค
|
| 215 |
-
โ true โ
|
| 216 |
โโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโ
|
| 217 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโ
|
| 218 |
โ source โ reason โ files โ
|
| 219 |
โ varchar โ varchar โ int64 โ
|
| 220 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโค
|
|
|
|
|
|
|
| 221 |
โ victoria-government-workbooks โ https://www.planning.vic.gov.au/__data/assets โ 2 โ
|
| 222 |
โ western-australia-government-workbooks โ https://catalogue.data.wa.gov.au/dataset/03b1 โ 1 โ
|
| 223 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโ
|
|
@@ -239,7 +241,7 @@ a workbook stores the cached results of its formulas, so a task that asks for co
|
|
| 239 |
**See** (truncated):
|
| 240 |
|
| 241 |
```text
|
| 242 |
-
Dataset commit
|
| 243 |
...
|
| 244 |
- Source: `austender-ict-procurement-statistics`
|
| 245 |
- License: CC-BY-3.0-AU
|
|
@@ -254,7 +256,7 @@ where it says `not stated`, `spdx`, `licence_url` and `publisher` in `take/_meta
|
|
| 254 |
## How-to guides
|
| 255 |
|
| 256 |
One goal each;
|
| 257 |
-
commands are pinned to `release-2026-11-10.
|
| 258 |
|
| 259 |
### Filter by file type and license
|
| 260 |
|
|
@@ -284,11 +286,11 @@ The dataset also has `cards/<source>.md` and `.yaml`.
|
|
| 284 |
|
| 285 |
### Pin a release
|
| 286 |
|
| 287 |
-
The page's commands pin the script URL and `--revision` to `release-2026-11-10.
|
| 288 |
the script's first line names the commit:
|
| 289 |
|
| 290 |
```text
|
| 291 |
-
reading hf://datasets/Battam/agentic-env-index@
|
| 292 |
```
|
| 293 |
|
| 294 |
Later, `--revision <commit>`, the same tag or the selection list take the same files;
|
|
@@ -297,7 +299,7 @@ On the data page, `rev=` reads another commit.
|
|
| 297 |
A misspelt source name stops with exit code 2:
|
| 298 |
|
| 299 |
```text
|
| 300 |
-
error: --source dosfstools-fsk: no such source in meta/catalog.parquet at commit
|
| 301 |
```
|
| 302 |
|
| 303 |
### Take only part
|
|
@@ -331,7 +333,7 @@ The query on the dataset card (output truncated):
|
|
| 331 |
|
| 332 |
```sql
|
| 333 |
SELECT source, collection, title, units, files, top_types, license.tier AS license_tier, needs_auth
|
| 334 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 335 |
WHERE kind = 'material_set' AND needs_auth = 'no' AND license.tier IN ('permissive', 'share-alike')
|
| 336 |
ORDER BY collection, units DESC;
|
| 337 |
```
|
|
@@ -350,7 +352,7 @@ Let DuckDB write the `--source` options, say for the disk image material sets wi
|
|
| 350 |
|
| 351 |
```sql
|
| 352 |
SELECT string_agg('--source ' || source, ' ' ORDER BY source) AS options
|
| 353 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 354 |
WHERE kind = 'material_set' AND collection = 'B'
|
| 355 |
AND license.tier = 'share-alike';
|
| 356 |
```
|
|
@@ -359,7 +361,7 @@ WHERE kind = 'material_set' AND collection = 'B'
|
|
| 359 |
--source adflib-adf-deleted --source adflib-adf-malformed --source btrfs-progs-convert-needs-recovery --source btrfs-progs-fsck-raw-images --source btrfs-progs-fuzz-images --source dfir-ntfs-fat-deleted --source dftt-deleted-files --source dftt-partition-tables --source dissect-hypervisor-disk-fixtures --source dosfstools-fsck-cases --source e2fsprogs-fsck-cases --source exfatprogs-fsck-cases --source libcdio-iso-malformed --source nist-cfreds-dfr-exfat --source ntfs-corrupted-images --source oe-ext4-hostile-fixtures --source qemu-iotests-sample-images
|
| 360 |
```
|
| 361 |
|
| 362 |
-
Put them after `fetch.py --revision release-2026-11-10.
|
| 363 |
Read only the catalog, not all of `meta/`.
|
| 364 |
|
| 365 |
### Take disk images and check them
|
|
@@ -371,8 +373,8 @@ Run "1 Take" (output truncated):
|
|
| 371 |

|
| 372 |
|
| 373 |
```sh
|
| 374 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 375 |
-
--revision release-2026-11-10.
|
| 376 |
--source dosfstools-fsck-cases \
|
| 377 |
--source e2fsprogs-fsck-cases \
|
| 378 |
--source exfatprogs-fsck-cases \
|
|
@@ -388,7 +390,7 @@ ok dosfstools-fsck-cases README.md: 3.68kB
|
|
| 388 |
ok exfatprogs-fsck-cases cases/bad_dentries/exfat.img: 5.24MB
|
| 389 |
22 files fetched and checked (15.9MB) into data; details in data/_meta/fetch_run.json
|
| 390 |
files of units: 15 ok; files of the sources themselves: 7 ok
|
| 391 |
-
run_id
|
| 392 |
```
|
| 393 |
|
| 394 |
An exFAT image is `exfat.img.tar.xz` upstream;
|
|
@@ -429,7 +431,7 @@ One, the catalog's `contamination.level` (values in the reference), say for mate
|
|
| 429 |
|
| 430 |
```sql
|
| 431 |
SELECT contamination.level AS level, count(*) AS material_sets, sum(units) AS units
|
| 432 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 433 |
WHERE kind = 'material_set'
|
| 434 |
AND list_has_any(top_types, ['docx', 'xlsx', 'pptx', 'pdf'])
|
| 435 |
GROUP BY ALL;
|
|
@@ -449,8 +451,8 @@ GROUP BY ALL;
|
|
| 449 |
Two, take the picked sources and read `removed by benchmark` and `removed by not compared`, printed before the first download (output truncated):
|
| 450 |
|
| 451 |
```sh
|
| 452 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 453 |
-
--revision release-2026-11-10.
|
| 454 |
--source python-docx-acceptance-test-files \
|
| 455 |
--source python-pptx-acceptance-test-files \
|
| 456 |
--source docling-docx-groundtruth \
|
|
@@ -479,7 +481,7 @@ Three, match file by file against the overlap table;
|
|
| 479 |
```sql
|
| 480 |
SELECT o.bench, o.hit_class, count(*) AS files
|
| 481 |
FROM read_json_auto('office3/_meta/fetch_manifest.jsonl') AS f
|
| 482 |
-
JOIN 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 483 |
ON o.source = f.source AND o.uri = f.uri
|
| 484 |
WHERE f.status = 'ok'
|
| 485 |
GROUP BY ALL;
|
|
@@ -511,7 +513,7 @@ to cite it:
|
|
| 511 |
@misc{agentic_env_index,
|
| 512 |
title = {agentic-env-index: an index of public materials for building agent environments},
|
| 513 |
howpublished = {https://huggingface.co/datasets/Battam/agentic-env-index},
|
| 514 |
-
note = {Release release-2026-11-10.
|
| 515 |
year = {2026}
|
| 516 |
}
|
| 517 |
```
|
|
@@ -593,7 +595,7 @@ Rows per class:
|
|
| 593 |
|
| 594 |
```sql
|
| 595 |
SELECT hit_class, scope, count(*) AS rows
|
| 596 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 597 |
GROUP BY ALL ORDER BY hit_class, scope;
|
| 598 |
```
|
| 599 |
|
|
|
|
| 8 |
Entry points: the data page https://huggingface.co/spaces/Battam/agentic-env-catalog (to share a state, use the page itself, https://battam-agentic-env-catalog.static.hf.space/index.html );
|
| 9 |
the dataset https://huggingface.co/datasets/Battam/agentic-env-index .
|
| 10 |
|
| 11 |
+
The numbers and command outputs below were filled in at deploy time, from the index commit `83dae072040e8688c08c84fb72fdc8690d0956e5` (release tag `release-2026-11-10.2`).
|
| 12 |
|
| 13 |
## Tutorial
|
| 14 |
|
|
|
|
| 87 |
### 5. Go to "Get the data" and keep the selection
|
| 88 |
|
| 89 |
**Do**: click "Get the data" at the top;
|
| 90 |
+
under "Keep this selection" click "Download the selection list (JSON)" and put `agentic-env-selection-83dae07.json` in your fetch folder.
|
| 91 |
|
| 92 |
**See**: the heading "Take 3 units from each of 52 material sets", then "Units per source", the card "1 Take", "What it writes", the six kinds "Left out by default", and "2 Take it again from the file", which reads the list.
|
| 93 |
The list records release, filters and sources:
|
|
|
|
| 96 |
{
|
| 97 |
"format": "agentic-env-selection/1",
|
| 98 |
"dataset": "Battam/agentic-env-index",
|
| 99 |
+
"revision": "release-2026-11-10.2",
|
| 100 |
"filters": {
|
| 101 |
"collection": "A",
|
| 102 |
"train": "yes",
|
|
|
|
| 120 |
**Do**: in the list's folder, run the command of "Take it again from the file" with `--sample 3` replaced by `--limit 220` (220 units in all, in a fixed order: a margin over 200, since some sites fail) and `--out ./data` by `--out ./take`:
|
| 121 |
|
| 122 |
```sh
|
| 123 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 124 |
+
--selection agentic-env-selection-83dae07.json \
|
| 125 |
+
--revision release-2026-11-10.2 \
|
| 126 |
--ext xlsx \
|
| 127 |
--ext xlsm \
|
| 128 |
--ext xls \
|
|
|
|
| 147 |
ok austender-ict-procurement-statistics workbooks/201314austenderictcontractstatistcs.xlsx: 26.3kB
|
| 148 |
ok austender-ict-procurement-statistics workbooks/2016-17-austender-ict-contract-data-category-sme-and-small-participation.xlsx: 32.9kB
|
| 149 |
...
|
| 150 |
+
317 files fetched and checked (96.4MB) into take; details in take/_meta/fetch_run.json
|
| 151 |
+
files of units: 5 failed, 215 ok; files of the sources themselves: 102 ok
|
| 152 |
+
run_id 20261110T233416.347081Z: 5 failed, 317 ok (lines in take/_meta/fetch_manifest.jsonl)
|
| 153 |
+
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('take/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '20261110T233416.347081Z' AND json_extract_string(json, 'status') <> 'ok'
|
| 154 |
```
|
| 155 |
|
| 156 |
Before the first download, `removed by` gives the units the default profile removed and the option that keeps them, counted in the data files read as far as `--limit` needed;
|
|
|
|
| 165 |
incomplete: 1 unit miss a file (the files fetched are kept; _meta/fetch_manifest.jsonl says why): ...
|
| 166 |
5 files fetched and checked (1.35MB) into probe; details in probe/_meta/fetch_run.json
|
| 167 |
files of units: 1 failed, 1 ok; files of the sources themselves: 4 ok
|
| 168 |
+
run_id 20261110T233732.458275Z: 1 failed, 5 ok (lines in probe/_meta/fetch_manifest.jsonl)
|
| 169 |
+
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('probe/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '20261110T233732.458275Z' AND json_extract_string(json, 'status') <> 'ok'
|
| 170 |
```
|
| 171 |
|
| 172 |
**Get**: every `ok` file matched its recorded size and hash.
|
|
|
|
| 206 |
โ excel_units โ
|
| 207 |
โ int64 โ
|
| 208 |
โโโโโโโโโโโโโโโค
|
| 209 |
+
โ 215 โ
|
| 210 |
โโโโโโโโโโโโโโโ
|
| 211 |
โโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโโ
|
| 212 |
โ verified โ files โ bytes โ
|
| 213 |
โ boolean โ int64 โ int128 โ
|
| 214 |
โโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโค
|
| 215 |
+
โ true โ 317 โ 96365587 โ
|
| 216 |
โโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโ
|
| 217 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโ
|
| 218 |
โ source โ reason โ files โ
|
| 219 |
โ varchar โ varchar โ int64 โ
|
| 220 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโค
|
| 221 |
+
โ scotland-government-workbooks โ size differs: expected 98409, got 0 โ 1 โ
|
| 222 |
+
โ scotland-government-workbooks โ size differs: expected 74961, got 0 โ 1 โ
|
| 223 |
โ victoria-government-workbooks โ https://www.planning.vic.gov.au/__data/assets โ 2 โ
|
| 224 |
โ western-australia-government-workbooks โ https://catalogue.data.wa.gov.au/dataset/03b1 โ 1 โ
|
| 225 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโ
|
|
|
|
| 241 |
**See** (truncated):
|
| 242 |
|
| 243 |
```text
|
| 244 |
+
Dataset commit 83dae072040e8688c08c84fb72fdc8690d0956e5: https://huggingface.co/datasets/Battam/agentic-env-index/tree/83dae072040e8688c08c84fb72fdc8690d0956e5
|
| 245 |
...
|
| 246 |
- Source: `austender-ict-procurement-statistics`
|
| 247 |
- License: CC-BY-3.0-AU
|
|
|
|
| 256 |
## How-to guides
|
| 257 |
|
| 258 |
One goal each;
|
| 259 |
+
commands are pinned to `release-2026-11-10.2`.
|
| 260 |
|
| 261 |
### Filter by file type and license
|
| 262 |
|
|
|
|
| 286 |
|
| 287 |
### Pin a release
|
| 288 |
|
| 289 |
+
The page's commands pin the script URL and `--revision` to `release-2026-11-10.2`;
|
| 290 |
the script's first line names the commit:
|
| 291 |
|
| 292 |
```text
|
| 293 |
+
reading hf://datasets/Battam/agentic-env-index@83dae072040e8688c08c84fb72fdc8690d0956e5 (commit 83dae072040e8688c08c84fb72fdc8690d0956e5); pass --revision 83dae072040e8688c08c84fb72fdc8690d0956e5 to select the same rows later
|
| 294 |
```
|
| 295 |
|
| 296 |
Later, `--revision <commit>`, the same tag or the selection list take the same files;
|
|
|
|
| 299 |
A misspelt source name stops with exit code 2:
|
| 300 |
|
| 301 |
```text
|
| 302 |
+
error: --source dosfstools-fsk: no such source in meta/catalog.parquet at commit 83dae072040e8688c08c84fb72fdc8690d0956e5; closest: dosfstools-fsck-cases
|
| 303 |
```
|
| 304 |
|
| 305 |
### Take only part
|
|
|
|
| 333 |
|
| 334 |
```sql
|
| 335 |
SELECT source, collection, title, units, files, top_types, license.tier AS license_tier, needs_auth
|
| 336 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 337 |
WHERE kind = 'material_set' AND needs_auth = 'no' AND license.tier IN ('permissive', 'share-alike')
|
| 338 |
ORDER BY collection, units DESC;
|
| 339 |
```
|
|
|
|
| 352 |
|
| 353 |
```sql
|
| 354 |
SELECT string_agg('--source ' || source, ' ' ORDER BY source) AS options
|
| 355 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 356 |
WHERE kind = 'material_set' AND collection = 'B'
|
| 357 |
AND license.tier = 'share-alike';
|
| 358 |
```
|
|
|
|
| 361 |
--source adflib-adf-deleted --source adflib-adf-malformed --source btrfs-progs-convert-needs-recovery --source btrfs-progs-fsck-raw-images --source btrfs-progs-fuzz-images --source dfir-ntfs-fat-deleted --source dftt-deleted-files --source dftt-partition-tables --source dissect-hypervisor-disk-fixtures --source dosfstools-fsck-cases --source e2fsprogs-fsck-cases --source exfatprogs-fsck-cases --source libcdio-iso-malformed --source nist-cfreds-dfr-exfat --source ntfs-corrupted-images --source oe-ext4-hostile-fixtures --source qemu-iotests-sample-images
|
| 362 |
```
|
| 363 |
|
| 364 |
+
Put them after `fetch.py --revision release-2026-11-10.2`.
|
| 365 |
Read only the catalog, not all of `meta/`.
|
| 366 |
|
| 367 |
### Take disk images and check them
|
|
|
|
| 373 |

|
| 374 |
|
| 375 |
```sh
|
| 376 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 377 |
+
--revision release-2026-11-10.2 \
|
| 378 |
--source dosfstools-fsck-cases \
|
| 379 |
--source e2fsprogs-fsck-cases \
|
| 380 |
--source exfatprogs-fsck-cases \
|
|
|
|
| 390 |
ok exfatprogs-fsck-cases cases/bad_dentries/exfat.img: 5.24MB
|
| 391 |
22 files fetched and checked (15.9MB) into data; details in data/_meta/fetch_run.json
|
| 392 |
files of units: 15 ok; files of the sources themselves: 7 ok
|
| 393 |
+
run_id 20261110T233737.867574Z: 22 ok (lines in data/_meta/fetch_manifest.jsonl)
|
| 394 |
```
|
| 395 |
|
| 396 |
An exFAT image is `exfat.img.tar.xz` upstream;
|
|
|
|
| 431 |
|
| 432 |
```sql
|
| 433 |
SELECT contamination.level AS level, count(*) AS material_sets, sum(units) AS units
|
| 434 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 435 |
WHERE kind = 'material_set'
|
| 436 |
AND list_has_any(top_types, ['docx', 'xlsx', 'pptx', 'pdf'])
|
| 437 |
GROUP BY ALL;
|
|
|
|
| 451 |
Two, take the picked sources and read `removed by benchmark` and `removed by not compared`, printed before the first download (output truncated):
|
| 452 |
|
| 453 |
```sh
|
| 454 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 455 |
+
--revision release-2026-11-10.2 \
|
| 456 |
--source python-docx-acceptance-test-files \
|
| 457 |
--source python-pptx-acceptance-test-files \
|
| 458 |
--source docling-docx-groundtruth \
|
|
|
|
| 481 |
```sql
|
| 482 |
SELECT o.bench, o.hit_class, count(*) AS files
|
| 483 |
FROM read_json_auto('office3/_meta/fetch_manifest.jsonl') AS f
|
| 484 |
+
JOIN 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/bench_overlap.parquet' AS o
|
| 485 |
ON o.source = f.source AND o.uri = f.uri
|
| 486 |
WHERE f.status = 'ok'
|
| 487 |
GROUP BY ALL;
|
|
|
|
| 513 |
@misc{agentic_env_index,
|
| 514 |
title = {agentic-env-index: an index of public materials for building agent environments},
|
| 515 |
howpublished = {https://huggingface.co/datasets/Battam/agentic-env-index},
|
| 516 |
+
note = {Release release-2026-11-10.2},
|
| 517 |
year = {2026}
|
| 518 |
}
|
| 519 |
```
|
|
|
|
| 595 |
|
| 596 |
```sql
|
| 597 |
SELECT hit_class, scope, count(*) AS rows
|
| 598 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/bench_overlap.parquet'
|
| 599 |
GROUP BY ALL ORDER BY hit_class, scope;
|
| 600 |
```
|
| 601 |
|
manual/manual.zh.md
CHANGED
|
@@ -8,7 +8,7 @@
|
|
| 8 |
ๅ
ฅๅฃ๏ผๆฐๆฎ้กต https://huggingface.co/spaces/Battam/agentic-env-catalog ๏ผๅธฆๅๆฐๅไบซๆถ็จ้กต้ขๆฌไฝ https://battam-agentic-env-catalog.static.hf.space/index.html ๏ผ๏ผ
|
| 9 |
ๆฐๆฎ้ https://huggingface.co/datasets/Battam/agentic-env-index ใ
|
| 10 |
|
| 11 |
-
ไธ้ข็ๆฐๅญๅๅฝไปค่พๅบๆฏ้จ็ฝฒๆถๅกซๅ
ฅ็๏ผๅ่ช็ดขๅผๆไบค `
|
| 12 |
|
| 13 |
## ๆ็จ
|
| 14 |
|
|
@@ -83,7 +83,7 @@
|
|
| 83 |
|
| 84 |
### 5. ๅฐใ่ทๅๆฐๆฎใ๏ผไฟๅญ้ๆฉๆธ
ๅ
|
| 85 |
|
| 86 |
-
**ๅ**๏ผ็น้กถ้จใ่ทๅๆฐๆฎใ๏ผๅจใไฟๅญ่ฟๆฌก้ๆฉใ้็นใไธ่ฝฝ้ๆฉๆธ
ๅ๏ผJSON๏ผใ๏ผๆ `agentic-env-selection-
|
| 87 |
|
| 88 |
**็ๅฐ**๏ผๆ ้ขใไป 52 ไธชๆๆ้ไธญๅ่ทๅ 3 ไธชๅๅ
ใ๏ผไธ้ขๆฏใๆฏไธชๆฅๆบๅใใๅฝไปคๅกใ1 ่ทๅใใใๅๅ
ฅๅ
ๅฎนใใใ้ป่ฎคๆ้คใๅ
ญ็ฑปๅ่ฏป้ๆฉๆธ
ๅ็ใ2 ๆๆไปถๅๅไธๆฌกใใ
|
| 89 |
ๆธ
ๅ่ฎฐ็็ๆฌใ็ญ้ๅๆฅๆบ๏ผ
|
|
@@ -92,7 +92,7 @@
|
|
| 92 |
{
|
| 93 |
"format": "agentic-env-selection/1",
|
| 94 |
"dataset": "Battam/agentic-env-index",
|
| 95 |
-
"revision": "release-2026-11-10.
|
| 96 |
"filters": {
|
| 97 |
"collection": "A",
|
| 98 |
"train": "yes",
|
|
@@ -116,9 +116,9 @@
|
|
| 116 |
**ๅ**๏ผๅจๆธ
ๅๆๅจ็ฎๅฝ่ทใๆๆไปถๅๅไธๆฌกใ้ฃๆกๅฝไปค๏ผ`--sample 3` ๆขๆ `--limit 220`๏ผไธๅ
ฑ 220 ไธชๅๅ
๏ผๅบๅฎ้กบๅบ๏ผๆฏ 200 ๅคๅไธไบ็ไฝ้๏ผๆ็ๅ็ซไผๅคฑๆ๏ผ๏ผ`--out ./data` ๆขๆ `--out ./take`๏ผ
|
| 117 |
|
| 118 |
```sh
|
| 119 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 120 |
-
--selection agentic-env-selection-
|
| 121 |
-
--revision release-2026-11-10.
|
| 122 |
--ext xlsx \
|
| 123 |
--ext xlsm \
|
| 124 |
--ext xls \
|
|
@@ -143,10 +143,10 @@ ok abhishekumarr00-supply-chain-mastery workbooks/abhishekumarr00-supply-chain-m
|
|
| 143 |
ok austender-ict-procurement-statistics workbooks/201314austenderictcontractstatistcs.xlsx: 26.3kB
|
| 144 |
ok austender-ict-procurement-statistics workbooks/2016-17-austender-ict-contract-data-category-sme-and-small-participation.xlsx: 32.9kB
|
| 145 |
...
|
| 146 |
-
|
| 147 |
-
files of units:
|
| 148 |
-
run_id
|
| 149 |
-
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('take/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '
|
| 150 |
```
|
| 151 |
|
| 152 |
็ฌฌไธๆฌกไธ่ฝฝไนๅ๏ผ`removed by` ๆฏ้ป่ฎค้
็ฝฎๅปๆ็ๅๅ
ๅไฟ็ๅฎไปฌ็ๅผๅ
ณ๏ผๆฐ็ๆฏ `--limit` ้่ฆ่ฏปๅฐ็้ฃไบๆฐๆฎๆไปถ๏ผ
|
|
@@ -161,8 +161,8 @@ failed western-australia-government-workbooks workbooks/wa-gross-regional-produc
|
|
| 161 |
incomplete: 1 unit miss a file (the files fetched are kept; _meta/fetch_manifest.jsonl says why): ...
|
| 162 |
5 files fetched and checked (1.35MB) into probe; details in probe/_meta/fetch_run.json
|
| 163 |
files of units: 1 failed, 1 ok; files of the sources themselves: 4 ok
|
| 164 |
-
run_id
|
| 165 |
-
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('probe/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '
|
| 166 |
```
|
| 167 |
|
| 168 |
**ๅพๅฐ**๏ผ`ok` ็ๆไปถๅทฒๆ่ฎฐๅฝ็ๅคงๅฐๅๅๅธๆ ธๅฏนใ
|
|
@@ -201,18 +201,20 @@ GROUP BY ALL ORDER BY source;
|
|
| 201 |
โ excel_units โ
|
| 202 |
โ int64 โ
|
| 203 |
โโโโโโโโโโโโโโโค
|
| 204 |
-
โ
|
| 205 |
โโโโโโโโโโโโโโโ
|
| 206 |
โโโโโโโโโโโโฌโโโโโโโโฌโ๏ฟฝ๏ฟฝ๏ฟฝโโโโโโโโโ
|
| 207 |
โ verified โ files โ bytes โ
|
| 208 |
โ boolean โ int64 โ int128 โ
|
| 209 |
โโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโค
|
| 210 |
-
โ true โ
|
| 211 |
โโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโ
|
| 212 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโ
|
| 213 |
โ source โ reason โ files โ
|
| 214 |
โ varchar โ varchar โ int64 โ
|
| 215 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโค
|
|
|
|
|
|
|
| 216 |
โ victoria-government-workbooks โ https://www.planning.vic.gov.au/__data/assets โ 2 โ
|
| 217 |
โ western-australia-government-workbooks โ https://catalogue.data.wa.gov.au/dataset/03b1 โ 1 โ
|
| 218 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโ
|
|
@@ -233,7 +235,7 @@ GROUP BY ALL ORDER BY source;
|
|
| 233 |
**็ๅฐ**๏ผๆช็ญ๏ผ๏ผ
|
| 234 |
|
| 235 |
```text
|
| 236 |
-
Dataset commit
|
| 237 |
...
|
| 238 |
- Source: `austender-ict-procurement-statistics`
|
| 239 |
- License: CC-BY-3.0-AU
|
|
@@ -247,7 +249,7 @@ Dataset commit 5b9f5a4d2bde4bee82f6a0b178b5ab088603564f: https://huggingface.co/
|
|
| 247 |
|
| 248 |
## ๆไฝๆๅ
|
| 249 |
|
| 250 |
-
ๆฏๆกไธไธช็ฎๆ ๏ผๅฝไปค้ฝ้ๅจ `release-2026-11-10.
|
| 251 |
|
| 252 |
### ๆๆไปถ็ฑปๅไธ่ฎธๅฏ็ญ
|
| 253 |
|
|
@@ -275,10 +277,10 @@ DuckDB ้็จ็ฎๅฝ่กจ็ `license.tier` ๅ `top_types`๏ผๆๅธธ่ง็ๆฉๅฑๅ
|
|
| 275 |
|
| 276 |
### ้ไฝๅๅธ็ๆฌๅ
|
| 277 |
|
| 278 |
-
้กต้ข็ป็ๅฝไปคๆ่ๆฌ็ฝๅๅ `--revision` ้ฝ้ๅจ `release-2026-11-10.
|
| 279 |
|
| 280 |
```text
|
| 281 |
-
reading hf://datasets/Battam/agentic-env-index@
|
| 282 |
```
|
| 283 |
|
| 284 |
ๆฅๅ็จ `--revision <ๆไบคๅท>`ใๅไธไธชๆ ็ญพๆ้ๆฉๆธ
ๅ๏ผๅๅฐ็ๅฐฑๆฏๅๆ ท็ๆไปถ๏ผ
|
|
@@ -287,7 +289,7 @@ reading hf://datasets/Battam/agentic-env-index@5b9f5a4d2bde4bee82f6a0b178b5ab088
|
|
| 287 |
ๆฅๆบๅๅ้ๆถ้ๅบ็ 2๏ผ
|
| 288 |
|
| 289 |
```text
|
| 290 |
-
error: --source dosfstools-fsk: no such source in meta/catalog.parquet at commit
|
| 291 |
```
|
| 292 |
|
| 293 |
### ๅชๅไธ้จๅ
|
|
@@ -319,7 +321,7 @@ error: --source dosfstools-fsk: no such source in meta/catalog.parquet at commit
|
|
| 319 |
|
| 320 |
```sql
|
| 321 |
SELECT source, collection, title, units, files, top_types, license.tier AS license_tier, needs_auth
|
| 322 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 323 |
WHERE kind = 'material_set' AND needs_auth = 'no' AND license.tier IN ('permissive', 'share-alike')
|
| 324 |
ORDER BY collection, units DESC;
|
| 325 |
```
|
|
@@ -338,7 +340,7 @@ ORDER BY collection, units DESC;
|
|
| 338 |
|
| 339 |
```sql
|
| 340 |
SELECT string_agg('--source ' || source, ' ' ORDER BY source) AS options
|
| 341 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 342 |
WHERE kind = 'material_set' AND collection = 'B'
|
| 343 |
AND license.tier = 'share-alike';
|
| 344 |
```
|
|
@@ -347,7 +349,7 @@ WHERE kind = 'material_set' AND collection = 'B'
|
|
| 347 |
--source adflib-adf-deleted --source adflib-adf-malformed --source btrfs-progs-convert-needs-recovery --source btrfs-progs-fsck-raw-images --source btrfs-progs-fuzz-images --source dfir-ntfs-fat-deleted --source dftt-deleted-files --source dftt-partition-tables --source dissect-hypervisor-disk-fixtures --source dosfstools-fsck-cases --source e2fsprogs-fsck-cases --source exfatprogs-fsck-cases --source libcdio-iso-malformed --source nist-cfreds-dfr-exfat --source ntfs-corrupted-images --source oe-ext4-hostile-fixtures --source qemu-iotests-sample-images
|
| 348 |
```
|
| 349 |
|
| 350 |
-
ๆฅๅจ `fetch.py --revision release-2026-11-10.
|
| 351 |
ๅช่ฏป็ฎๅฝ่กจ๏ผๅซๆดไธชไธ่ฝฝ `meta/`ใ
|
| 352 |
|
| 353 |
### ๅ็ฃ็้ๅ๏ผๅๅๅๆ ธๅฏนๆฒกๅ
|
|
@@ -359,8 +361,8 @@ WHERE kind = 'material_set' AND collection = 'B'
|
|
| 359 |

|
| 360 |
|
| 361 |
```sh
|
| 362 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 363 |
-
--revision release-2026-11-10.
|
| 364 |
--source dosfstools-fsck-cases \
|
| 365 |
--source e2fsprogs-fsck-cases \
|
| 366 |
--source exfatprogs-fsck-cases \
|
|
@@ -376,7 +378,7 @@ ok dosfstools-fsck-cases README.md: 3.68kB
|
|
| 376 |
ok exfatprogs-fsck-cases cases/bad_dentries/exfat.img: 5.24MB
|
| 377 |
22 files fetched and checked (15.9MB) into data; details in data/_meta/fetch_run.json
|
| 378 |
files of units: 15 ok; files of the sources themselves: 7 ok
|
| 379 |
-
run_id
|
| 380 |
```
|
| 381 |
|
| 382 |
exFAT ้ๅๅจๅ็ซๆฏ `exfat.img.tar.xz`๏ผ่ๆฌ่งฃๅบๅๅๆๅๅ
่ชๅทฑ็ SHA-256 ๆ ธๅฏน๏ผ็ไธๅช็้ๅใ
|
|
@@ -416,7 +418,7 @@ Pass 1: Checking inodes, blocks, and sizes
|
|
| 416 |
|
| 417 |
```sql
|
| 418 |
SELECT contamination.level AS level, count(*) AS material_sets, sum(units) AS units
|
| 419 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 420 |
WHERE kind = 'material_set'
|
| 421 |
AND list_has_any(top_types, ['docx', 'xlsx', 'pptx', 'pdf'])
|
| 422 |
GROUP BY ALL;
|
|
@@ -436,8 +438,8 @@ GROUP BY ALL;
|
|
| 436 |
ไบ๏ผๅๅๆไธญ็ๆฅๆบ๏ผ็็ฌฌไธๆฌกไธ่ฝฝไนๅๆๅฐ็ `removed by benchmark` ๅ `removed by not compared`๏ผ่พๅบๆช็ญ๏ผ๏ผ
|
| 437 |
|
| 438 |
```sh
|
| 439 |
-
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.
|
| 440 |
-
--revision release-2026-11-10.
|
| 441 |
--source python-docx-acceptance-test-files \
|
| 442 |
--source python-pptx-acceptance-test-files \
|
| 443 |
--source docling-docx-groundtruth \
|
|
@@ -465,7 +467,7 @@ fetching: 41 units: 45 files (4 files of the sources themselves among them), 25.
|
|
| 465 |
```sql
|
| 466 |
SELECT o.bench, o.hit_class, count(*) AS files
|
| 467 |
FROM read_json_auto('office3/_meta/fetch_manifest.jsonl') AS f
|
| 468 |
-
JOIN 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 469 |
ON o.source = f.source AND o.uri = f.uri
|
| 470 |
WHERE f.status = 'ok'
|
| 471 |
GROUP BY ALL;
|
|
@@ -497,7 +499,7 @@ GDPval ๅจๆฏๅฏนๆธ
ๅ `meta/benchmarks.parquet` ้๏ผ220 ้้ข๏ผ๏ผ่ฟไธ็
|
|
| 497 |
@misc{agentic_env_index,
|
| 498 |
title = {agentic-env-index: an index of public materials for building agent environments},
|
| 499 |
howpublished = {https://huggingface.co/datasets/Battam/agentic-env-index},
|
| 500 |
-
note = {Release release-2026-11-10.
|
| 501 |
year = {2026}
|
| 502 |
}
|
| 503 |
```
|
|
@@ -579,7 +581,7 @@ scope ไธบ `container` ๆถๅฝไธญๆดไธชๅ็ป๏ผๅชๅจๅ็ปๆฏไธไธชๅฎๆดๅไฝ
|
|
| 579 |
|
| 580 |
```sql
|
| 581 |
SELECT hit_class, scope, count(*) AS rows
|
| 582 |
-
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.
|
| 583 |
GROUP BY ALL ORDER BY hit_class, scope;
|
| 584 |
```
|
| 585 |
|
|
|
|
| 8 |
ๅ
ฅๅฃ๏ผๆฐๆฎ้กต https://huggingface.co/spaces/Battam/agentic-env-catalog ๏ผๅธฆๅๆฐๅไบซๆถ็จ้กต้ขๆฌไฝ https://battam-agentic-env-catalog.static.hf.space/index.html ๏ผ๏ผ
|
| 9 |
ๆฐๆฎ้ https://huggingface.co/datasets/Battam/agentic-env-index ใ
|
| 10 |
|
| 11 |
+
ไธ้ข็ๆฐๅญๅๅฝไปค่พๅบๆฏ้จ็ฝฒๆถๅกซๅ
ฅ็๏ผๅ่ช็ดขๅผๆไบค `83dae072040e8688c08c84fb72fdc8690d0956e5`๏ผๅๅธๆ ็ญพ `release-2026-11-10.2`๏ผใ
|
| 12 |
|
| 13 |
## ๆ็จ
|
| 14 |
|
|
|
|
| 83 |
|
| 84 |
### 5. ๅฐใ่ทๅๆฐๆฎใ๏ผไฟๅญ้ๆฉๆธ
ๅ
|
| 85 |
|
| 86 |
+
**ๅ**๏ผ็น้กถ้จใ่ทๅๆฐๆฎใ๏ผๅจใไฟๅญ่ฟๆฌก้ๆฉใ้็นใไธ่ฝฝ้ๆฉๆธ
ๅ๏ผJSON๏ผใ๏ผๆ `agentic-env-selection-83dae07.json` ๆพ่ฟๅไปถ็ฎๅฝใ
|
| 87 |
|
| 88 |
**็ๅฐ**๏ผๆ ้ขใไป 52 ไธชๆๆ้ไธญๅ่ทๅ 3 ไธชๅๅ
ใ๏ผไธ้ขๆฏใๆฏไธชๆฅๆบๅใใๅฝไปคๅกใ1 ่ทๅใใใๅๅ
ฅๅ
ๅฎนใใใ้ป่ฎคๆ้คใๅ
ญ็ฑปๅ่ฏป้ๆฉๆธ
ๅ็ใ2 ๆๆไปถๅๅไธๆฌกใใ
|
| 89 |
ๆธ
ๅ่ฎฐ็็ๆฌใ็ญ้ๅๆฅๆบ๏ผ
|
|
|
|
| 92 |
{
|
| 93 |
"format": "agentic-env-selection/1",
|
| 94 |
"dataset": "Battam/agentic-env-index",
|
| 95 |
+
"revision": "release-2026-11-10.2",
|
| 96 |
"filters": {
|
| 97 |
"collection": "A",
|
| 98 |
"train": "yes",
|
|
|
|
| 116 |
**ๅ**๏ผๅจๆธ
ๅๆๅจ็ฎๅฝ่ทใๆๆไปถๅๅไธๆฌกใ้ฃๆกๅฝไปค๏ผ`--sample 3` ๆขๆ `--limit 220`๏ผไธๅ
ฑ 220 ไธชๅๅ
๏ผๅบๅฎ้กบๅบ๏ผๆฏ 200 ๅคๅไธไบ็ไฝ้๏ผๆ็ๅ็ซไผๅคฑๆ๏ผ๏ผ`--out ./data` ๆขๆ `--out ./take`๏ผ
|
| 117 |
|
| 118 |
```sh
|
| 119 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 120 |
+
--selection agentic-env-selection-83dae07.json \
|
| 121 |
+
--revision release-2026-11-10.2 \
|
| 122 |
--ext xlsx \
|
| 123 |
--ext xlsm \
|
| 124 |
--ext xls \
|
|
|
|
| 143 |
ok austender-ict-procurement-statistics workbooks/201314austenderictcontractstatistcs.xlsx: 26.3kB
|
| 144 |
ok austender-ict-procurement-statistics workbooks/2016-17-austender-ict-contract-data-category-sme-and-small-participation.xlsx: 32.9kB
|
| 145 |
...
|
| 146 |
+
317 files fetched and checked (96.4MB) into take; details in take/_meta/fetch_run.json
|
| 147 |
+
files of units: 5 failed, 215 ok; files of the sources themselves: 102 ok
|
| 148 |
+
run_id 20261110T233416.347081Z: 5 failed, 317 ok (lines in take/_meta/fetch_manifest.jsonl)
|
| 149 |
+
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('take/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '20261110T233416.347081Z' AND json_extract_string(json, 'status') <> 'ok'
|
| 150 |
```
|
| 151 |
|
| 152 |
็ฌฌไธๆฌกไธ่ฝฝไนๅ๏ผ`removed by` ๆฏ้ป่ฎค้
็ฝฎๅปๆ็ๅๅ
ๅไฟ็ๅฎไปฌ็ๅผๅ
ณ๏ผๆฐ็ๆฏ `--limit` ้่ฆ่ฏปๅฐ็้ฃไบๆฐๆฎๆไปถ๏ผ
|
|
|
|
| 161 |
incomplete: 1 unit miss a file (the files fetched are kept; _meta/fetch_manifest.jsonl says why): ...
|
| 162 |
5 files fetched and checked (1.35MB) into probe; details in probe/_meta/fetch_run.json
|
| 163 |
files of units: 1 failed, 1 ok; files of the sources themselves: 4 ok
|
| 164 |
+
run_id 20261110T233732.458275Z: 1 failed, 5 ok (lines in probe/_meta/fetch_manifest.jsonl)
|
| 165 |
+
its lines that are not ok, in DuckDB: SELECT json FROM read_json_objects('probe/_meta/fetch_manifest.jsonl') WHERE json_extract_string(json, 'run_id') = '20261110T233732.458275Z' AND json_extract_string(json, 'status') <> 'ok'
|
| 166 |
```
|
| 167 |
|
| 168 |
**ๅพๅฐ**๏ผ`ok` ็ๆไปถๅทฒๆ่ฎฐๅฝ็ๅคงๅฐๅๅๅธๆ ธๅฏนใ
|
|
|
|
| 201 |
โ excel_units โ
|
| 202 |
โ int64 โ
|
| 203 |
โโโโโโโโโโโโโโโค
|
| 204 |
+
โ 215 โ
|
| 205 |
โโโโโโโโโโโโโโโ
|
| 206 |
โโโโโโโโโโโโฌโโโโโโโโฌโ๏ฟฝ๏ฟฝ๏ฟฝโโโโโโโโโ
|
| 207 |
โ verified โ files โ bytes โ
|
| 208 |
โ boolean โ int64 โ int128 โ
|
| 209 |
โโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโโค
|
| 210 |
+
โ true โ 317 โ 96365587 โ
|
| 211 |
โโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโ
|
| 212 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโ
|
| 213 |
โ source โ reason โ files โ
|
| 214 |
โ varchar โ varchar โ int64 โ
|
| 215 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโค
|
| 216 |
+
โ scotland-government-workbooks โ size differs: expected 98409, got 0 โ 1 โ
|
| 217 |
+
โ scotland-government-workbooks โ size differs: expected 74961, got 0 โ 1 โ
|
| 218 |
โ victoria-government-workbooks โ https://www.planning.vic.gov.au/__data/assets โ 2 โ
|
| 219 |
โ western-australia-government-workbooks โ https://catalogue.data.wa.gov.au/dataset/03b1 โ 1 โ
|
| 220 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโ
|
|
|
|
| 235 |
**็ๅฐ**๏ผๆช็ญ๏ผ๏ผ
|
| 236 |
|
| 237 |
```text
|
| 238 |
+
Dataset commit 83dae072040e8688c08c84fb72fdc8690d0956e5: https://huggingface.co/datasets/Battam/agentic-env-index/tree/83dae072040e8688c08c84fb72fdc8690d0956e5
|
| 239 |
...
|
| 240 |
- Source: `austender-ict-procurement-statistics`
|
| 241 |
- License: CC-BY-3.0-AU
|
|
|
|
| 249 |
|
| 250 |
## ๆไฝๆๅ
|
| 251 |
|
| 252 |
+
ๆฏๆกไธไธช็ฎๆ ๏ผๅฝไปค้ฝ้ๅจ `release-2026-11-10.2`ใ
|
| 253 |
|
| 254 |
### ๆๆไปถ็ฑปๅไธ่ฎธๅฏ็ญ
|
| 255 |
|
|
|
|
| 277 |
|
| 278 |
### ้ไฝๅๅธ็ๆฌๅ
|
| 279 |
|
| 280 |
+
้กต้ข็ป็ๅฝไปคๆ่ๆฌ็ฝๅๅ `--revision` ้ฝ้ๅจ `release-2026-11-10.2`๏ผ่ๆฌ้ฆ่กๆๅฐๅฏนๅบ็ๆไบค๏ผ
|
| 281 |
|
| 282 |
```text
|
| 283 |
+
reading hf://datasets/Battam/agentic-env-index@83dae072040e8688c08c84fb72fdc8690d0956e5 (commit 83dae072040e8688c08c84fb72fdc8690d0956e5); pass --revision 83dae072040e8688c08c84fb72fdc8690d0956e5 to select the same rows later
|
| 284 |
```
|
| 285 |
|
| 286 |
ๆฅๅ็จ `--revision <ๆไบคๅท>`ใๅไธไธชๆ ็ญพๆ้ๆฉๆธ
ๅ๏ผๅๅฐ็ๅฐฑๆฏๅๆ ท็ๆไปถ๏ผ
|
|
|
|
| 289 |
ๆฅๆบๅๅ้ๆถ้ๅบ็ 2๏ผ
|
| 290 |
|
| 291 |
```text
|
| 292 |
+
error: --source dosfstools-fsk: no such source in meta/catalog.parquet at commit 83dae072040e8688c08c84fb72fdc8690d0956e5; closest: dosfstools-fsck-cases
|
| 293 |
```
|
| 294 |
|
| 295 |
### ๅชๅไธ้จๅ
|
|
|
|
| 321 |
|
| 322 |
```sql
|
| 323 |
SELECT source, collection, title, units, files, top_types, license.tier AS license_tier, needs_auth
|
| 324 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 325 |
WHERE kind = 'material_set' AND needs_auth = 'no' AND license.tier IN ('permissive', 'share-alike')
|
| 326 |
ORDER BY collection, units DESC;
|
| 327 |
```
|
|
|
|
| 340 |
|
| 341 |
```sql
|
| 342 |
SELECT string_agg('--source ' || source, ' ' ORDER BY source) AS options
|
| 343 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 344 |
WHERE kind = 'material_set' AND collection = 'B'
|
| 345 |
AND license.tier = 'share-alike';
|
| 346 |
```
|
|
|
|
| 349 |
--source adflib-adf-deleted --source adflib-adf-malformed --source btrfs-progs-convert-needs-recovery --source btrfs-progs-fsck-raw-images --source btrfs-progs-fuzz-images --source dfir-ntfs-fat-deleted --source dftt-deleted-files --source dftt-partition-tables --source dissect-hypervisor-disk-fixtures --source dosfstools-fsck-cases --source e2fsprogs-fsck-cases --source exfatprogs-fsck-cases --source libcdio-iso-malformed --source nist-cfreds-dfr-exfat --source ntfs-corrupted-images --source oe-ext4-hostile-fixtures --source qemu-iotests-sample-images
|
| 350 |
```
|
| 351 |
|
| 352 |
+
ๆฅๅจ `fetch.py --revision release-2026-11-10.2` ๅ้ขใ
|
| 353 |
ๅช่ฏป็ฎๅฝ่กจ๏ผๅซๆดไธชไธ่ฝฝ `meta/`ใ
|
| 354 |
|
| 355 |
### ๅ็ฃ็้ๅ๏ผๅๅๅๆ ธๅฏนๆฒกๅ
|
|
|
|
| 361 |

|
| 362 |
|
| 363 |
```sh
|
| 364 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 365 |
+
--revision release-2026-11-10.2 \
|
| 366 |
--source dosfstools-fsck-cases \
|
| 367 |
--source e2fsprogs-fsck-cases \
|
| 368 |
--source exfatprogs-fsck-cases \
|
|
|
|
| 378 |
ok exfatprogs-fsck-cases cases/bad_dentries/exfat.img: 5.24MB
|
| 379 |
22 files fetched and checked (15.9MB) into data; details in data/_meta/fetch_run.json
|
| 380 |
files of units: 15 ok; files of the sources themselves: 7 ok
|
| 381 |
+
run_id 20261110T233737.867574Z: 22 ok (lines in data/_meta/fetch_manifest.jsonl)
|
| 382 |
```
|
| 383 |
|
| 384 |
exFAT ้ๅๅจๅ็ซๆฏ `exfat.img.tar.xz`๏ผ่ๆฌ่งฃๅบๅๅๆๅๅ
่ชๅทฑ็ SHA-256 ๆ ธๅฏน๏ผ็ไธๅช็้ๅใ
|
|
|
|
| 418 |
|
| 419 |
```sql
|
| 420 |
SELECT contamination.level AS level, count(*) AS material_sets, sum(units) AS units
|
| 421 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/catalog.parquet'
|
| 422 |
WHERE kind = 'material_set'
|
| 423 |
AND list_has_any(top_types, ['docx', 'xlsx', 'pptx', 'pdf'])
|
| 424 |
GROUP BY ALL;
|
|
|
|
| 438 |
ไบ๏ผๅๅๆไธญ็ๆฅๆบ๏ผ็็ฌฌไธๆฌกไธ่ฝฝไนๅๆๅฐ็ `removed by benchmark` ๅ `removed by not compared`๏ผ่พๅบๆช็ญ๏ผ๏ผ
|
| 439 |
|
| 440 |
```sh
|
| 441 |
+
uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/release-2026-11-10.2/tools/fetch.py \
|
| 442 |
+
--revision release-2026-11-10.2 \
|
| 443 |
--source python-docx-acceptance-test-files \
|
| 444 |
--source python-pptx-acceptance-test-files \
|
| 445 |
--source docling-docx-groundtruth \
|
|
|
|
| 467 |
```sql
|
| 468 |
SELECT o.bench, o.hit_class, count(*) AS files
|
| 469 |
FROM read_json_auto('office3/_meta/fetch_manifest.jsonl') AS f
|
| 470 |
+
JOIN 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/bench_overlap.parquet' AS o
|
| 471 |
ON o.source = f.source AND o.uri = f.uri
|
| 472 |
WHERE f.status = 'ok'
|
| 473 |
GROUP BY ALL;
|
|
|
|
| 499 |
@misc{agentic_env_index,
|
| 500 |
title = {agentic-env-index: an index of public materials for building agent environments},
|
| 501 |
howpublished = {https://huggingface.co/datasets/Battam/agentic-env-index},
|
| 502 |
+
note = {Release release-2026-11-10.2},
|
| 503 |
year = {2026}
|
| 504 |
}
|
| 505 |
```
|
|
|
|
| 581 |
|
| 582 |
```sql
|
| 583 |
SELECT hit_class, scope, count(*) AS rows
|
| 584 |
+
FROM 'hf://datasets/Battam/agentic-env-index@release-2026-11-10.2/meta/bench_overlap.parquet'
|
| 585 |
GROUP BY ALL ORDER BY hit_class, scope;
|
| 586 |
```
|
| 587 |
|
portal.json
CHANGED
|
@@ -1,8 +1,9 @@
|
|
| 1 |
{
|
| 2 |
-
"commit": "
|
| 3 |
-
"digest": "
|
| 4 |
"files": 19,
|
| 5 |
"format": "portal-deploy/1",
|
| 6 |
-
"
|
|
|
|
| 7 |
"stages": "70a0fac1235edf355cc4272613f43bb75be76c9759c43f8dc68cb254cea303cf"
|
| 8 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"commit": "a015777686bf1cedb9df5ff6afd17d43fff21610",
|
| 3 |
+
"digest": "63a115cc1a819d10fc6a572ea53d2b8ee4aa63e824ef957d062826b8892bf2ff",
|
| 4 |
"files": 19,
|
| 5 |
"format": "portal-deploy/1",
|
| 6 |
+
"funnel": "83dae072040e8688c08c84fb72fdc8690d0956e5",
|
| 7 |
+
"index": "83dae072040e8688c08c84fb72fdc8690d0956e5",
|
| 8 |
"stages": "70a0fac1235edf355cc4272613f43bb75be76c9759c43f8dc68cb254cea303cf"
|
| 9 |
}
|
stages.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
"counted": false,
|
| 3 |
"format": "portal-stages/1",
|
| 4 |
-
"index": "
|
| 5 |
"rules": [
|
| 6 |
"benchmark",
|
| 7 |
"not compared",
|
|
|
|
| 1 |
{
|
| 2 |
"counted": false,
|
| 3 |
"format": "portal-stages/1",
|
| 4 |
+
"index": "83dae072040e8688c08c84fb72fdc8690d0956e5",
|
| 5 |
"rules": [
|
| 6 |
"benchmark",
|
| 7 |
"not compared",
|