data-line commited on
Commit
a942796
·
1 Parent(s): 75ee15c

portal: catalog page of repository commit c57c7ccec904

Browse files

Files of catalog/portal/ (7), digest 7e4a5ec6e9e718ecf4bffd856914e5b9869cfe30f8c9ced8a6b4818ebebd76ad.

Files changed (8) hide show
  1. README.md +72 -5
  2. app.js +815 -0
  3. charts.js +303 -0
  4. data.js +294 -0
  5. icons.js +114 -0
  6. index.html +150 -17
  7. portal.json +6 -0
  8. style.css +290 -20
README.md CHANGED
@@ -1,11 +1,78 @@
1
  ---
2
  title: Agentic Env Catalog
3
- emoji: 🏆
4
- colorFrom: blue
5
- colorTo: gray
6
  sdk: static
 
7
  pinned: false
8
- short_description: Browse and fetch the agentic env index by domain and form
 
 
9
  ---
10
 
11
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Agentic Env Catalog
3
+ emoji: 🧭
4
+ colorFrom: gray
5
+ colorTo: blue
6
  sdk: static
7
+ app_file: index.html
8
  pinned: false
9
+ short_description: What the Agentic Env Index holds and how to take it
10
+ datasets:
11
+ - Battam/agentic-env-index
12
  ---
13
 
14
+ # Agentic Env Catalog
15
+
16
+ A static page into the public pointer index [Battam/agentic-env-index](https://huggingface.co/datasets/Battam/agentic-env-index):
17
+ what it holds by domain and by form, a few filters, and the SQL, the fetch command or a CSV file for the rows picked.
18
+ There is no backend; the page reads the dataset in the browser.
19
+
20
+ ## What it reads
21
+
22
+ The page asks the Hub for the latest commit of the dataset
23
+ (`https://huggingface.co/api/datasets/Battam/agentic-env-index/revision/main`, field `sha`) and reads every file at that
24
+ commit (`.../resolve/<sha>/<path>`), so all numbers on the page come from one commit and a new daily snapshot shows up
25
+ without a redeploy.
26
+
27
+ | File | Read | Fields used |
28
+ |---|---|---|
29
+ | `meta/catalog.parquet` | at load | `source`, `title`, `summary`, `description`, `status`, `status_reasons`, `domain`, `form` (when published), `license` (`spdx`, `tier`, `note`, `ship`), `contamination` (`level`, `benchmark`, `scope`), `pii` (`level`, `evidence`), `files`, `bytes`, `files_without_size`, `unit`, `top_types`, `needs_auth`, `link_check` |
30
+ | `meta/sources.json` | at load | per source: `license_rows` (training and redistribution, rows per answer), `license` (`name`, `statement`, `attribution`, `url`), `by_ext`, `size_quantiles_upper_bytes` |
31
+ | `meta/coverage.json` | at load | `by_license.training_ok` (rows per answer) |
32
+ | `meta/batches.parquet` | at load | `source`, `batch`, `rows`, `status` |
33
+ | `meta/not_included.parquet` | at load | `source`, `level`, `unit`, `category`, `detail`, `reviewed` |
34
+ | `meta/vocabulary.json` | at load, when the commit has it | `domains[].code`, `name`, `parent`; `forms[].id` |
35
+ | `CHANGELOG.md` | at load | the entries: date, title, sources added and removed, rows per source `(a to b)` |
36
+ | `README.md` | at load | the table of section "Not included, and why": the requirement of each category |
37
+ | `data/<source>/<batch>/*.parquet` | on demand | `source`, `batch`, `uri`, `range_offset`, `range_length`, `member_path`, `size_bytes`, `sha256`; the footer row count of a batch that `meta/batches.parquet` does not list |
38
+ | Hub API `revision/<sha>` and `tree/<sha>/data/<source>` | at load, on demand | the file list of the commit; the date of the last commit of each batch folder |
39
+
40
+ `meta/tags.parquet` is not read.
41
+
42
+ Domains and forms follow the classification contract of the index: `domain` is a list of codes (D1 to D10 and their leaves),
43
+ or `multi`, or `unclassified`; `form` is one id. Until the catalog carries them, sources show as not classified.
44
+ `meta/vocabulary.json` gives the names: `{"version": 1, "domains": [{"code", "name", "zh", "about", "parent"}],
45
+ "forms": [{"id", "zh", "about"}]}`. Names are shown in sentence case (a one-word slug such as `software-engineering`
46
+ becomes `Software engineering`); a form without an English name shows its id the same way; a code without a name shows as
47
+ the code. Maps from code to name, and nested `children`, are read too.
48
+
49
+ ## Query parameters
50
+
51
+ | Parameter | Meaning |
52
+ |---|---|
53
+ | `rev=<sha or branch>` | read another commit of the dataset |
54
+ | `catalog=<url>` | read the catalog from this URL (parquet, or JSON: a list of rows) instead of `meta/catalog.parquet` |
55
+ | `vocabulary=<url>` | read the names of domains and forms from this URL instead of `meta/vocabulary.json` |
56
+ | `domain`, `leaf`, `form`, `train`, `redist`, `tier`, `size`, `status`, `q` | the filters (`train` and `redist`: `yes` for every row allowed, `cond` for allowed or with conditions) |
57
+ | `metric`, `sort`, `dir`, `open` | the map's count, the sort of the source list, the open source |
58
+
59
+ The filters scope the map, the source list and "Get the data"; the overview, the history and "Not included" describe the
60
+ whole index. The licence filters keep a source only when every one of its rows has that permission
61
+ (`meta/sources.json`, `license_rows`), so the counts, the SQL and the CSV agree without a per-row licence condition.
62
+ "Source size" is the total known size of a source.
63
+
64
+ ## Files
65
+
66
+ - `index.html`, `style.css`: the frame; it keeps its layout while the data loads.
67
+ - `data.js`: reading the commit (hyparquet for the parquet files, byte ranges for data files).
68
+ - `charts.js`: thin bars, columns, the ring, sparklines, the heatmap, the tooltip, the chart and table toggle.
69
+ - `icons.js`: the pictograms (inline SVG, one stroke style) for the forms, the facts, the steps and the four parts.
70
+ - `app.js`: the filters and their URL state, the sections, the SQL, the command and the CSV.
71
+
72
+ Libraries come from cdn.jsdelivr.net, pinned: `hyparquet@1.31.5` and `hyparquet-compressors@1.1.2` (the meta files
73
+ are zstd-compressed). There is no build step. To look at it locally: `python -m http.server` in this folder.
74
+
75
+ ## Deploy
76
+
77
+ The data line pushes this folder to the Space when its sha256 differs from the one recorded in the Space's `portal.json`,
78
+ after the push has passed its outbound checks.
app.js ADDED
@@ -0,0 +1,815 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // The page: filter state in the URL, the sections, the selection and how to take it.
2
+ // Graphics first: short headlines, numbers with pictograms, small charts; every chart keeps a short label and a table view.
3
+ import { DATASET_URL, fileUrl, loadBatchDates, loadIndex, num, parquetRows, readColumns, REPO, topOf } from './data.js';
4
+ import {
5
+ clear, columns, el, figureWithToggle, fmtBytes, fmtCompact, fmtInt, fmtPct, fmtShort, heatmap, hbars, hideTip, meter,
6
+ ring, segmented, spark, stack, table, withTip,
7
+ } from './charts.js';
8
+ import { formIcon, icon } from './icons.js';
9
+
10
+ const GITHUB = 'https://github.com/msra-sg/agentic-env-pipeline';
11
+ const CSV_MAX_ROWS = 200000;
12
+ const ROW_COLUMNS = ['source', 'batch', 'uri', 'range_offset', 'range_length', 'member_path', 'size_bytes', 'sha256'];
13
+ const FIRST_ROWS = 20;
14
+ const HEAT_FORMS = 8;
15
+ const FORM_BARS = 10;
16
+ const $ = (id) => document.getElementById(id);
17
+
18
+ // ---- plain names (codes only inside the SQL and the command)
19
+ const TIER = {
20
+ permissive: 'Permissive', 'share-alike': 'Share-alike', 'nc-research': 'Non-commercial',
21
+ 'no-training': 'No training', 'internal-only': 'Internal only', mixed: 'Mixed', unknown: 'Not stated',
22
+ };
23
+ const PERMIT = { yes: 'Allowed', no: 'Not allowed', conditional: 'Conditional', unknown: 'Not stated' };
24
+ const PERMIT_ORDER = ['yes', 'no', 'conditional', 'unknown'];
25
+ const PERMIT_COLOR = { yes: 'var(--ours)', no: 'var(--problem)', conditional: 'var(--fill-3)', unknown: 'var(--other)' };
26
+ const CONTAM = { none: 'No overlap found', well_known_problems: 'Well-known problems', public_benchmark: 'Is a benchmark', tb_overlap: 'Overlaps a benchmark' };
27
+ const PII = { none_known: 'None known', suspected: 'Suspected', present: 'Present' };
28
+ const CATEGORY = {
29
+ not_english: 'Not English', licence_restricted: 'Licence too tight', access_gated: 'Behind a login',
30
+ personal_data: 'Personal data', answer_not_verifiable: 'Answer uncheckable', answer_cells_insufficient: 'Few answer cells',
31
+ not_in_scope: 'Out of scope', below_threshold: 'Too little activity', series_cap: 'Series cap', duplicate: 'Duplicate',
32
+ other: 'Other',
33
+ };
34
+ const NONE = '_none';
35
+ const MULTI = '_multi';
36
+ const OTHER = '_other';
37
+ const human = (code) => String(code).replace(/[_-]+/g, ' ').replace(/^./, (c) => c.toUpperCase());
38
+ /** Sentence case; a one-word slug (software-engineering) becomes words first. */
39
+ function sentence(s) {
40
+ let t = String(s).trim();
41
+ if (!/\s/.test(t)) t = t.replace(/[-_]+/g, ' ');
42
+ return t.charAt(0).toUpperCase() + t.slice(1);
43
+ }
44
+ const plural = (n, w) => `${fmtInt(n)} ${w}${n === 1 ? '' : 's'}`;
45
+
46
+ let IX = null;
47
+ const F = {};
48
+ const UI = { expanded: new Set(), showAll: false, allForms: false, detail: new Map(), csv: null, tab: 'sql' };
49
+ const KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q', 'metric', 'sort', 'dir', 'open'];
50
+
51
+ function readUrl() {
52
+ const p = new URLSearchParams(location.search);
53
+ for (const k of KEYS) F[k] = p.get(k) || '';
54
+ if (!['sources', 'rows', 'bytes'].includes(F.metric)) F.metric = 'sources';
55
+ if (!F.sort) { F.sort = 'rows'; F.dir = 'desc'; }
56
+ }
57
+ function writeUrl() {
58
+ const p = new URLSearchParams(location.search);
59
+ for (const k of KEYS) {
60
+ const v = F[k];
61
+ const dflt = (k === 'metric' && v === 'sources') || (k === 'sort' && v === 'rows') || (k === 'dir' && v === 'desc');
62
+ if (v && !dflt) p.set(k, v); else p.delete(k);
63
+ }
64
+ const s = p.toString();
65
+ history.replaceState(null, '', s ? `?${s}` : location.pathname);
66
+ }
67
+
68
+ // ---- names
69
+ const vocabName = (sec, code) => {
70
+ const n = IX.vocab[sec].names.get(code);
71
+ return n ? sentence(n) : null;
72
+ };
73
+ function domainName(key) {
74
+ if (key === NONE) return 'Not classified';
75
+ if (key === MULTI) return vocabName('domain', 'multi') || 'Several domains';
76
+ return vocabName('domain', key) || key;
77
+ }
78
+ function formName(key) {
79
+ if (key === NONE || key == null) return 'Not classified';
80
+ if (key === OTHER) return 'Other forms';
81
+ return vocabName('form', key) || sentence(key);
82
+ }
83
+ const domainKeysOf = (s) => (s.domainKind === 'codes' ? s.tops : [s.domainKind === 'multi' ? MULTI : NONE]);
84
+ const formKeyOf = (s) => s.form ?? NONE;
85
+ function sourceDomainText(s) {
86
+ if (s.domainKind !== 'codes') return domainName(s.domainKind === 'multi' ? MULTI : NONE);
87
+ return s.codes.map((c) => vocabName('domain', c) || c).join('; ');
88
+ }
89
+
90
+ // ---- filters
91
+ function passes(s, skip = {}) {
92
+ if (!skip.domain && F.domain) {
93
+ if (F.domain === MULTI) { if (s.domainKind !== 'multi') return false; }
94
+ else if (F.domain === NONE) { if (s.domainKind !== 'none') return false; }
95
+ else if (!s.tops.includes(F.domain)) return false;
96
+ if (F.leaf && !s.codes.includes(F.leaf)) return false;
97
+ }
98
+ if (!skip.form && F.form && formKeyOf(s) !== F.form) return false;
99
+ if (F.train === 'yes' && s.train !== 'yes') return false;
100
+ if (F.train === 'cond' && s.train === 'no') return false;
101
+ if (F.redist === 'yes' && s.redist !== 'yes') return false;
102
+ if (F.redist === 'cond' && s.redist === 'no') return false;
103
+ if (F.tier && s.tier !== F.tier) return false;
104
+ if (F.size) {
105
+ const b = s.bytes;
106
+ if (F.size === 'unknown') { if (b != null) return false; }
107
+ else if (b == null) return false;
108
+ else if (F.size === 'lt1g' && !(b < 1e9)) return false;
109
+ else if (F.size === '1g1t' && !(b >= 1e9 && b < 1e12)) return false;
110
+ else if (F.size === 'ge1t' && !(b >= 1e12)) return false;
111
+ }
112
+ if (F.status && s.status !== F.status) return false;
113
+ if (F.q) {
114
+ const q = F.q.toLowerCase();
115
+ if (!s.title.toLowerCase().includes(q) && !s.description.toLowerCase().includes(q)) return false;
116
+ }
117
+ return true;
118
+ }
119
+ const selected = () => IX.sources.filter((s) => passes(s));
120
+ const FILTER_KEYS = ['domain', 'leaf', 'form', 'train', 'redist', 'tier', 'size', 'status', 'q'];
121
+ const activeFilters = () => FILTER_KEYS.filter((k) => F[k]);
122
+
123
+ /** The active filters as short chips: [label, value]. */
124
+ function filterChips() {
125
+ const out = [];
126
+ const opt = (id) => $(id).selectedOptions[0]?.textContent || '';
127
+ if (F.domain) out.push(['Domain', F.leaf ? domainName(F.leaf) : domainName(F.domain)]);
128
+ if (F.form) out.push(['Form', formName(F.form)]);
129
+ if (F.train) out.push(['Training', opt('f-train')]);
130
+ if (F.redist) out.push(['Redistribution', opt('f-redist')]);
131
+ if (F.tier) out.push(['Tier', TIER[F.tier] || human(F.tier)]);
132
+ if (F.size) out.push(['Size', opt('f-size')]);
133
+ if (F.status) out.push(['Status', opt('f-status')]);
134
+ if (F.q) out.push(['Search', F.q]);
135
+ return out;
136
+ }
137
+
138
+ function metricOf(s, m = F.metric) { return m === 'sources' ? 1 : m === 'rows' ? s.rows : s.bytes || 0; }
139
+ const metricFmt = (m = F.metric) => (m === 'bytes' ? fmtBytes : m === 'rows' ? (v) => fmtInt(v) : (v) => fmtInt(v));
140
+ const METRIC_WORD = { sources: 'Sources', rows: 'Pointer rows', bytes: 'Size' };
141
+
142
+ function setFilter(patch) {
143
+ Object.assign(F, patch);
144
+ UI.showAll = false;
145
+ writeUrl();
146
+ renderScoped();
147
+ }
148
+
149
+ // ---- filter row
150
+ function option(value, text) { return el('option', { value }, text); }
151
+ function fillFilterRow() {
152
+ clear($('f-domain')).append(option('', 'All'), domainRows().map((k) => option(k, domainName(k))));
153
+ clear($('f-form')).append(option('', 'All'), formsByCount().map((k) => option(k, formName(k))));
154
+ const tiers = [...new Set(IX.sources.map((s) => s.tier))].sort((a, b) => (TIER[a] || a).localeCompare(TIER[b] || b));
155
+ clear($('f-tier')).append(option('', 'All'), tiers.map((t) => option(t, TIER[t] || human(t))));
156
+ const on = (id, key) => $(id).addEventListener('change', (e) => {
157
+ const patch = { [key]: e.target.value };
158
+ if (key === 'domain') patch.leaf = '';
159
+ setFilter(patch);
160
+ });
161
+ on('f-domain', 'domain'); on('f-leaf', 'leaf'); on('f-form', 'form'); on('f-train', 'train'); on('f-redist', 'redist');
162
+ on('f-tier', 'tier'); on('f-size', 'size'); on('f-status', 'status');
163
+ let t = null;
164
+ $('f-q').addEventListener('input', (e) => { clearTimeout(t); t = setTimeout(() => setFilter({ q: e.target.value.trim() }), 200); });
165
+ $('f-clear').addEventListener('click', () => setFilter(Object.fromEntries(FILTER_KEYS.map((k) => [k, '']))));
166
+ $('filter-toggle').addEventListener('click', () => {
167
+ const open = !$('filters').classList.contains('open');
168
+ $('filters').classList.toggle('open', open);
169
+ $('filter-toggle').setAttribute('aria-expanded', String(open));
170
+ });
171
+ }
172
+ function syncFilterRow(sel) {
173
+ const set = (id, v) => { if ($(id).value !== v) $(id).value = v; };
174
+ set('f-domain', F.domain); set('f-form', F.form); set('f-train', F.train); set('f-redist', F.redist);
175
+ set('f-tier', F.tier); set('f-size', F.size); set('f-status', F.status);
176
+ if (document.activeElement !== $('f-q')) $('f-q').value = F.q;
177
+ const leaf = $('f-leaf');
178
+ const leaves = F.domain && F.domain !== NONE && F.domain !== MULTI ? leafKeys(F.domain) : [];
179
+ clear(leaf).append(option('', 'All'), leaves.map((k) => option(k, domainName(k))));
180
+ leaf.disabled = !leaves.length;
181
+ leaf.value = F.leaf;
182
+ const n = activeFilters().length;
183
+ $('f-clear').disabled = !n;
184
+ $('filter-state').textContent = n ? `${n} on · ${plural(sel.length, 'source')}` : 'none on';
185
+ $('f-count').textContent = plural(sel.length, 'source');
186
+ }
187
+
188
+ // ---- axes of the map
189
+ function numericCodeSort(a, b) {
190
+ const pa = a.replace(/^D/, '').split('.').map(Number);
191
+ const pb = b.replace(/^D/, '').split('.').map(Number);
192
+ for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
193
+ const d = (pa[i] ?? -1) - (pb[i] ?? -1);
194
+ if (d) return d;
195
+ }
196
+ return a.localeCompare(b);
197
+ }
198
+ function domainRows() {
199
+ const tops = new Set();
200
+ for (const c of IX.vocab.domain.order) if (!c.includes('.') && c !== 'multi' && c !== 'unclassified') tops.add(c);
201
+ for (const s of IX.sources) for (const t of s.tops) tops.add(t);
202
+ const out = [...tops].sort(numericCodeSort);
203
+ if (IX.sources.some((s) => s.domainKind === 'multi') || IX.vocab.domain.names.has('multi')) out.push(MULTI);
204
+ out.push(NONE);
205
+ return out;
206
+ }
207
+ function leafKeys(top) {
208
+ const leaves = new Set();
209
+ for (const c of IX.vocab.domain.order) if (c.startsWith(`${top}.`)) leaves.add(c);
210
+ for (const s of IX.sources) for (const c of s.codes) if (c.startsWith(`${top}.`)) leaves.add(c);
211
+ return [...leaves].sort(numericCodeSort);
212
+ }
213
+ /** Every form, most sources first over the whole index (a fixed order: it does not move with the filters). */
214
+ let FORM_ORDER = null;
215
+ function formsByCount() {
216
+ if (FORM_ORDER) return FORM_ORDER;
217
+ const n = new Map();
218
+ for (const c of IX.vocab.form.order) if (c !== 'unclassified') n.set(c, 0);
219
+ for (const s of IX.sources) if (s.form) n.set(s.form, (n.get(s.form) || 0) + 1);
220
+ FORM_ORDER = [...n].sort((a, b) => b[1] - a[1] || formName(a[0]).localeCompare(formName(b[0]))).map(([k]) => k);
221
+ FORM_ORDER.push(NONE);
222
+ return FORM_ORDER;
223
+ }
224
+ /** The heatmap's columns: the forms with the most sources, the rest as one column, and the unclassified. */
225
+ function heatCols() {
226
+ const all = formsByCount().filter((k) => k !== NONE && IX.sources.some((s) => s.form === k));
227
+ const cols = all.length > HEAT_FORMS + 1 ? [...all.slice(0, HEAT_FORMS), OTHER] : all;
228
+ cols.push(NONE);
229
+ return cols;
230
+ }
231
+ const heatColOf = (s, cols) => (cols.includes(formKeyOf(s)) ? formKeyOf(s) : OTHER);
232
+
233
+ // ---- overview and history (the whole index; above the filters)
234
+ function fact(iconName, value, label, graphic) {
235
+ return el('li', { class: 'fact' }, icon(iconName, { size: 26 }),
236
+ el('span', { class: 'fact-text' }, el('b', {}, value), el('span', {}, label)), graphic || null);
237
+ }
238
+ function snapshots() {
239
+ const log = IX.changelog;
240
+ const start = log.find((e) => e.kind === 'start');
241
+ const out = [];
242
+ let src = start ? start.sources : null;
243
+ let rows = start ? start.rows : null;
244
+ if (start) out.push({ date: start.date, sources: src, rows, added: start.sources, rowsAdded: start.rows, start: true });
245
+ for (const e of log) {
246
+ if (e.kind !== 'update') continue;
247
+ if (src != null) src += e.sourcesAdded - e.sourcesRemoved;
248
+ if (rows != null) rows += e.rowsAdded - e.rowsRemoved;
249
+ out.push({ date: e.date, sources: src, rows, added: e.sourcesAdded, removed: e.sourcesRemoved, rowsAdded: e.rowsAdded, rowsRemoved: e.rowsRemoved });
250
+ }
251
+ return out;
252
+ }
253
+
254
+ function renderOverview() {
255
+ const S = IX.sources;
256
+ const rows = S.reduce((a, s) => a + s.rows, 0);
257
+ const bytes = S.reduce((a, s) => a + (s.bytes || 0), 0);
258
+ const noSize = S.reduce((a, s) => a + s.rowsNoSize, 0);
259
+ const lic = IX.coverage.by_license?.training_ok || {};
260
+ const parts = PERMIT_ORDER.map((k) => ({ k, value: num(lic[k]?.rows) || 0, color: PERMIT_COLOR[k], label: PERMIT[k] }));
261
+ const licTotal = parts.reduce((a, p) => a + p.value, 0);
262
+ const snaps = snapshots();
263
+ const last = snaps.filter((x) => !x.start).pop();
264
+ $('overview-h').textContent = licTotal ? `${fmtPct(parts[0].value, licTotal)} cleared for training` : 'The index';
265
+ $('hero-big').textContent = fmtCompact(rows);
266
+ clear($('facts')).append(
267
+ fact('sources', fmtInt(S.length), 'sources', snaps.length > 1 ? spark(snaps.map((x) => x.sources)) : null),
268
+ fact('size', fmtBytes(bytes), 'known size', el('span', { class: 'fact-meter' }, meter(rows ? (rows - noSize) / rows : 0),
269
+ el('span', { class: 'small' }, `${fmtPct(rows - noSize, rows)} sized`))),
270
+ last ? fact('snapshot', `+${fmtShort(last.rowsAdded)}`, `rows on ${last.date.slice(5)}`) : null);
271
+ figureWithToggle($('licence-chart'), 'licence', 'Training, by pointer rows', {
272
+ chart: () => el('div', { class: 'ring-wrap' },
273
+ el('div', { class: 'ring-box' }, ring(parts, { size: 132, width: 16, label: parts.map((p) => `${p.label} ${fmtPct(p.value, licTotal)}`).join(', ') }),
274
+ el('span', { class: 'ring-center' }, icon('training', { size: 26 }))),
275
+ el('ul', { class: 'ring-legend' }, parts.map((p) => withTip(el('li', { tabindex: '0' },
276
+ el('span', { class: 'sw', style: { background: p.color } }), el('span', {}, p.label), el('b', {}, fmtPct(p.value, licTotal))),
277
+ fmtInt(p.value), `${p.label}, pointer rows`)))),
278
+ table: () => table(['Training', { label: 'Pointer rows', r: true }, { label: 'Share', r: true }],
279
+ parts.map((p) => [p.label, fmtInt(p.value), fmtPct(p.value, licTotal)])),
280
+ });
281
+ }
282
+
283
+ function renderHistory() {
284
+ const snaps = snapshots();
285
+ const last = snaps.filter((x) => !x.start).pop();
286
+ $('history-h').textContent = last ? `+${fmtInt(last.added)} sources on ${last.date.slice(5)}` : 'No snapshot yet';
287
+ figureWithToggle($('history-fig'), 'history', 'Sources after each snapshot', {
288
+ chart: () => columns(snaps.map((x) => ({
289
+ key: x.date, label: x.date.slice(5), value: x.sources || 0, top: fmtInt(x.sources),
290
+ sub: x.start ? 'start' : `+${fmtInt(x.added)}`,
291
+ }))),
292
+ table: () => table(['Date', 'Entry', { label: 'Sources added', r: true }, { label: 'Rows added', r: true }],
293
+ [...IX.changelog].reverse().map((e) => [e.date, e.kind === 'start' ? 'Log started' : sentence(e.title),
294
+ e.kind === 'update' ? fmtInt(e.sourcesAdded) : e.kind === 'start' ? fmtInt(e.sources) : '',
295
+ e.kind === 'update' ? fmtInt(e.rowsAdded) : e.kind === 'start' ? fmtInt(e.rows) : ''])),
296
+ });
297
+ }
298
+
299
+ // ---- the map
300
+ function renderMap() {
301
+ const m = F.metric;
302
+ const fmt = metricFmt(m);
303
+ const base = IX.sources.filter((s) => passes(s, { domain: true, form: true }));
304
+ const rows = domainRows();
305
+ const cols = heatCols();
306
+ const cell = new Map();
307
+ for (const s of base) for (const d of domainKeysOf(s)) {
308
+ const k = `${d}|${heatColOf(s, cols)}`;
309
+ cell.set(k, (cell.get(k) || 0) + metricOf(s, m));
310
+ }
311
+ const total = base.reduce((a, s) => a + metricOf(s, m), 0);
312
+ const unclassified = base.filter((s) => s.domainKind === 'none').reduce((a, s) => a + metricOf(s, m), 0);
313
+ const byDom = new Map(rows.map((r) => [r, 0]));
314
+ for (const s of base) for (const d of domainKeysOf(s)) byDom.set(d, byDom.get(d) + metricOf(s, m));
315
+ const ranked = [...byDom].filter(([k]) => k !== NONE && k !== MULTI).sort((a, b) => b[1] - a[1]);
316
+ const best = ranked[0] && ranked[0][1] > 0 ? ranked[0] : null;
317
+ let head;
318
+ if (!base.length) head = 'Nothing matches';
319
+ else if (unclassified / (total || 1) >= 0.5) head = `${fmtPct(unclassified, total)} not classified yet`;
320
+ else if (best) head = `Most ${METRIC_WORD[m].toLowerCase()}: ${domainName(best[0])}`;
321
+ else head = 'By domain and form';
322
+ $('map-h').textContent = head;
323
+ for (const b of $('metric-seg').querySelectorAll('button')) b.setAttribute('aria-pressed', String(b.dataset.metric === m));
324
+
325
+ const current = F.domain || F.form ? { row: F.domain || null, col: F.form || null } : null;
326
+ const rowObjs = rows.map((k) => ({ key: k, label: domainName(k) }));
327
+ const colObjs = cols.map((k) => ({ key: k, label: formName(k), head: formIcon(k, { size: 20 }) }));
328
+ figureWithToggle($('heat-fig'), 'heat', `${METRIC_WORD[m]}: domain × form`, {
329
+ chart: () => {
330
+ const h = heatmap({
331
+ rows: rowObjs, cols: colObjs, value: (r, c) => cell.get(`${r.key}|${c.key}`) || 0, fmt,
332
+ current: current && current.row && current.col ? current : null, metricName: METRIC_WORD[m],
333
+ onPick: (r, c) => {
334
+ const form = c.key === OTHER ? '' : c.key;
335
+ if (F.domain === r.key && F.form === form) setFilter({ domain: '', leaf: '', form: '' });
336
+ else setFilter({ domain: r.key, leaf: '', form });
337
+ },
338
+ });
339
+ const key = el('ul', { class: 'icon-key', 'aria-label': 'Columns' },
340
+ colObjs.map((c) => el('li', {}, formIcon(c.key, { size: 16 }), c.label)));
341
+ return el('div', {}, h.grid, key, h.legend);
342
+ },
343
+ table: () => table(['Domain', 'Form', { label: METRIC_WORD[m], r: true }],
344
+ rowObjs.flatMap((r) => colObjs.map((c) => [r, c, cell.get(`${r.key}|${c.key}`) || 0])).filter((x) => x[2] > 0)
345
+ .map(([r, c, v]) => [r.label, c.label, fmt(v)])),
346
+ });
347
+
348
+ // domain bars (leave out the domain filter), form bars (leave out the form filter)
349
+ const domBase = IX.sources.filter((s) => passes(s, { domain: true }));
350
+ const domVal = new Map(rows.map((r) => [r, 0]));
351
+ for (const s of domBase) for (const d of domainKeysOf(s)) domVal.set(d, domVal.get(d) + metricOf(s, m));
352
+ const items = [];
353
+ for (const r of rows) {
354
+ const leaves = r === NONE || r === MULTI ? [] : leafKeys(r).filter((l) => domBase.some((s) => s.codes.includes(l)));
355
+ const open = UI.expanded.has(r);
356
+ items.push({
357
+ key: r, label: domainName(r), value: domVal.get(r), text: fmt(domVal.get(r)), current: F.domain === r && !F.leaf,
358
+ expanded: leaves.length ? open : null, tipLabel: domainName(r),
359
+ onClick: leaves.length ? () => { if (open) UI.expanded.delete(r); else UI.expanded.add(r); renderMap(); } : () => setFilter({ domain: F.domain === r ? '' : r, leaf: '' }),
360
+ });
361
+ if (open) {
362
+ const lv = new Map(leaves.map((l) => [l, 0]));
363
+ for (const s of domBase) for (const c of s.codes) if (lv.has(c)) lv.set(c, lv.get(c) + metricOf(s, m));
364
+ for (const l of leaves) {
365
+ items.push({ key: l, label: domainName(l), value: lv.get(l), text: fmt(lv.get(l)), leaf: true, current: F.leaf === l,
366
+ tipLabel: domainName(l), onClick: () => setFilter(F.leaf === l ? { leaf: '' } : { domain: r, leaf: l }) });
367
+ }
368
+ }
369
+ }
370
+ const domMax = Math.max(0, ...rows.map((r) => domVal.get(r)));
371
+ figureWithToggle($('dom-fig'), 'dom', 'By domain', {
372
+ chart: () => hbars(items, { max: domMax }),
373
+ table: () => table(['Domain', { label: METRIC_WORD[m], r: true }], items.map((d) => [d.leaf ? ` ${d.label}` : d.label, d.text])),
374
+ });
375
+
376
+ const formBase = IX.sources.filter((s) => passes(s, { form: true }));
377
+ const fv = new Map();
378
+ for (const s of formBase) fv.set(formKeyOf(s), (fv.get(formKeyOf(s)) || 0) + metricOf(s, m));
379
+ const fAll = formsByCount().filter((k) => fv.get(k) > 0 || k === F.form)
380
+ .sort((a, b) => (fv.get(b) || 0) - (fv.get(a) || 0));
381
+ const fShown = UI.allForms ? fAll : fAll.slice(0, FORM_BARS);
382
+ const fItems = fShown.map((c) => ({ key: c, label: formName(c), icon: formIcon(c, { size: 16 }), value: fv.get(c) || 0, text: fmt(fv.get(c) || 0),
383
+ current: F.form === c, tipLabel: formName(c), onClick: () => setFilter({ form: F.form === c ? '' : c }) }));
384
+ figureWithToggle($('form-fig'), 'form', 'By form', {
385
+ chart: () => hbars(fItems),
386
+ table: () => table(['Form', { label: METRIC_WORD[m], r: true }], fAll.map((c) => [formName(c), fmt(fv.get(c) || 0)])),
387
+ }, () => (fAll.length > FORM_BARS ? el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.allForms),
388
+ onclick: () => { UI.allForms = !UI.allForms; renderMap(); } }, UI.allForms ? 'Fewer' : `All ${fAll.length}`) : null));
389
+ }
390
+
391
+ // ---- sources
392
+ const COLS = [
393
+ { key: 'title', label: 'Source' },
394
+ { key: 'domain', label: 'Domain' },
395
+ { key: 'form', label: 'Form' },
396
+ { key: 'rows', label: 'Rows', r: true },
397
+ { key: 'bytes', label: 'Size', r: true },
398
+ { key: 'tier', label: 'Licence' },
399
+ { key: 'link', label: 'Links' },
400
+ ];
401
+ function sortValue(s, k) {
402
+ switch (k) {
403
+ case 'title': return s.title.toLowerCase();
404
+ case 'domain': return sourceDomainText(s).toLowerCase();
405
+ case 'form': return formName(formKeyOf(s)).toLowerCase();
406
+ case 'rows': return s.rows;
407
+ case 'bytes': return s.bytes ?? -1;
408
+ case 'tier': return (TIER[s.tier] || s.tier).toLowerCase();
409
+ case 'link': return s.link.kind === 'ok' || s.link.kind === 'partial' ? s.link.ok / s.link.total : -1;
410
+ default: return 0;
411
+ }
412
+ }
413
+ function linkCell(s) {
414
+ const l = s.link;
415
+ if (l.kind === 'ok') return el('span', { class: 'ok-text' }, icon('link', { size: 14 }), '100%');
416
+ if (l.kind === 'partial') return el('span', { class: 'problem-text' }, el('span', { class: 'dot problem', 'aria-hidden': 'true' }), fmtPct(l.ok, l.total));
417
+ if (l.kind === 'login') return 'Login';
418
+ return el('span', { class: 'muted' }, 'Not checked');
419
+ }
420
+ function sizeText(s) { return s.bytes == null ? 'Unknown' : fmtBytes(s.bytes); }
421
+
422
+ function renderSources(sel) {
423
+ const dir = F.dir === 'asc' ? 1 : -1;
424
+ const list = [...sel].sort((a, b) => {
425
+ const x = sortValue(a, F.sort);
426
+ const y = sortValue(b, F.sort);
427
+ return (x < y ? -1 : x > y ? 1 : 0) * dir || a.title.localeCompare(b.title);
428
+ });
429
+ $('sources-h').textContent = sel.length ? plural(sel.length, 'source') : 'No source matches';
430
+ const body = clear($('sources-body'));
431
+ body.style.minHeight = '';
432
+ if (!sel.length) return;
433
+ const shown = UI.showAll ? list : list.slice(0, FIRST_ROWS);
434
+ const maxRows = Math.max(...sel.map((s) => s.rows));
435
+ const head = el('tr', {}, COLS.map((c) => {
436
+ const on = F.sort === c.key;
437
+ return el('th', { scope: 'col', class: c.r ? 'r' : null, 'aria-sort': on ? (F.dir === 'asc' ? 'ascending' : 'descending') : null },
438
+ el('button', { type: 'button', onclick: () => {
439
+ const d = on ? (F.dir === 'asc' ? 'desc' : 'asc') : (c.r ? 'desc' : 'asc');
440
+ F.sort = c.key; F.dir = d; writeUrl(); renderSources(selected());
441
+ } }, c.label, on ? (F.dir === 'asc' ? ' ↑' : ' ↓') : ''));
442
+ }));
443
+ const tbody = el('tbody');
444
+ for (const s of shown) {
445
+ const open = F.open === s.id;
446
+ tbody.append(el('tr', { class: `row${open ? ' open' : ''}` },
447
+ el('td', { class: 'title' },
448
+ el('button', { type: 'button', 'aria-expanded': String(open), 'aria-controls': `d-${s.id}`, onclick: () => {
449
+ F.open = open ? '' : s.id; writeUrl(); renderSources(selected());
450
+ } }, s.title),
451
+ s.status === 'flagged' ? el('span', { class: 'dot problem flag-dot', role: 'img', 'aria-label': 'Flagged', title: 'Flagged' }) : null),
452
+ el('td', { 'data-label': 'Domain' }, sourceDomainText(s)),
453
+ el('td', { 'data-label': 'Form', class: 'form-cell' }, formIcon(formKeyOf(s), { size: 16 }), formName(formKeyOf(s))),
454
+ el('td', { class: 'r rows-cell', 'data-label': 'Rows' }, fmtInt(s.rows), el('span', { class: 'mini', 'aria-hidden': 'true' },
455
+ el('span', { style: { width: `${maxRows ? Math.max(2, (100 * Math.log10(1 + s.rows)) / Math.log10(1 + maxRows)) : 0}%` } }))),
456
+ el('td', { class: 'r', 'data-label': 'Size' }, sizeText(s)),
457
+ el('td', { 'data-label': 'Licence' }, TIER[s.tier] || human(s.tier)),
458
+ el('td', { 'data-label': 'Links' }, linkCell(s))));
459
+ if (open) tbody.append(el('tr', { class: 'detail-row' }, el('td', { colspan: String(COLS.length), id: `d-${s.id}` }, detail(s))));
460
+ }
461
+ body.append(el('table', { class: 'src-table' }, el('caption', { class: 'visually-hidden' }, 'Sources; row bars on a log scale'), el('thead', {}, head), tbody));
462
+ if (list.length > FIRST_ROWS) {
463
+ body.append(el('p', { class: 'more' }, el('button', { type: 'button', class: 'link', 'aria-expanded': String(UI.showAll),
464
+ onclick: () => { UI.showAll = !UI.showAll; renderSources(selected()); } },
465
+ UI.showAll ? `First ${FIRST_ROWS}` : `All ${fmtInt(list.length)}`)));
466
+ }
467
+ }
468
+
469
+ // ---- the detail of one source, opened in place; batches and sample rows are read on demand
470
+ const hex = (v) => (v instanceof Uint8Array ? [...v].map((b) => b.toString(16).padStart(2, '0')).join('') : v == null ? '' : String(v));
471
+
472
+ /** Size spread: a thin log axis from 1 KB to the largest bound, the 10th to 90th percentile as a bar, the median as a dot. */
473
+ function sizeStrip(q) {
474
+ const v = (k) => num(q[k]) + 1;
475
+ const lo = Math.log10(Math.min(1e3, v('p10')));
476
+ const hi = Math.log10(Math.max(v('p99'), v('p90') * 10));
477
+ const at = (x) => `${(100 * (Math.log10(x) - lo)) / (hi - lo)}%`;
478
+ return el('div', { class: 'strip', role: 'img', 'aria-label': `median ${fmtBytes(v('p50'))}, 90% under ${fmtBytes(v('p90'))}` },
479
+ el('span', { class: 'strip-axis' }),
480
+ el('span', { class: 'strip-band', style: { left: at(v('p10')), width: `calc(${at(v('p90'))} - ${at(v('p10'))})` } }),
481
+ el('span', { class: 'strip-dot', style: { left: at(v('p50')) } }),
482
+ el('span', { class: 'strip-tick', style: { left: at(v('p99')) } }),
483
+ el('span', { class: 'strip-lab', style: { left: at(v('p50')) } }, `median ${fmtBytes(v('p50'))}`),
484
+ el('span', { class: 'strip-lab end' }, `99% < ${fmtBytes(v('p99'))}`));
485
+ }
486
+
487
+ function detail(s) {
488
+ const st = UI.detail.get(s.id) || {};
489
+ UI.detail.set(s.id, st);
490
+ const info = s.info || {};
491
+ const lic = info.license || {};
492
+ const box = el('div', { class: 'detail' });
493
+ const redraw = () => {
494
+ const cell = document.getElementById(`d-${s.id}`);
495
+ if (cell) clear(cell).append(detail(s));
496
+ };
497
+ const text = s.summary || s.description;
498
+ box.append(el('div', { class: 'full' },
499
+ text ? el('details', { class: 'about' }, el('summary', {}, el('span', { class: 'mono' }, s.id), ` · ${s.unit || 'file'}`), el('p', {}, text))
500
+ : el('p', { class: 'mono' }, s.id),
501
+ s.status === 'flagged' && s.reasons.length
502
+ ? el('details', { class: 'problem-text small' }, el('summary', {}, el('span', { class: 'dot problem', 'aria-hidden': 'true' }), 'Flagged'), el('p', {}, s.reasons.join('; '))) : null));
503
+
504
+ if (!st.batches) {
505
+ st.batches = 'loading';
506
+ loadBatchInfo(s).then((b) => { st.batches = b; redraw(); }, (e) => { st.batches = { error: e.message }; redraw(); });
507
+ }
508
+ const b = st.batches;
509
+ const bFig = el('figure', { class: 'chart' });
510
+ if (b === 'loading') bFig.append(el('p', { class: 'chart-title' }, 'Batches'), el('p', { class: 'muted' }, '…'));
511
+ else if (b.error) bFig.append(el('p', { class: 'chart-title' }, 'Batches'), el('p', { class: 'problem-text' }, b.error));
512
+ else {
513
+ figureWithToggle(bFig, `batches-${s.id}`, 'Batches, by rows', {
514
+ chart: () => hbars(b.map((x) => ({ key: x.batch, label: `${x.batch}${x.date ? ` · ${x.date.slice(5, 10)}` : ''}`, value: x.rows || 0,
515
+ text: x.rows == null ? '?' : fmtInt(x.rows), tipLabel: x.status ? human(x.status) : x.batch }))),
516
+ table: () => table(['Batch', { label: 'Rows', r: true }, 'Changed', 'State'],
517
+ b.map((x) => [x.batch, x.rows == null ? 'not counted' : fmtInt(x.rows), x.date ? x.date.slice(0, 10) : '', x.status ? human(x.status) : ''])),
518
+ });
519
+ }
520
+ box.append(bFig);
521
+
522
+ const ext = Object.entries(info.by_ext || {}).map(([k, v]) => ({ k, rows: num(v.rows) || 0 })).sort((a, z) => z.rows - a.rows);
523
+ const top = ext.slice(0, 6);
524
+ const rest = ext.slice(6).reduce((a, e) => a + e.rows, 0);
525
+ const extLabel = (k) => (k === '(none)' ? 'none' : k === '(other)' ? 'other' : `.${k}`);
526
+ const typeItems = top.map((e) => ({ key: e.k, label: extLabel(e.k), value: e.rows, text: fmtInt(e.rows), tipLabel: extLabel(e.k) }));
527
+ if (rest) typeItems.push({ key: 'rest', label: `+${ext.length - 6} types`, value: rest, text: fmtInt(rest), tipLabel: 'other types' });
528
+ const typesFig = el('figure', { class: 'chart' });
529
+ figureWithToggle(typesFig, `types-${s.id}`, 'File types', {
530
+ chart: () => (typeItems.length ? hbars(typeItems) : el('p', { class: 'muted' }, 'None recorded')),
531
+ table: () => table(['Type', { label: 'Rows', r: true }], ext.map((e) => [extLabel(e.k), fmtInt(e.rows)])),
532
+ });
533
+ box.append(typesFig);
534
+
535
+ const q = info.size_quantiles_upper_bytes || {};
536
+ box.append(el('div', {}, el('p', { class: 'chart-title' }, 'Size'),
537
+ el('p', { class: 'fact-line' }, icon('size', { size: 20 }), el('b', {}, sizeText(s)),
538
+ s.rowsNoSize ? el('span', { class: 'small' }, `${fmtCompact(s.rowsNoSize)} rows unsized`) : null),
539
+ q.p50 ? sizeStrip(q) : null));
540
+
541
+ const lr = info.license_rows || {};
542
+ const permit = (field) => PERMIT_ORDER.map((k) => ({ k, value: num(lr[field]?.[k]?.rows) || 0, color: PERMIT_COLOR[k], label: PERMIT[k] }));
543
+ const licBox = el('div', {}, el('p', { class: 'chart-title' }, 'Licence'),
544
+ el('p', { class: 'fact-line' }, el('b', {}, TIER[s.tier] || human(s.tier)), el('span', { class: 'small' }, lic.name || s.spdx || '')),
545
+ el('p', { class: 'mini-title' }, icon('training', { size: 16 }), 'Training'), stack(permit('training_ok'), s.rows),
546
+ el('p', { class: 'mini-title' }, icon('share', { size: 16 }), 'Redistribution'), stack(permit('redistribution_ok'), s.rows),
547
+ lic.statement || s.licenceNote || lic.url ? el('details', { class: 'small' }, el('summary', {}, 'Statement'),
548
+ el('p', {}, lic.statement || s.licenceNote || ''), lic.attribution ? el('p', {}, lic.attribution) : null,
549
+ lic.url ? el('p', {}, el('a', { href: lic.url, rel: 'noopener' }, 'Licence page')) : null) : null);
550
+ box.append(licBox);
551
+
552
+ const c = s.contamination || {};
553
+ const p = s.pii || {};
554
+ const flag = (bad) => el('span', { class: `dot ${bad ? 'problem' : 'ours'}`, 'aria-hidden': 'true' });
555
+ box.append(el('div', {}, el('p', { class: 'chart-title' }, 'Checks'),
556
+ el('p', { class: 'fact-line' }, icon('target', { size: 18 }), el('span', {}, 'Benchmarks'),
557
+ c.level ? [flag(c.level !== 'none'), el('b', {}, CONTAM[c.level] || human(c.level))] : el('span', { class: 'muted' }, 'Not checked')),
558
+ el('p', { class: 'fact-line' }, icon('person', { size: 18 }), el('span', {}, 'Personal data'),
559
+ p.level ? [flag(p.level !== 'none_known'), el('b', {}, PII[p.level] || human(p.level))] : el('span', { class: 'muted' }, 'Not judged'))));
560
+
561
+ const smp = st.sample;
562
+ const sampleBox = el('div', { class: 'full' });
563
+ if (!s.files.length) sampleBox.append(el('p', { class: 'muted' }, 'No data files'));
564
+ else if (!smp) {
565
+ sampleBox.append(el('button', { type: 'button', class: 'btn', onclick: () => {
566
+ st.sample = 'loading'; redraw();
567
+ readColumns(fileUrl(IX.sha, s.files[0].path), ROW_COLUMNS, { rowStart: 0, rowEnd: FIRST_ROWS })
568
+ .then((r) => { st.sample = r.rows; redraw(); }, (e) => { st.sample = { error: e.message }; redraw(); });
569
+ } }, icon('rows', { size: 16 }), ` ${FIRST_ROWS} sample rows`));
570
+ } else if (smp === 'loading') sampleBox.append(el('p', { class: 'muted' }, '…'));
571
+ else if (smp.error) sampleBox.append(el('p', { class: 'problem-text' }, smp.error));
572
+ else {
573
+ sampleBox.append(el('div', { class: 'table-wrap sample' }, table(['Address', { label: 'Size', r: true }, 'sha256'],
574
+ smp.map((r) => [r.member_path ? `${r.uri} ! ${r.member_path}` : r.range_offset != null ? `${r.uri} @ ${num(r.range_offset)}+${num(r.range_length)}` : r.uri,
575
+ r.size_bytes == null ? '' : fmtInt(num(r.size_bytes)), hex(r.sha256).slice(0, 16) + (r.sha256 ? '…' : '')]),
576
+ s.files[0].path)));
577
+ }
578
+ box.append(sampleBox);
579
+ return box;
580
+ }
581
+
582
+ async function loadBatchInfo(s) {
583
+ const dates = await loadBatchDates(IX.sha, s.id).catch(() => new Map());
584
+ const listed = new Map(s.batches.map((b) => [b.batch, b]));
585
+ const names = [...new Set([...s.files.map((f) => f.batch), ...listed.keys(), ...dates.keys()])].sort();
586
+ const out = [];
587
+ for (const name of names) {
588
+ const b = listed.get(name);
589
+ let rows = b ? num(b.rows) : null;
590
+ const files = s.files.filter((f) => f.batch === name);
591
+ if (rows == null && files.length && files.length <= 40) {
592
+ const counts = await Promise.all(files.map((f) => parquetRows(fileUrl(IX.sha, f.path))));
593
+ rows = counts.reduce((a, n) => a + n, 0);
594
+ }
595
+ out.push({ batch: name, rows, date: dates.get(name) || null, status: b?.status || (files.some((f) => /part-tail/.test(f.path)) ? 'collecting' : null) });
596
+ }
597
+ return out;
598
+ }
599
+
600
+ // ---- get the data: pick, copy, run
601
+ const sqlStr = (v) => `'${String(v).replace(/'/g, "''")}'`;
602
+ function selectionSql(sel) {
603
+ const root = `hf://datasets/${REPO}@${IX.sha}`;
604
+ const all = sel.length === IX.sources.length;
605
+ const paths = all ? [`${root}/data/*/*/*.parquet`] : sel.map((s) => `${root}/data/${s.id}/*/*.parquet`);
606
+ const rows = sel.reduce((a, s) => a + s.rows, 0);
607
+ const chips = filterChips();
608
+ return [
609
+ `-- Agentic Env Index @ ${IX.sha.slice(0, 7)}: ${plural(sel.length, 'source')}, ${fmtInt(rows)} rows`,
610
+ chips.length ? `-- ${chips.map(([k, v]) => `${k}: ${v}`).join('; ')}` : '-- all sources',
611
+ `SELECT ${ROW_COLUMNS.map((c) => (c === 'sha256' ? 'lower(hex(sha256)) AS sha256' : c)).join(', ')}`,
612
+ 'FROM read_parquet([',
613
+ paths.map((p) => ` ${sqlStr(p)}`).join(',\n'),
614
+ '], union_by_name = true);',
615
+ ].join('\n');
616
+ }
617
+ function fetchCommand(sel) {
618
+ const all = sel.length === IX.sources.length;
619
+ const tool = (f) => `curl -L -o ${f} ${fileUrl(IX.sha, `tools/${f}`)}`;
620
+ const args = [`python fetch.py --data hf://datasets/${REPO}@${IX.sha}`];
621
+ if (!all) for (const s of sel) args.push(`--source ${s.id}`);
622
+ args.push('--out ./fetched');
623
+ return [
624
+ '# pip install duckdb; add --dry-run to only list the rows',
625
+ tool('fetch.py'),
626
+ tool('sevenzip.py'),
627
+ args.join(' \\\n '),
628
+ ].join('\n');
629
+ }
630
+ const csvCell = (v) => {
631
+ if (v == null) return '';
632
+ const s = typeof v === 'bigint' ? v.toString() : String(v);
633
+ return /[",\n\r]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
634
+ };
635
+ async function buildCsv(sel, onStep) {
636
+ const files = sel.flatMap((s) => s.files);
637
+ const lines = [ROW_COLUMNS.join(',')];
638
+ let done = 0;
639
+ const queue = [...files];
640
+ const parts = new Map();
641
+ const worker = async () => {
642
+ while (queue.length) {
643
+ const f = queue.shift();
644
+ const { rows } = await readColumns(fileUrl(IX.sha, f.path), ROW_COLUMNS);
645
+ parts.set(f.path, rows);
646
+ done++;
647
+ onStep(done, files.length);
648
+ }
649
+ };
650
+ await Promise.all([worker(), worker(), worker()]);
651
+ for (const f of files) {
652
+ for (const r of parts.get(f.path)) lines.push(ROW_COLUMNS.map((c) => csvCell(c === 'sha256' ? hex(r[c]) : r[c])).join(','));
653
+ }
654
+ return lines.join('\n') + '\n';
655
+ }
656
+
657
+ function step(n, iconName, title, ...body) {
658
+ return el('li', { class: 'step' },
659
+ el('div', { class: 'step-head' }, el('span', { class: 'step-ico' }, icon(iconName, { size: 30 })),
660
+ el('span', { class: 'step-n' }, String(n)), el('h3', {}, title)),
661
+ el('div', { class: 'step-body' }, ...body));
662
+ }
663
+ function renderGet(sel) {
664
+ const rows = sel.reduce((a, s) => a + s.rows, 0);
665
+ $('get-h').textContent = sel.length ? `Take ${fmtCompact(rows)} rows` : 'Nothing to take';
666
+ const body = clear($('get-body'));
667
+ body.style.minHeight = '';
668
+ if (!sel.length) return;
669
+ const chips = filterChips();
670
+ const pick = step(1, 'pick', 'Pick',
671
+ el('p', { class: 'fact-line' }, icon('sources', { size: 18 }), el('b', {}, fmtInt(sel.length)), sel.length === 1 ? 'source' : 'sources'),
672
+ el('p', { class: 'fact-line' }, icon('rows', { size: 18 }), el('b', {}, fmtInt(rows)), 'rows'),
673
+ el('p', { class: 'chips' }, chips.length ? chips.map(([k, v]) => el('span', { class: 'chip', title: k }, v)) : el('span', { class: 'chip' }, 'All')),
674
+ el('a', { class: 'small', href: '#filters' }, 'Change'));
675
+ const code = UI.tab === 'sql' ? selectionSql(sel) : fetchCommand(sel);
676
+ const state = el('span', { class: 'small', 'aria-live': 'polite' });
677
+ const copy = step(2, 'copy', 'Copy',
678
+ segmented('', [['sql', 'SQL'], ['fetch', 'fetch.py']], UI.tab, (v) => { UI.tab = v; renderGet(selected()); }),
679
+ el('pre', { class: 'code' }, el('code', {}, code)),
680
+ el('div', { class: 'copy-row' }, el('button', { type: 'button', class: 'btn', onclick: async () => {
681
+ try { await navigator.clipboard.writeText(code); state.textContent = 'Copied'; } catch { state.textContent = 'Select the text'; }
682
+ setTimeout(() => { state.textContent = ''; }, 2400);
683
+ } }, icon('copy', { size: 16 }), ' Copy'), state));
684
+ const key = sel.map((s) => s.id).join(',');
685
+ const csvState = el('span', { class: 'small', 'aria-live': 'polite' }, UI.csv?.key === key ? UI.csv.text : '');
686
+ const tooMany = rows > CSV_MAX_ROWS;
687
+ const btn = el('button', { type: 'button', class: 'btn solid', disabled: tooMany, onclick: async () => {
688
+ btn.disabled = true;
689
+ const say = (t) => { csvState.textContent = t; UI.csv = { key, text: t }; };
690
+ try {
691
+ const csv = await buildCsv(sel, (d, n) => say(`${d}/${n} files`));
692
+ const blob = new Blob([csv], { type: 'text/csv' });
693
+ const a = el('a', { href: URL.createObjectURL(blob), download: `agentic-env-index-${IX.sha.slice(0, 7)}-${sel.length === 1 ? sel[0].id : `${sel.length}-sources`}.csv` });
694
+ document.body.append(a); a.click(); a.remove();
695
+ setTimeout(() => URL.revokeObjectURL(a.href), 10000);
696
+ say(`${fmtInt(csv.split('\n').length - 2)} rows`);
697
+ } catch (e) { say(e.message); }
698
+ btn.disabled = false;
699
+ } }, icon('csv', { size: 16 }), ' CSV file');
700
+ const run = step(3, 'run', 'Run',
701
+ el('p', { class: 'fact-line' }, icon('run', { size: 18 }), UI.tab === 'sql' ? 'DuckDB' : 'Terminal'),
702
+ el('div', { class: 'copy-row' }, btn, tooMany ? el('span', { class: 'small' }, `over ${fmtShort(CSV_MAX_ROWS)} rows: use ${UI.tab === 'sql' ? 'SQL' : 'fetch.py'}`) : csvState));
703
+ body.append(pick, copy, run);
704
+ }
705
+
706
+ // ---- not included (the whole index; below the scoped part)
707
+ function renderExcluded() {
708
+ const rows = IX.notIncluded;
709
+ const by = new Map();
710
+ for (const r of rows) by.set(r.category, (by.get(r.category) || 0) + 1);
711
+ const ranked = [...by].sort((a, b) => b[1] - a[1]);
712
+ const name = (k) => CATEGORY[k] || human(k);
713
+ $('excluded-h').textContent = rows.length ? `${fmtInt(rows.length)} left out` : 'Nothing left out';
714
+ const body = clear($('excluded-body'));
715
+ body.style.minHeight = '';
716
+ if (!rows.length) return;
717
+ const fig = el('figure', { class: 'chart', style: { maxWidth: '720px' } });
718
+ const items = ranked.map(([k, v]) => ({ key: k, label: name(k), value: v, text: fmtInt(v), tipLabel: IX.requirements.get(k) || name(k) }));
719
+ figureWithToggle(fig, 'excluded', 'Reason, by items', {
720
+ chart: () => hbars(items),
721
+ table: () => table(['Reason', 'Requirement', { label: 'Items', r: true }], ranked.map(([k, v]) => [name(k), IX.requirements.get(k) || '', fmtInt(v)])),
722
+ });
723
+ body.append(fig);
724
+ const list = [...rows].sort((a, b) => (a.category || '').localeCompare(b.category || '') || (a.source || '').localeCompare(b.source || ''));
725
+ body.append(el('details', { class: 'long' }, el('summary', {}, `All ${fmtInt(rows.length)}`),
726
+ el('div', { class: 'table-wrap' }, table(['Source', 'Unit', 'Requirement not met', 'Detail', 'Reviewed'],
727
+ list.map((r) => [r.source, r.level === 'source' ? 'whole source' : r.unit, IX.requirements.get(r.category) || name(r.category), r.detail || '', r.reviewed || ''])))));
728
+ }
729
+
730
+ // ---- header, request link, progress bar
731
+ function renderHeader() {
732
+ const link = $('pin-link');
733
+ link.textContent = IX.sha.slice(0, 7);
734
+ link.href = `${DATASET_URL}/tree/${IX.sha}`;
735
+ $('pin-date').textContent = IX.date ? IX.date.slice(0, 10) : '';
736
+ if (IX.overrides.catalog || IX.overrides.vocabulary) {
737
+ const chip = $('preview-chip');
738
+ clear(chip).append(el('span', { class: 'dot other', 'aria-hidden': 'true' }), 'Preview data');
739
+ chip.title = [IX.overrides.catalog, IX.overrides.vocabulary].filter(Boolean).join(' ');
740
+ chip.hidden = false;
741
+ }
742
+ }
743
+ function updateRequestLinks() {
744
+ const chips = filterChips();
745
+ const u = new URL(`${GITHUB}/issues/new`);
746
+ u.searchParams.set('template', 'data_request.yml');
747
+ if (chips.length) u.searchParams.set('what', `Like this catalog selection (${chips.map(([k, v]) => `${k}: ${v}`).join('; ')}), but: `);
748
+ for (const a of document.querySelectorAll('.request-link')) a.href = u.toString();
749
+ }
750
+ function decorate() {
751
+ for (const a of document.querySelectorAll('.progress a')) a.prepend(icon(a.dataset.icon, { size: 14 }));
752
+ for (const n of document.querySelectorAll('[data-icon]:not(a)')) n.append(icon(n.dataset.icon, { size: 28 }));
753
+ }
754
+
755
+ function progress() {
756
+ const parts = ['overview', 'browse', 'get', 'excluded'];
757
+ const line = 34 + 90;
758
+ let active = 0;
759
+ const frac = [];
760
+ parts.forEach((p, i) => {
761
+ const nodes = [...document.querySelectorAll(`main [data-part="${p}"], footer[data-part="${p}"]`)];
762
+ const top = Math.min(...nodes.map((n) => n.getBoundingClientRect().top));
763
+ const bottom = Math.max(...nodes.map((n) => n.getBoundingClientRect().bottom));
764
+ if (top <= line) active = i;
765
+ frac.push(Math.max(0, Math.min(1, (line - top) / Math.max(1, bottom - top))));
766
+ });
767
+ if (window.innerHeight + window.scrollY >= document.documentElement.scrollHeight - 4) { active = 3; frac[3] = 1; }
768
+ document.querySelectorAll('.progress a').forEach((a, i) => {
769
+ a.setAttribute('aria-current', i === active ? 'true' : 'false');
770
+ a.querySelector('.fill').style.width = `${Math.round((i < active ? 1 : i === active ? Math.max(0.04, frac[i]) : 0) * 100)}%`;
771
+ });
772
+ }
773
+
774
+ function renderScoped() {
775
+ hideTip();
776
+ const sel = selected();
777
+ syncFilterRow(sel);
778
+ renderMap();
779
+ renderSources(sel);
780
+ renderGet(sel);
781
+ updateRequestLinks();
782
+ }
783
+
784
+ async function main() {
785
+ readUrl();
786
+ decorate();
787
+ let ticking = false;
788
+ const onScroll = () => { if (!ticking) { ticking = true; requestAnimationFrame(() => { ticking = false; progress(); }); } };
789
+ addEventListener('scroll', onScroll, { passive: true });
790
+ addEventListener('resize', onScroll);
791
+ progress();
792
+ try {
793
+ IX = await loadIndex(new URLSearchParams(location.search), (msg) => { const n = $('notice'); n.textContent = msg; n.hidden = false; });
794
+ } catch (e) {
795
+ const n = $('notice');
796
+ clear(n).append(el('span', { class: 'dot problem', 'aria-hidden': 'true' }), `Could not read the index: ${e.message}`);
797
+ n.hidden = false;
798
+ console.error(e);
799
+ return;
800
+ }
801
+ renderHeader();
802
+ renderOverview();
803
+ renderHistory();
804
+ fillFilterRow();
805
+ for (const b of $('metric-seg').querySelectorAll('button')) {
806
+ b.addEventListener('click', () => { F.metric = b.dataset.metric; writeUrl(); renderMap(); });
807
+ }
808
+ if (F.domain && F.domain !== NONE && F.domain !== MULTI && F.leaf) UI.expanded.add(F.domain);
809
+ renderScoped();
810
+ renderExcluded();
811
+ progress();
812
+ if (F.open) document.getElementById(`d-${F.open}`)?.scrollIntoView({ block: 'center' });
813
+ }
814
+
815
+ main();
charts.js ADDED
@@ -0,0 +1,303 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Small chart kit: elements, number formats, tooltip, chart/table toggle, thin bars, heatmap.
2
+ // Every string from the data goes in with textContent (through el), never innerHTML.
3
+
4
+ export function el(tag, attrs = {}, ...kids) {
5
+ const n = document.createElement(tag);
6
+ for (const [k, v] of Object.entries(attrs || {})) {
7
+ if (v == null || v === false) continue;
8
+ if (k === 'class') n.className = v;
9
+ else if (k === 'style' && typeof v === 'object') Object.assign(n.style, v);
10
+ else if (k.startsWith('on') && typeof v === 'function') n.addEventListener(k.slice(2), v);
11
+ else if (k === 'dataset') Object.assign(n.dataset, v);
12
+ else n.setAttribute(k, v === true ? '' : String(v));
13
+ }
14
+ for (const c of kids.flat(Infinity)) {
15
+ if (c == null || c === false) continue;
16
+ n.append(c instanceof Node ? c : document.createTextNode(String(c)));
17
+ }
18
+ return n;
19
+ }
20
+
21
+ export const clear = (n) => { while (n.firstChild) n.firstChild.remove(); return n; };
22
+
23
+ // ---- numbers
24
+ const intFmt = new Intl.NumberFormat('en-US');
25
+ export const fmtInt = (v) => (v == null ? 'unknown' : intFmt.format(Math.round(v)));
26
+ export function fmtCompact(v) {
27
+ if (v == null) return 'unknown';
28
+ const a = Math.abs(v);
29
+ const three = (x) => (x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2));
30
+ if (a >= 1e9) return `${three(v / 1e9)} billion`;
31
+ if (a >= 1e6) return `${three(v / 1e6)} million`;
32
+ if (a >= 1e4) return `${three(v / 1e3)} thousand`;
33
+ return intFmt.format(v);
34
+ }
35
+ export function fmtShort(v) {
36
+ if (v == null) return '';
37
+ const a = Math.abs(v);
38
+ const two = (x) => (x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(0) : x.toFixed(1));
39
+ if (a >= 1e9) return `${two(v / 1e9)}B`;
40
+ if (a >= 1e6) return `${two(v / 1e6)}M`;
41
+ if (a >= 1e3) return `${two(v / 1e3)}K`;
42
+ return String(Math.round(v));
43
+ }
44
+ export function fmtBytes(v) {
45
+ if (v == null) return 'unknown';
46
+ const units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB'];
47
+ let i = 0;
48
+ let x = v;
49
+ while (x >= 1000 && i < units.length - 1) { x /= 1000; i++; }
50
+ const s = i === 0 ? String(x) : x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
51
+ return `${s} ${units[i]}`;
52
+ }
53
+ export const fmtPct = (part, whole) => {
54
+ if (!whole) return '0%';
55
+ const p = (100 * part) / whole;
56
+ return `${p > 0 && p < 1 ? '<1' : Math.round(p)}%`;
57
+ };
58
+
59
+ // ---- tooltip: the value leads, the label follows; also shown on keyboard focus
60
+ const tip = () => document.getElementById('tip');
61
+ export function showTip(target, value, ...labels) {
62
+ const t = tip();
63
+ clear(t).append(el('span', { class: 'v' }, value), ...labels.filter(Boolean).map((l) => el('span', { class: 'k' }, l)));
64
+ t.hidden = false;
65
+ const r = target.getBoundingClientRect();
66
+ const w = t.offsetWidth;
67
+ const h = t.offsetHeight;
68
+ let x = r.left + r.width / 2 - w / 2;
69
+ let y = r.top - h - 8;
70
+ if (y < 44) y = r.bottom + 8;
71
+ x = Math.max(8, Math.min(x, window.innerWidth - w - 8));
72
+ t.style.left = `${x}px`;
73
+ t.style.top = `${y}px`;
74
+ }
75
+ export function hideTip() { tip().hidden = true; }
76
+ export function withTip(node, value, ...labels) {
77
+ const on = () => showTip(node, typeof value === 'function' ? value() : value, ...labels);
78
+ node.addEventListener('pointerenter', on);
79
+ node.addEventListener('focus', on);
80
+ node.addEventListener('pointerleave', hideTip);
81
+ node.addEventListener('blur', hideTip);
82
+ return node;
83
+ }
84
+
85
+ // ---- a segmented control that shows its state next to itself
86
+ export function segmented(label, options, current, onPick) {
87
+ const id = `seg-${Math.random().toString(36).slice(2, 8)}`;
88
+ const group = el('span', { class: 'seg', role: 'group', 'aria-labelledby': id });
89
+ for (const [value, text] of options) {
90
+ group.append(el('button', { type: 'button', 'aria-pressed': value === current ? 'true' : 'false', onclick: () => onPick(value) }, text));
91
+ }
92
+ return el('span', { class: 'seg-wrap' }, el('span', { class: 'seg-label', id }, label), group);
93
+ }
94
+
95
+ /** A figure with a title and a Chart / Table toggle. views: {chart: () => Node, table: () => Node}. */
96
+ const viewState = new Map();
97
+ export function figureWithToggle(fig, key, title, views, extra) {
98
+ const mode = viewState.get(key) || 'chart';
99
+ const draw = () => figureWithToggle(fig, key, title, views, extra);
100
+ clear(fig).append(
101
+ el('div', { class: 'chart-head' },
102
+ el('p', { class: 'chart-title' }, title),
103
+ segmented('View', [['chart', 'Chart'], ['table', 'Table']], mode, (v) => { viewState.set(key, v); draw(); })),
104
+ mode === 'chart' ? views.chart() : el('div', { class: 'table-wrap' }, views.table()),
105
+ );
106
+ const more = extra ? extra() : null;
107
+ if (more) fig.append(more);
108
+ }
109
+
110
+ export function table(headers, rows, caption) {
111
+ return el('table', { class: 'data' },
112
+ caption ? el('caption', {}, caption) : null,
113
+ el('thead', {}, el('tr', {}, headers.map((h) => el('th', { class: h.r ? 'r' : null, scope: 'col' }, h.label ?? h)))),
114
+ el('tbody', {}, rows.map((r) => el('tr', {}, r.map((c, i) => el('td', { class: headers[i]?.r ? 'r' : null }, c))))));
115
+ }
116
+
117
+ /** Thin horizontal bars, one color, value at the tip. items: [{key, label, value, text, leaf, current,
118
+ * expanded, onClick, tipLabel}] */
119
+ export function hbars(items, { max } = {}) {
120
+ const top = max ?? Math.max(0, ...items.map((d) => d.value || 0));
121
+ const list = el('ul', { class: 'bars' });
122
+ for (const d of items) {
123
+ const w = top > 0 && d.value > 0 ? Math.max(2, (100 * d.value) / top) : 0;
124
+ const caret = d.expanded == null ? null : el('span', { class: 'caret', 'aria-hidden': 'true' }, d.expanded ? '▾' : '▸');
125
+ const inner = [
126
+ el('span', { class: 'bar-label' }, caret, d.icon || null, el('span', { class: 'name' }, d.label)),
127
+ el('span', { class: 'bar-track' },
128
+ el('span', { class: 'bar', style: { width: `calc((100% - 6rem) * ${w / 100})` } }),
129
+ el('span', { class: 'bar-value' }, d.text)),
130
+ ];
131
+ const row = d.onClick
132
+ ? el('button', { type: 'button', class: `bar-row${d.current ? ' current' : ''}`, onclick: d.onClick, 'aria-expanded': d.expanded == null ? null : String(d.expanded), 'aria-current': d.current ? 'true' : null }, inner)
133
+ : el('div', { class: `bar-row${d.current ? ' current' : ''}` }, inner);
134
+ if (d.tipLabel) withTip(row, d.text, ...[].concat(d.tipLabel));
135
+ list.append(el('li', { class: d.leaf ? 'leaf' : null }, row));
136
+ }
137
+ return list;
138
+ }
139
+
140
+ // ---- sequential scale: six steps of the slate-blue ramp on log bins; zero has no fill
141
+ export const RAMP = ['#f4f6f7', '#d2dade', '#b0bdc6', '#8ea2ae', '#6f8797', '#4f6d80'];
142
+ export function logBins(values) {
143
+ const pos = values.filter((v) => v > 0);
144
+ if (!pos.length) return { color: () => null, edges: [] };
145
+ const lo = Math.min(...pos);
146
+ const hi = Math.max(...pos);
147
+ const n = RAMP.length;
148
+ const edges = [];
149
+ for (let i = 0; i <= n; i++) edges.push(hi === lo ? lo : Math.exp(Math.log(lo) + ((Math.log(hi) - Math.log(lo)) * i) / n));
150
+ const color = (v) => {
151
+ if (!(v > 0)) return null;
152
+ if (hi === lo) return RAMP[n - 1];
153
+ let i = Math.floor(((Math.log(v) - Math.log(lo)) / (Math.log(hi) - Math.log(lo))) * n);
154
+ i = Math.max(0, Math.min(n - 1, i));
155
+ return RAMP[i];
156
+ };
157
+ return { color, edges, lo, hi };
158
+ }
159
+ // ink on light steps, white on the two darkest (both clear 4.5:1)
160
+ export const inkOn = (hex) => (hex === RAMP[5] || hex === RAMP[4] ? '#ffffff' : '#1a1a1a');
161
+
162
+ /** Heatmap as a CSS grid of buttons with a roving tab stop. rows/cols: [{key, label, head}] (head: a node shown as the
163
+ * column header, such as a pictogram; the label stays in the tooltip, the aria-label and the legend); value(r, c) ->
164
+ * number; fmt(v) -> text; onPick(r, c); current {row, col}. */
165
+ export function heatmap({ rows, cols, value, fmt, onPick, current, metricName }) {
166
+ const vals = [];
167
+ for (const r of rows) for (const c of cols) vals.push(value(r, c));
168
+ const scale = logBins(vals);
169
+ const maxV = Math.max(0, ...vals);
170
+ const grid = el('div', { class: 'heat', role: 'grid', 'aria-label': `${metricName} by domain and form`,
171
+ style: { gridTemplateColumns: `minmax(min-content, 10rem) repeat(${cols.length}, minmax(22px, 3rem))` } });
172
+ grid.append(el('div', { class: 'corner', role: 'presentation' }));
173
+ for (const c of cols) {
174
+ const h = el('div', { class: 'col-head', role: 'columnheader', 'aria-label': c.label, tabindex: '-1' }, c.head || c.label);
175
+ withTip(h, c.label);
176
+ grid.append(h);
177
+ }
178
+ const cells = [];
179
+ let labelled = false;
180
+ rows.forEach((r, ri) => {
181
+ grid.append(el('div', { class: 'row-head', role: 'rowheader' }, r.label));
182
+ cols.forEach((c, ci) => {
183
+ const v = value(r, c);
184
+ const fill = scale.color(v);
185
+ const isCur = current && current.row === r.key && current.col === c.key;
186
+ const cell = el('button', {
187
+ type: 'button', class: `cell${fill ? '' : ' empty'}${isCur ? ' current' : ''}`, role: 'gridcell', tabindex: '-1',
188
+ 'aria-label': `${r.label}, ${c.label}: ${fmt(v)}`, 'aria-pressed': isCur ? 'true' : 'false',
189
+ style: fill ? { background: fill } : null,
190
+ onclick: () => onPick(r, c),
191
+ });
192
+ if (!labelled && v > 0 && v === maxV) {
193
+ labelled = true;
194
+ cell.append(el('span', { class: 'lab', style: { color: inkOn(fill) }, 'aria-hidden': 'true' }, fmtShort(v)));
195
+ }
196
+ withTip(cell, fmt(v), `${r.label} × ${c.label}`);
197
+ cell.dataset.ri = ri;
198
+ cell.dataset.ci = ci;
199
+ cells.push(cell);
200
+ grid.append(cell);
201
+ });
202
+ });
203
+ const start = cells.find((c) => c.classList.contains('current')) || cells[0];
204
+ if (start) start.tabIndex = 0;
205
+ grid.addEventListener('keydown', (e) => {
206
+ const t = e.target.closest('.cell');
207
+ if (!t) return;
208
+ const ri = Number(t.dataset.ri);
209
+ const ci = Number(t.dataset.ci);
210
+ const move = { ArrowRight: [0, 1], ArrowLeft: [0, -1], ArrowDown: [1, 0], ArrowUp: [-1, 0] }[e.key];
211
+ if (!move) return;
212
+ e.preventDefault();
213
+ const nr = Math.max(0, Math.min(rows.length - 1, ri + move[0]));
214
+ const nc = Math.max(0, Math.min(cols.length - 1, ci + move[1]));
215
+ const next = cells[nr * cols.length + nc];
216
+ t.tabIndex = -1;
217
+ next.tabIndex = 0;
218
+ next.focus();
219
+ });
220
+
221
+ const steps = el('span', { class: 'steps', 'aria-hidden': 'true' },
222
+ el('span', { class: 'zero', style: { background: '#ffffff' } }), RAMP.map((h) => el('span', { style: { background: h } })));
223
+ const ticks = el('span', { class: 'ticks', 'aria-hidden': 'true' },
224
+ el('span', {}, '0'), scale.edges.slice(0, RAMP.length).map((e) => el('span', {}, fmtShort(e))));
225
+ const legend = el('div', { class: 'scale' },
226
+ el('span', {}, `${metricName}, log scale`),
227
+ el('span', {}, steps, ticks));
228
+ return { grid, legend };
229
+ }
230
+
231
+ // ---- small graphics for facts
232
+ const SVG = 'http://www.w3.org/2000/svg';
233
+ function svg(tag, attrs = {}) {
234
+ const n = document.createElementNS(SVG, tag);
235
+ for (const [k, v] of Object.entries(attrs)) n.setAttribute(k, String(v));
236
+ return n;
237
+ }
238
+
239
+ /** A ring of parts [{value, color, label}] with a 2px surface gap between them. */
240
+ export function ring(parts, { size = 120, width = 14, label } = {}) {
241
+ const total = parts.reduce((a, p) => a + p.value, 0) || 1;
242
+ const r = (size - width) / 2;
243
+ const c = 2 * Math.PI * r;
244
+ const s = svg('svg', { viewBox: `0 0 ${size} ${size}`, width: size, height: size, role: 'img', 'aria-label': label || '' });
245
+ s.append(svg('circle', { cx: size / 2, cy: size / 2, r, fill: 'none', stroke: '#e6e6e6', 'stroke-width': width }));
246
+ let at = 0;
247
+ const shown = parts.filter((p) => p.value > 0);
248
+ for (const p of shown) {
249
+ const len = (p.value / total) * c;
250
+ const gap = shown.length > 1 ? Math.min(2, len / 2) : 0;
251
+ const arc = svg('circle', {
252
+ cx: size / 2, cy: size / 2, r, fill: 'none', stroke: p.color, 'stroke-width': width,
253
+ 'stroke-dasharray': `${Math.max(0.5, len - gap)} ${c}`, 'stroke-dashoffset': -at,
254
+ transform: `rotate(-90 ${size / 2} ${size / 2})`,
255
+ });
256
+ at += len;
257
+ s.append(arc);
258
+ }
259
+ return s;
260
+ }
261
+
262
+ /** A sparkline of values with a dot at the end. */
263
+ export function spark(values, { w = 72, h = 22 } = {}) {
264
+ const s = svg('svg', { viewBox: `0 0 ${w} ${h}`, width: w, height: h, 'aria-hidden': 'true', class: 'spark' });
265
+ if (values.length < 1) return s;
266
+ const lo = Math.min(...values);
267
+ const hi = Math.max(...values);
268
+ const x = (i) => (values.length === 1 ? w - 4 : 4 + (i * (w - 8)) / (values.length - 1));
269
+ const y = (v) => (hi === lo ? h / 2 : h - 4 - ((v - lo) * (h - 8)) / (hi - lo));
270
+ if (values.length > 1) {
271
+ s.append(svg('polyline', { points: values.map((v, i) => `${x(i)},${y(v)}`).join(' '), fill: 'none', stroke: '#4f6d80',
272
+ 'stroke-width': 2, 'stroke-linecap': 'round', 'stroke-linejoin': 'round' }));
273
+ }
274
+ s.append(svg('circle', { cx: x(values.length - 1), cy: y(values[values.length - 1]), r: 4, fill: '#4f6d80', stroke: '#fff', 'stroke-width': 2 }));
275
+ return s;
276
+ }
277
+
278
+ /** A tiny bar: share filled in slate blue on a gray track. */
279
+ export function meter(share, { w = 72 } = {}) {
280
+ return el('span', { class: 'meter', style: { width: `${w}px` }, 'aria-hidden': 'true' },
281
+ el('span', { style: { width: `${Math.max(0, Math.min(1, share)) * 100}%` } }));
282
+ }
283
+
284
+ /** Vertical columns: [{key, label, value, top, sub}]; value labels on the caps, label and sub below. */
285
+ export function columns(items) {
286
+ const max = Math.max(0, ...items.map((d) => d.value));
287
+ return el('div', { class: 'cols' }, items.map((d) => withTip(el('div', { class: 'col', tabindex: '0' },
288
+ el('span', { class: 'col-top' }, d.top),
289
+ el('span', { class: 'col-track' }, el('span', { class: 'col-bar', style: { height: `${max ? Math.max(2, (100 * d.value) / max) : 0}%` } })),
290
+ el('span', { class: 'col-label' }, d.label),
291
+ d.sub ? el('span', { class: 'col-sub' }, d.sub) : null), d.top, d.label)));
292
+ }
293
+
294
+ /** One thin stacked bar of parts [{value, color, label}] with its short legend. */
295
+ export function stack(parts, total) {
296
+ const shown = parts.filter((p) => p.value > 0);
297
+ return el('div', {},
298
+ el('div', { class: 'stack', role: 'img', 'aria-label': shown.map((p) => `${p.label} ${fmtPct(p.value, total)}`).join(', ') },
299
+ shown.map((p) => withTip(el('span', { tabindex: '0', style: { flex: `${p.value} 1 0`, background: p.color } }),
300
+ fmtInt(p.value), `${p.label}: ${fmtPct(p.value, total)}`))),
301
+ el('div', { class: 'stack-labels' }, shown.map((p) => el('span', {},
302
+ el('span', { class: 'sw', style: { background: p.color } }), `${p.label} `, el('b', {}, fmtPct(p.value, total))))));
303
+ }
data.js ADDED
@@ -0,0 +1,294 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Reading the index in the browser. Every file comes from one commit of the dataset, so every number agrees.
2
+ import { asyncBufferFromUrl, parquetMetadataAsync, parquetReadObjects } from 'https://cdn.jsdelivr.net/npm/hyparquet@1.31.5/+esm';
3
+ import { compressors } from 'https://cdn.jsdelivr.net/npm/hyparquet-compressors@1.1.2/+esm';
4
+
5
+ export const REPO = 'Battam/agentic-env-index';
6
+ export const HF = 'https://huggingface.co';
7
+ export const DATASET_URL = `${HF}/datasets/${REPO}`;
8
+
9
+ export const fileUrl = (sha, path) => `${DATASET_URL}/resolve/${sha}/${path}`;
10
+ export const num = (v) => (v == null ? null : typeof v === 'bigint' ? Number(v) : Number(v));
11
+
12
+ async function getJson(url) {
13
+ const r = await fetch(url);
14
+ if (!r.ok) throw new Error(`${url}: HTTP ${r.status}`);
15
+ return r.json();
16
+ }
17
+
18
+ async function getText(url) {
19
+ const r = await fetch(url);
20
+ if (!r.ok) throw new Error(`${url}: HTTP ${r.status}`);
21
+ return r.text();
22
+ }
23
+
24
+ /** A small file read whole (the meta tables): one request instead of a size probe and ranges. */
25
+ export async function readParquet(url, options = {}) {
26
+ const r = await fetch(url);
27
+ if (!r.ok) throw new Error(`${url}: HTTP ${r.status}`);
28
+ const file = await r.arrayBuffer();
29
+ return parquetReadObjects({ file, compressors, utf8: false, ...options });
30
+ }
31
+
32
+ export async function parquetRows(url) {
33
+ const file = await asyncBufferFromUrl({ url });
34
+ return num((await parquetMetadataAsync(file)).num_rows);
35
+ }
36
+
37
+ /** The columns of cols that the file has, read with hyparquet (byte ranges); range: {rowStart, rowEnd}. */
38
+ export async function readColumns(url, cols, range = {}) {
39
+ const file = await asyncBufferFromUrl({ url });
40
+ const metadata = await parquetMetadataAsync(file);
41
+ const names = new Set(metadata.schema.slice(1).map((s) => s.name));
42
+ const columns = cols.filter((c) => names.has(c));
43
+ const rows = await parquetReadObjects({ file, metadata, columns, compressors, utf8: false, ...range });
44
+ return { rows, total: num(metadata.num_rows) };
45
+ }
46
+
47
+ /** The commit to read: ?rev=<sha or branch>, else main. Returns {sha, date, files}. */
48
+ export async function loadRevision(rev) {
49
+ const j = await getJson(`${HF}/api/datasets/${REPO}/revision/${encodeURIComponent(rev || 'main')}`);
50
+ if (!/^[0-9a-f]{40}$/.test(j.sha || '')) throw new Error('the Hub gave no commit id');
51
+ return { sha: j.sha, date: j.lastModified || null, files: (j.siblings || []).map((s) => s.rfilename) };
52
+ }
53
+
54
+ /** Batch folders of one source with the date of their last commit (Hub tree API). */
55
+ export async function loadBatchDates(sha, source) {
56
+ const j = await getJson(`${HF}/api/datasets/${REPO}/tree/${sha}/data/${encodeURIComponent(source)}?expand=true`);
57
+ const out = new Map();
58
+ for (const e of j) {
59
+ if (e.type !== 'directory') continue;
60
+ out.set(e.path.split('/').pop(), e.lastCommit?.date || null);
61
+ }
62
+ return out;
63
+ }
64
+
65
+ // ---- the classification contract (#252): domain is a list of codes (D1 to D10, with leaves), or multi, or
66
+ // unclassified; form is one id; names come from meta/vocabulary.json. Without them everything shows as unclassified.
67
+ const pick = (o, keys) => {
68
+ if (!o || typeof o !== 'object') return undefined;
69
+ for (const k of keys) if (o[k] != null) return o[k];
70
+ return undefined;
71
+ };
72
+ const NAME_KEYS = ['name', 'label', 'title', 'en'];
73
+ const CODE_KEYS = ['code', 'id', 'key', 'value'];
74
+ const KIDS_KEYS = ['children', 'leaves', 'subdomains', 'items'];
75
+ const ABOUT_KEYS = ['description', 'about', 'definition'];
76
+
77
+ function vocabSection(node) {
78
+ const names = new Map();
79
+ const about = new Map();
80
+ const order = [];
81
+ const add = (code, name, text) => {
82
+ if (code == null) return;
83
+ code = String(code);
84
+ if (!names.has(code)) order.push(code);
85
+ if (typeof name === 'string' && name.trim()) names.set(code, name.trim());
86
+ else if (!names.has(code)) names.set(code, null);
87
+ if (typeof text === 'string' && text.trim()) about.set(code, text.trim());
88
+ };
89
+ const walk = (n) => {
90
+ if (!n) return;
91
+ if (Array.isArray(n)) {
92
+ for (const it of n) {
93
+ if (typeof it === 'string') add(it, null);
94
+ else if (it && typeof it === 'object') {
95
+ add(pick(it, CODE_KEYS), pick(it, NAME_KEYS), pick(it, ABOUT_KEYS));
96
+ walk(pick(it, KIDS_KEYS));
97
+ }
98
+ }
99
+ } else if (typeof n === 'object') {
100
+ for (const [k, v] of Object.entries(n)) {
101
+ if (typeof v === 'string') add(k, v);
102
+ else if (v && typeof v === 'object') {
103
+ add(k, pick(v, NAME_KEYS), pick(v, ABOUT_KEYS));
104
+ walk(pick(v, KIDS_KEYS));
105
+ } else add(k, null);
106
+ }
107
+ }
108
+ };
109
+ walk(node);
110
+ return { names, about, order };
111
+ }
112
+
113
+ export function parseVocabulary(j) {
114
+ return { domain: vocabSection(pick(j, ['domain', 'domains'])), form: vocabSection(pick(j, ['form', 'forms'])) };
115
+ }
116
+
117
+ export const topOf = (code) => String(code).split('.')[0];
118
+
119
+ function domainOf(v) {
120
+ let list = v;
121
+ if (typeof list === 'string') list = [list];
122
+ if (list && typeof list.toArray === 'function') list = list.toArray();
123
+ list = Array.isArray(list) ? list.map(String).map((s) => s.trim()).filter(Boolean) : [];
124
+ if (!list.length || list.includes('unclassified')) return { kind: 'none', codes: [] };
125
+ if (list.includes('multi')) return { kind: 'multi', codes: [] };
126
+ return { kind: 'codes', codes: [...new Set(list)] };
127
+ }
128
+
129
+ // ---- small parsers of the index's own text files
130
+ const intOf = (s) => Number(String(s).replace(/,/g, ''));
131
+
132
+ /** CHANGELOG.md -> [{date, title, kind, sourcesAdded, sourcesRemoved, rowsAdded, rowsRemoved, sources, rows}] */
133
+ export function parseChangelog(md) {
134
+ const out = [];
135
+ const parts = md.split(/^## /m).slice(1);
136
+ for (const part of parts) {
137
+ const [head, ...lines] = part.split('\n');
138
+ const m = head.match(/^(\d{4}-\d{2}-\d{2}):\s*(.+)$/);
139
+ if (!m) continue;
140
+ const e = { date: m[1], title: m[2].trim(), kind: 'other', sourcesAdded: null, sourcesRemoved: null, rowsAdded: 0, rowsRemoved: 0 };
141
+ const body = lines.join('\n');
142
+ const start = body.match(/lists ([\d,]+) sources with ([\d,]+) rows/);
143
+ if (start) {
144
+ e.kind = 'start';
145
+ e.sources = intOf(start[1]);
146
+ e.rows = intOf(start[2]);
147
+ }
148
+ const sa = body.match(/^- Sources added \((\d+)\)/m);
149
+ const sr = body.match(/^- Sources removed \((\d+)\)/m);
150
+ if (sa || sr) {
151
+ e.kind = 'update';
152
+ e.sourcesAdded = sa ? Number(sa[1]) : 0;
153
+ e.sourcesRemoved = sr ? Number(sr[1]) : 0;
154
+ }
155
+ for (const r of body.matchAll(/^\s+- \S+: .*?\(([\d,]+) to ([\d,]+)\)\s*$/gm)) {
156
+ const d = intOf(r[2]) - intOf(r[1]);
157
+ if (d > 0) e.rowsAdded += d;
158
+ else e.rowsRemoved -= d;
159
+ }
160
+ out.push(e);
161
+ }
162
+ return out;
163
+ }
164
+
165
+ /** README.md, section "Not included, and why": category -> requirement sentence. */
166
+ export function parseRequirements(md) {
167
+ const out = new Map();
168
+ const sec = md.split(/^## /m).find((s) => s.startsWith('Not included'));
169
+ if (!sec) return out;
170
+ for (const m of sec.matchAll(/^\| `([a-z0-9_]+)` \| (.+?) \| \d+ \| \d+ \|\s*$/gm)) out.set(m[1], m[2].trim());
171
+ return out;
172
+ }
173
+
174
+ const LINK_RE = /^(\d{4}-\d{2}-\d{2}): ([\d,]+) of ([\d,]+) reachable/;
175
+ function linkOf(text) {
176
+ if (!text) return { kind: 'unchecked' };
177
+ const m = String(text).match(LINK_RE);
178
+ if (m) {
179
+ const ok = intOf(m[2]);
180
+ const total = intOf(m[3]);
181
+ return { kind: ok >= total ? 'ok' : 'partial', ok, total, date: m[1] };
182
+ }
183
+ if (/^needs /.test(text)) return { kind: 'login', text: String(text) };
184
+ return { kind: 'other', text: String(text) };
185
+ }
186
+
187
+ /** What every row of a source allows, from sources.json license_rows: 'yes' (every row), 'cond' (yes or
188
+ * conditional for every row) or 'no' (some rows are no or unknown). */
189
+ function permission(lr) {
190
+ if (!lr) return 'no';
191
+ const keys = Object.entries(lr).filter(([, v]) => num(v?.rows) > 0).map(([k]) => k);
192
+ if (!keys.length) return 'no';
193
+ if (keys.every((k) => k === 'yes')) return 'yes';
194
+ if (keys.every((k) => k === 'yes' || k === 'conditional')) return 'cond';
195
+ return 'no';
196
+ }
197
+
198
+ async function readCatalog(url) {
199
+ if (/\.json(\?|$)/i.test(url)) {
200
+ const j = await getJson(url);
201
+ return Array.isArray(j) ? j : j.sources || j.rows || [];
202
+ }
203
+ return readParquet(url);
204
+ }
205
+
206
+ /** Everything the page shows, from one commit. params: URLSearchParams (rev, catalog, vocabulary). */
207
+ export async function loadIndex(params, onNote) {
208
+ const rev = await loadRevision(params.get('rev'));
209
+ const at = (p) => fileUrl(rev.sha, p);
210
+ const catalogUrl = params.get('catalog') || at('meta/catalog.parquet');
211
+ const has = (p) => rev.files.includes(p);
212
+ const vocabUrl = params.get('vocabulary') || (has('meta/vocabulary.json') ? at('meta/vocabulary.json') : null);
213
+
214
+ const [catalog, sourcesJson, coverage, batches, notIncluded, changelog, readme, vocabJson] = await Promise.all([
215
+ readCatalog(catalogUrl),
216
+ getJson(at('meta/sources.json')),
217
+ getJson(at('meta/coverage.json')),
218
+ readParquet(at('meta/batches.parquet')).catch(() => []),
219
+ readParquet(at('meta/not_included.parquet')).catch(() => []),
220
+ getText(at('CHANGELOG.md')).catch(() => ''),
221
+ getText(at('README.md')).catch(() => ''),
222
+ vocabUrl ? getJson(vocabUrl).catch((e) => { onNote?.(`The vocabulary could not be read (${e.message}); codes are shown as they are.`); return null; }) : null,
223
+ ]);
224
+
225
+ const vocab = parseVocabulary(vocabJson || {});
226
+ const srcInfo = new Map((sourcesJson.sources || []).map((e) => [e.source, e]));
227
+ const cov = coverage.by_source || {};
228
+ const batchesBy = new Map();
229
+ for (const b of batches) {
230
+ if (!batchesBy.has(b.source)) batchesBy.set(b.source, []);
231
+ batchesBy.get(b.source).push(b);
232
+ }
233
+ const filesBy = new Map();
234
+ for (const f of rev.files) {
235
+ const m = f.match(/^data\/([^/]+)\/([^/]+)\/[^/]+\.parquet$/);
236
+ if (!m) continue;
237
+ if (!filesBy.has(m[1])) filesBy.set(m[1], []);
238
+ filesBy.get(m[1]).push({ path: f, batch: m[2] });
239
+ }
240
+
241
+ const sources = catalog.map((r) => {
242
+ const info = srcInfo.get(r.source) || {};
243
+ const lic = r.license || r.licence || {};
244
+ const dom = domainOf(r.domain);
245
+ const rows = num(r.files ?? info.rows) || 0;
246
+ const bytes = num(r.bytes);
247
+ const form = r.form == null || r.form === '' || r.form === 'unclassified' ? null : String(r.form);
248
+ return {
249
+ id: r.source,
250
+ title: r.title || r.source,
251
+ description: r.description || info.description || '',
252
+ summary: r.summary || '',
253
+ unit: r.unit || '',
254
+ status: r.status || null,
255
+ reasons: Array.isArray(r.status_reasons) ? r.status_reasons : r.status_reasons?.toArray?.() || [],
256
+ domainKind: dom.kind,
257
+ codes: dom.codes,
258
+ tops: [...new Set(dom.codes.map(topOf))],
259
+ form,
260
+ tier: lic.tier || 'unknown',
261
+ ship: lic.ship || null,
262
+ spdx: lic.spdx || null,
263
+ licenceNote: lic.note || null,
264
+ contamination: r.contamination || {},
265
+ pii: r.pii || {},
266
+ rows,
267
+ bytes: bytes && bytes > 0 ? bytes : null,
268
+ rowsNoSize: num(r.files_without_size) || 0,
269
+ topTypes: Array.isArray(r.top_types) ? r.top_types : r.top_types?.toArray?.() || [],
270
+ needsAuth: r.needs_auth || null,
271
+ link: linkOf(r.link_check),
272
+ linkText: r.link_check || null,
273
+ train: permission(info.license_rows?.training_ok),
274
+ redist: permission(info.license_rows?.redistribution_ok),
275
+ info,
276
+ cov: cov[r.source] || null,
277
+ batches: batchesBy.get(r.source) || [],
278
+ files: (filesBy.get(r.source) || []).sort((a, b) => a.path.localeCompare(b.path)),
279
+ };
280
+ });
281
+
282
+ return {
283
+ sha: rev.sha,
284
+ date: rev.date,
285
+ overrides: { catalog: params.get('catalog'), vocabulary: params.get('vocabulary') },
286
+ hasVocabulary: Boolean(vocabJson),
287
+ vocab,
288
+ sources,
289
+ coverage,
290
+ notIncluded,
291
+ requirements: parseRequirements(readme),
292
+ changelog: parseChangelog(changelog),
293
+ };
294
+ }
icons.js ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Pictograms: one stroke style (24 px grid, 1.6 px round strokes, no fill), drawn in currentColor.
2
+ // Built with createElementNS, never from markup strings.
3
+ const NS = 'http://www.w3.org/2000/svg';
4
+
5
+ // each glyph: a list of [tag, attributes]
6
+ const P = (d) => ['path', { d }];
7
+ const R = (x, y, w, h, rx = 1.5) => ['rect', { x, y, width: w, height: h, rx }];
8
+ const Ci = (cx, cy, r) => ['circle', { cx, cy, r }];
9
+ const L = (x1, y1, x2, y2) => ['line', { x1, y1, x2, y2 }];
10
+
11
+ const G = {
12
+ // the four parts of the page
13
+ overview: [Ci(12, 12, 8.5), P('M12 7.5v4.5l3 2')],
14
+ browse: [R(4, 4, 7, 7), R(13, 4, 7, 7), R(4, 13, 7, 7), R(13, 13, 7, 7)],
15
+ get: [P('M12 4v11'), P('M7.5 10.5 12 15l4.5-4.5'), P('M5 19.5h14')],
16
+ excluded: [Ci(12, 12, 8.5), L(6, 18, 18, 6)],
17
+ // facts and steps
18
+ sources: [['ellipse', { cx: 12, cy: 6.5, rx: 7, ry: 2.5 }], P('M5 6.5v11c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5v-11'), P('M5 12c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5')],
19
+ rows: [L(9, 6, 20, 6), L(9, 12, 20, 12), L(9, 18, 20, 18), Ci(5, 6, 1), Ci(5, 12, 1), Ci(5, 18, 1)],
20
+ size: [R(3.5, 7, 17, 10, 2), L(7, 12, 13, 12), Ci(16.5, 12, 1)],
21
+ snapshot: [R(4, 5.5, 16, 14.5, 2), L(4, 10, 20, 10), L(8, 3.5, 8, 7), L(16, 3.5, 16, 7), P('M12 12.5v5M9.5 15h5')],
22
+ pick: [P('M4 5h16l-6 7.5V19l-4-2v-4.5z')],
23
+ copy: [R(8, 8, 12, 12, 2), P('M16 8V5.5A1.5 1.5 0 0 0 14.5 4h-9A1.5 1.5 0 0 0 4 5.5v9A1.5 1.5 0 0 0 5.5 16H8')],
24
+ run: [R(3.5, 4.5, 17, 15, 2), P('M7.5 10l3 2.5-3 2.5'), L(12.5, 15.5, 16.5, 15.5)],
25
+ csv: [P('M6 3.5h8l4 4V20.5H6z'), P('M14 3.5v4h4'), L(9, 12, 15, 12), L(9, 15, 15, 15), L(12, 10, 12, 18)],
26
+ training: [P('M12 4l8 4-8 4-8-4z'), P('M7 10v4.5c0 1.4 2.2 2.5 5 2.5s5-1.1 5-2.5V10'), L(20, 8, 20, 13)],
27
+ share: [Ci(6, 12, 2.2), Ci(18, 6, 2.2), Ci(18, 18, 2.2), L(8, 11, 16, 7), L(8, 13, 16, 17)],
28
+ link: [P('M10 14a4 4 0 0 0 5.7 0l3-3a4 4 0 0 0-5.7-5.7l-1 1'), P('M14 10a4 4 0 0 0-5.7 0l-3 3a4 4 0 0 0 5.7 5.7l1-1')],
29
+ person: [Ci(12, 8, 3.5), P('M5 20c.8-3.6 3.6-5.5 7-5.5s6.2 1.9 7 5.5')],
30
+ target: [Ci(12, 12, 8.5), Ci(12, 12, 4.5), Ci(12, 12, 0.8)],
31
+ flagged: [P('M6 21V4'), P('M6 4.5h11l-2 4 2 4H6')],
32
+ request: [Ci(12, 12, 8.5), L(12, 8, 12, 16), L(8, 12, 16, 12)],
33
+ // forms
34
+ tickets: [P('M4 7h16v3a2 2 0 0 0 0 4v3H4v-3a2 2 0 0 0 0-4z'), L(14, 7.5, 14, 16.5)],
35
+ meetings: [Ci(8, 8, 2.5), Ci(16, 8, 2.5), P('M3.5 18c.5-2.8 2.3-4.3 4.5-4.3s4 1.5 4.5 4.3'), P('M11.5 18c.5-2.8 2.3-4.3 4.5-4.3s4 1.5 4.5 4.3')],
36
+ chat: [P('M4 5.5h16v10H10l-4 3.5v-3.5H4z')],
37
+ mail: [R(3.5, 5.5, 17, 13, 2), P('M4 7l8 6 8-6')],
38
+ doc: [P('M6 3.5h8l4 4V20.5H6z'), P('M14 3.5v4h4'), L(9, 12, 15, 12), L(9, 15.5, 15, 15.5)],
39
+ book: [P('M4 5.5c2.5-1 5.5-1 8 .8 2.5-1.8 5.5-1.8 8-.8v13c-2.5-1-5.5-1-8 .8-2.5-1.8-5.5-1.8-8-.8z'), L(12, 6.3, 12, 19.3)],
40
+ package: [P('M12 3.5l8 4v9l-8 4-8-4v-9z'), P('M4 7.5l8 4 8-4'), L(12, 11.5, 12, 20.5)],
41
+ log: [R(4, 3.5, 16, 17, 2), L(7.5, 8, 16.5, 8), L(7.5, 12, 14, 12), L(7.5, 16, 16.5, 16)],
42
+ terminal: [R(3.5, 4.5, 17, 15, 2), P('M7.5 10l3 2.5-3 2.5'), L(12.5, 15.5, 16.5, 15.5)],
43
+ code: [P('M9 7l-5 5 5 5'), P('M15 7l5 5-5 5'), L(13.5, 5, 10.5, 19)],
44
+ shield: [P('M12 3.5l7.5 3v5.5c0 4.5-3.2 7.5-7.5 8.5-4.3-1-7.5-4-7.5-8.5V6.5z'), L(12, 8.5, 12, 12.5), Ci(12, 15.5, 0.6)],
45
+ rule: [P('M12 3.5l7.5 3v5.5c0 4.5-3.2 7.5-7.5 8.5-4.3-1-7.5-4-7.5-8.5V6.5z'), P('M8.5 12l2.5 2.5 4.5-5')],
46
+ flag: [P('M6 21V4'), P('M6 4.5h11l-2 4 2 4H6')],
47
+ search: [Ci(10.5, 10.5, 6), L(15, 15, 20, 20)],
48
+ fileWarn: [P('M6 3.5h8l4 4V20.5H6z'), P('M14 3.5v4h4'), L(12, 10.5, 12, 14.5), Ci(12, 17.3, 0.6)],
49
+ layers: [P('M12 4l8.5 4.5L12 13 3.5 8.5z'), P('M3.5 12.5L12 17l8.5-4.5'), P('M3.5 16.5L12 21l8.5-4.5')],
50
+ pawn: [Ci(12, 7, 2.8), P('M9.5 12.5h5l1.5 5.5H8z'), L(6.5, 20.5, 17.5, 20.5)],
51
+ quill: [P('M19.5 4.5c-6 .5-10.5 5-12 13'), P('M19.5 4.5c-1 5.5-5 9.5-10.5 10.5'), L(4.5, 20, 7.5, 17.5)],
52
+ question: [P('M4 5.5h16v10H10l-4 3.5v-3.5H4z'), P('M10 9a2 2 0 1 1 2.8 1.8c-.5.3-.8.7-.8 1.2'), Ci(12, 13.6, 0.5)],
53
+ forum: [P('M3.5 5h11v7.5H8L5 15v-2.5H3.5z'), P('M17.5 9.5h3v7.5H19V19l-3-2h-5.5v-2.5')],
54
+ social: [Ci(12, 12, 3.5), P('M15.5 12v1.5a2.5 2.5 0 0 0 5 0V12a8.5 8.5 0 1 0-3.3 6.7')],
55
+ wiki: [P('M6 3.5h8l4 4V20.5H6z'), P('M8.5 11l1.5 5 2-4 2 4 1.5-5')],
56
+ globe: [Ci(12, 12, 8.5), L(3.5, 12, 20.5, 12), P('M12 3.5c2.5 2.5 3.5 5.5 3.5 8.5s-1 6-3.5 8.5c-2.5-2.5-3.5-5.5-3.5-8.5s1-6 3.5-8.5')],
57
+ database: [['ellipse', { cx: 12, cy: 6.5, rx: 7, ry: 2.5 }], P('M5 6.5v11c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5v-11'), P('M5 12c0 1.4 3.1 2.5 7 2.5s7-1.1 7-2.5')],
58
+ stream: [P('M3.5 9c2.5-2.5 4.5-2.5 6 0s3.5 2.5 6 0 3.5-2.5 5 0'), P('M3.5 15c2.5-2.5 4.5-2.5 6 0s3.5 2.5 6 0 3.5-2.5 5 0')],
59
+ bank: [P('M3.5 9.5L12 4l8.5 5.5z'), L(6, 10.5, 6, 17), L(10, 10.5, 10, 17), L(14, 10.5, 14, 17), L(18, 10.5, 18, 17), L(3.5, 19.5, 20.5, 19.5)],
60
+ tests: [R(4, 4, 16, 16, 2), P('M7.5 9l1.5 1.5 3-3'), L(14, 9, 17, 9), P('M7.5 15l1.5 1.5 3-3'), L(14, 15, 17, 15)],
61
+ flask: [P('M9.5 3.5h5'), P('M10 3.5v6l-5 8.5a1.7 1.7 0 0 0 1.5 2.5h11a1.7 1.7 0 0 0 1.5-2.5L14 9.5v-6'), L(7.5, 15, 16.5, 15)],
62
+ chip: [R(7, 7, 10, 10, 1.5), L(10, 3.5, 10, 7), L(14, 3.5, 14, 7), L(10, 17, 10, 20.5), L(14, 17, 14, 20.5), L(3.5, 10, 7, 10), L(3.5, 14, 7, 14), L(17, 10, 20.5, 10), L(17, 14, 20.5, 14)],
63
+ tag: [P('M3.5 12.5V4h8.5l8.5 8.5-8 8z'), Ci(8, 8.5, 1.3)],
64
+ chart: [L(4, 20, 20, 20), L(7, 16.5, 7, 12), L(12, 16.5, 12, 6), L(17, 16.5, 17, 9.5)],
65
+ repeat: [P('M4 11a8 8 0 0 1 14-5.3'), P('M18.5 3v3.5H15'), P('M20 13a8 8 0 0 1-14 5.3'), P('M5.5 21v-3.5H9')],
66
+ calendar: [R(4, 5.5, 16, 14.5, 2), L(4, 10, 20, 10), L(8, 3.5, 8, 7), L(16, 3.5, 16, 7), Ci(9, 14, 0.6), Ci(12, 14, 0.6), Ci(15, 14, 0.6)],
67
+ briefcase: [R(3.5, 7.5, 17, 12, 2), P('M9 7.5V5.5h6v2'), L(3.5, 12.5, 20.5, 12.5)],
68
+ clipboard: [R(5, 5, 14, 16, 2), R(9, 3, 6, 4, 1), P('M8.5 12l2 2 4-4'), L(8.5, 17, 15.5, 17)],
69
+ sheet: [R(4, 4, 16, 16, 2), L(4, 9.5, 20, 9.5), L(4, 14.5, 20, 14.5), L(10, 4, 10, 20)],
70
+ disk: [Ci(12, 12, 8.5), Ci(12, 12, 2.5)],
71
+ archive: [R(3.5, 4.5, 17, 4.5, 1), P('M5 9v10.5h14V9'), L(10, 13, 14, 13)],
72
+ unknown: [Ci(12, 12, 8.5), P('M9.6 9.5a2.5 2.5 0 1 1 3.5 2.3c-.7.3-1.1.9-1.1 1.7'), Ci(12, 16.6, 0.6)],
73
+ more: [Ci(6, 12, 1.2), Ci(12, 12, 1.2), Ci(18, 12, 1.2)],
74
+ };
75
+
76
+ // form id -> glyph (ids of the index's vocabulary; an unknown id gets the generic document)
77
+ const FORM = {
78
+ tickets: 'tickets', meetings: 'meetings', chat: 'chat', 'mailing-lists': 'mail', 'document-collections': 'doc',
79
+ publications: 'book', packages: 'package', 'build-ci-logs': 'log', 'system-logs': 'log', 'source-code': 'code',
80
+ 'terminal-sessions': 'terminal', vulnerabilities: 'shield', ctf: 'flag', forensics: 'search',
81
+ 'file-format-research': 'fileWarn', 'model-hubs': 'layers', chess: 'pawn', 'interactive-fiction': 'quill',
82
+ 'qa-sites': 'question', forums: 'forum', 'social-media': 'social', 'wiki-talk': 'wiki', 'open-data': 'globe',
83
+ 'dataset-repositories': 'database', 'event-streams': 'stream', 'benchmark-instances': 'target',
84
+ 'regulatory-filings': 'bank', 'test-suites': 'tests', 'detection-rules': 'rule', 'scientific-reference-data': 'flask',
85
+ 'hardware-designs': 'chip', 'annotated-data': 'tag', 'evaluation-results': 'chart', reproductions: 'repeat',
86
+ calendars: 'calendar', 'business-documents': 'briefcase', 'rules-with-worked-examples': 'clipboard',
87
+ 'spreadsheets-with-answers': 'sheet', 'disk-images': 'disk', 'mixed-archives': 'archive',
88
+ _none: 'unknown', _other: 'more',
89
+ };
90
+
91
+ export function icon(name, { size = 20, label = null, cls = '' } = {}) {
92
+ const s = document.createElementNS(NS, 'svg');
93
+ s.setAttribute('viewBox', '0 0 24 24');
94
+ s.setAttribute('width', String(size));
95
+ s.setAttribute('height', String(size));
96
+ s.setAttribute('fill', 'none');
97
+ s.setAttribute('stroke', 'currentColor');
98
+ s.setAttribute('stroke-width', '1.6');
99
+ s.setAttribute('stroke-linecap', 'round');
100
+ s.setAttribute('stroke-linejoin', 'round');
101
+ s.setAttribute('class', `ico ${cls}`.trim());
102
+ if (label) {
103
+ s.setAttribute('role', 'img');
104
+ s.setAttribute('aria-label', label);
105
+ } else s.setAttribute('aria-hidden', 'true');
106
+ for (const [tag, attrs] of G[name] || G.doc) {
107
+ const n = document.createElementNS(NS, tag);
108
+ for (const [k, v] of Object.entries(attrs)) n.setAttribute(k, String(v));
109
+ s.append(n);
110
+ }
111
+ return s;
112
+ }
113
+
114
+ export const formIcon = (id, opts) => icon(FORM[id] || 'doc', opts);
index.html CHANGED
@@ -1,19 +1,152 @@
1
  <!doctype html>
2
- <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
16
- </p>
17
- </div>
18
- </body>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  </html>
 
1
  <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>Agentic Env Catalog</title>
7
+ <link rel="icon" href="data:,">
8
+ <meta name="description" content="What the public pointer index Battam/agentic-env-index holds, by domain and by form, and how to take a part of it.">
9
+ <link rel="preconnect" href="https://fonts.googleapis.com">
10
+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
11
+ <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@500&family=Noto+Sans+SC:wght@400;500;700&display=swap">
12
+ <link rel="stylesheet" href="style.css">
13
+ <script type="module" src="app.js"></script>
14
+ </head>
15
+ <body>
16
+ <nav class="progress" aria-label="Page sections">
17
+ <ol>
18
+ <li><a href="#overview" data-part="overview" data-icon="overview"><span class="pl">Overview</span><span class="track"><span class="fill"></span></span></a></li>
19
+ <li><a href="#map" data-part="browse" data-icon="browse"><span class="pl">Browse</span><span class="track"><span class="fill"></span></span></a></li>
20
+ <li><a href="#get" data-part="get" data-icon="get"><span class="pl">Get the data</span><span class="track"><span class="fill"></span></span></a></li>
21
+ <li><a href="#excluded" data-part="excluded" data-icon="excluded"><span class="pl">Not included</span><span class="track"><span class="fill"></span></span></a></li>
22
+ </ol>
23
+ </nav>
24
+
25
+ <header class="top wrap" id="top">
26
+ <h1>Agentic Env Catalog</h1>
27
+ <p class="meta-line">
28
+ <span class="pin-wrap">Index <a class="pin" id="pin-link" href="https://huggingface.co/datasets/Battam/agentic-env-index">loading</a> <span id="pin-date"></span></span>
29
+ <span class="chip" id="preview-chip" hidden></span>
30
+ <a href="https://huggingface.co/datasets/Battam/agentic-env-index">Dataset</a>
31
+ <a href="https://github.com/msra-sg/agentic-env-pipeline/blob/main/catalog/README.md">Catalog README</a>
32
+ <a class="request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
33
+ </p>
34
+ <p class="notice problem-text" id="notice" hidden></p>
35
+ </header>
36
+
37
+ <main>
38
+ <div data-part="overview">
39
+ <section class="unit wrap" id="overview" aria-labelledby="overview-h">
40
+ <h2 id="overview-h">&nbsp;</h2>
41
+ <div class="overview-grid">
42
+ <div>
43
+ <div class="hero">
44
+ <div class="big" id="hero-big">&nbsp;</div>
45
+ <div class="big-label">pointer rows</div>
46
+ </div>
47
+ <ul class="facts" id="facts"><li>&nbsp;</li><li>&nbsp;</li><li>&nbsp;</li></ul>
48
+ </div>
49
+ <figure class="chart" id="licence-chart" style="min-height: 190px"></figure>
50
+ </div>
51
+ </section>
52
+
53
+ <section class="unit wrap" id="history" aria-labelledby="history-h">
54
+ <h2 id="history-h">&nbsp;</h2>
55
+ <figure class="chart" id="history-fig" style="min-height: 210px"></figure>
56
+ </section>
57
+ </div>
58
+
59
+ <div class="scoped" id="scoped">
60
+ <div class="filters" id="filters" role="region" aria-label="Filters">
61
+ <div class="wrap">
62
+ <button class="filter-toggle" id="filter-toggle" type="button" aria-expanded="false" aria-controls="filter-row">
63
+ <span>Filters</span><span class="state" id="filter-state">none on</span>
64
+ </button>
65
+ <div class="filter-row" id="filter-row">
66
+ <label>Domain<select id="f-domain"><option value="">All</option></select></label>
67
+ <label>Subdomain<select id="f-leaf" disabled><option value="">All</option></select></label>
68
+ <label>Form<select id="f-form"><option value="">All</option></select></label>
69
+ <label>Training<select id="f-train">
70
+ <option value="">Any</option>
71
+ <option value="yes">Allowed, all rows</option>
72
+ <option value="cond">Allowed or conditional</option>
73
+ </select></label>
74
+ <label>Redistribution<select id="f-redist">
75
+ <option value="">Any</option>
76
+ <option value="yes">Allowed, all rows</option>
77
+ <option value="cond">Allowed or conditional</option>
78
+ </select></label>
79
+ <label>Licence tier<select id="f-tier"><option value="">All</option></select></label>
80
+ <label>Source size<select id="f-size">
81
+ <option value="">Any</option>
82
+ <option value="lt1g">Under 1 GB</option>
83
+ <option value="1g1t">1 GB to 1 TB</option>
84
+ <option value="ge1t">1 TB or more</option>
85
+ <option value="unknown">Not published</option>
86
+ </select></label>
87
+ <label>Status<select id="f-status">
88
+ <option value="">Any</option>
89
+ <option value="ok">No issues</option>
90
+ <option value="flagged">Flagged</option>
91
+ </select></label>
92
+ <label class="wide">Search<input id="f-q" type="search" placeholder="Title or text" autocomplete="off"></label>
93
+ <button class="clear" id="f-clear" type="button" disabled>Clear</button>
94
+ <span class="count" id="f-count" aria-live="polite">&nbsp;</span>
95
+ </div>
96
+ </div>
97
+ </div>
98
+
99
+ <div data-part="browse">
100
+ <section class="unit wrap" id="map" aria-labelledby="map-h">
101
+ <h2 id="map-h">&nbsp;</h2>
102
+ <div class="chart-head">
103
+ <span class="seg-wrap"><span class="seg-label" id="metric-label">Count</span>
104
+ <span class="seg" role="group" aria-labelledby="metric-label" id="metric-seg">
105
+ <button type="button" data-metric="sources" aria-pressed="true">Sources</button>
106
+ <button type="button" data-metric="rows" aria-pressed="false">Pointer rows</button>
107
+ <button type="button" data-metric="bytes" aria-pressed="false">Size</button>
108
+ </span>
109
+ </span>
110
+ </div>
111
+ <div class="map-grid">
112
+ <figure class="chart" id="heat-fig" style="min-height: 420px"></figure>
113
+ <div class="side">
114
+ <figure class="chart block" id="dom-fig" style="min-height: 200px"></figure>
115
+ <figure class="chart block" id="form-fig" style="min-height: 120px"></figure>
116
+ </div>
117
+ </div>
118
+ </section>
119
+
120
+ <section class="unit wrap" id="sources" aria-labelledby="sources-h">
121
+ <h2 id="sources-h">&nbsp;</h2>
122
+ <div id="sources-body" style="min-height: 240px"></div>
123
+ </section>
124
+ </div>
125
+
126
+ <div data-part="get">
127
+ <section class="unit wrap" id="get" aria-labelledby="get-h">
128
+ <h2 id="get-h">&nbsp;</h2>
129
+ <ol id="get-body" class="steps" style="min-height: 320px"></ol>
130
+ </section>
131
+ </div>
132
+ </div>
133
+
134
+ <div data-part="excluded">
135
+ <section class="unit wrap" id="excluded" aria-labelledby="excluded-h">
136
+ <h2 id="excluded-h">&nbsp;</h2>
137
+ <div id="excluded-body" style="min-height: 220px"></div>
138
+ </section>
139
+ </div>
140
+ </main>
141
+
142
+ <footer class="request" id="request" data-part="excluded">
143
+ <div class="wrap request-row">
144
+ <span class="request-ico" data-icon="request"></span>
145
+ <h2>Missing data?</h2>
146
+ <a class="btn solid request-link" href="https://github.com/msra-sg/agentic-env-pipeline/issues/new?template=data_request.yml">Request data</a>
147
+ </div>
148
+ </footer>
149
+
150
+ <div class="tip" id="tip" role="tooltip" hidden></div>
151
+ </body>
152
  </html>
portal.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "commit": "c57c7ccec904fcc350c34df62b424ccf186f21f8",
3
+ "digest": "7e4a5ec6e9e718ecf4bffd856914e5b9869cfe30f8c9ced8a6b4818ebebd76ad",
4
+ "files": 7,
5
+ "format": "portal-deploy/1"
6
+ }
style.css CHANGED
@@ -1,28 +1,298 @@
1
- body {
2
- padding: 2rem;
3
- font-family: -apple-system, BlinkMacSystemFont, "Arial", sans-serif;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  }
5
 
6
- h1 {
7
- font-size: 16px;
8
- margin-top: 0;
9
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
 
11
- p {
12
- color: rgb(107, 114, 128);
13
- font-size: 15px;
14
- margin-bottom: 10px;
15
- margin-top: 5px;
 
 
 
 
16
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
 
18
- .card {
19
- max-width: 620px;
20
- margin: 0 auto;
21
- padding: 16px;
22
- border: 1px solid lightgray;
23
- border-radius: 16px;
24
  }
 
 
 
 
 
 
 
 
 
 
 
25
 
26
- .card p:last-child {
27
- margin-bottom: 0;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  }
 
1
+ /* Colors carry meaning only: slate blue = ours / worth remembering, berry = a problem, gray = others or a baseline.
2
+ The page is light only. Graphics first: pictograms in one stroke style, short labels. */
3
+ :root {
4
+ color-scheme: light;
5
+ --bg: #ffffff;
6
+ --ink: #1a1a1a;
7
+ --ink-2: #4d4d4d;
8
+ --ink-muted: #767676;
9
+ --grid: #e6e6e6;
10
+ --block: #f5f5f5;
11
+ --ours: #4f6d80;
12
+ --problem: #b98b86;
13
+ --problem-text: #966a66;
14
+ --fill-3: #d7cea3;
15
+ --other: #8c8c8c;
16
+ --link: #1d5ab0;
17
+ --visited: #ab3276;
18
+ --sans: "Noto Sans SC", system-ui, "PingFang SC", "Microsoft YaHei", sans-serif;
19
+ --mono: "IBM Plex Mono", ui-monospace, "Cascadia Mono", Consolas, monospace;
20
+ --page: 1180px;
21
+ --pad: 24px;
22
+ --progress-h: 36px;
23
  }
24
 
25
+ * { box-sizing: border-box; }
26
+ [hidden] { display: none !important; }
27
+ html { scroll-padding-top: calc(var(--progress-h) + 72px); }
28
+ body { margin: 0; background: var(--bg); color: var(--ink); font-family: var(--sans); font-size: 16px; line-height: 1.5; -webkit-text-size-adjust: 100%; }
29
+ a { color: var(--link); text-underline-offset: 2px; }
30
+ a:visited { color: var(--visited); }
31
+ button, input, select { font: inherit; color: inherit; }
32
+ :focus-visible { outline: 2px solid var(--ink); outline-offset: 2px; }
33
+ .visually-hidden { position: absolute !important; width: 1px; height: 1px; padding: 0; margin: -1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; border: 0; }
34
+ .wrap { max-width: var(--page); margin: 0 auto; padding-left: var(--pad); padding-right: var(--pad); }
35
+ .mono, code, pre { font-family: var(--mono); font-weight: 500; }
36
+ .small { font-size: 12px; color: var(--ink-2); }
37
+ .muted { color: var(--ink-muted); }
38
+ .problem-text { color: var(--problem-text); }
39
+ .ico { flex: none; vertical-align: -0.2em; color: var(--ours); }
40
+ .dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; vertical-align: 1px; margin-right: 5px; flex: none; }
41
+ .dot.problem { background: var(--problem); }
42
+ .dot.ours { background: var(--ours); }
43
+ .dot.other { background: var(--other); }
44
+ .chip { display: inline-flex; align-items: center; gap: 2px; font-size: 12px; color: var(--ink-2); border: 1px solid var(--grid); border-radius: 999px; padding: 1px 9px; }
45
 
46
+ /* ---- the slim four-part progress bar, with a pictogram per part ---- */
47
+ .progress { position: sticky; top: 0; z-index: 30; background: var(--bg); height: var(--progress-h); }
48
+ .progress ol {
49
+ list-style: none; margin: 0 auto; padding: 0 var(--pad); max-width: var(--page);
50
+ display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 6px; height: 100%; align-items: end;
51
+ }
52
+ .progress a {
53
+ display: grid; grid-template-columns: auto minmax(0, 1fr); align-items: center; column-gap: 5px; text-decoration: none;
54
+ color: var(--ink-muted); font-size: 12px; line-height: 1; padding: 8px 0 6px;
55
  }
56
+ .progress a .ico { color: var(--other); }
57
+ .progress a .pl { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
58
+ .progress a:visited { color: var(--ink-muted); }
59
+ .progress a[aria-current="true"] { color: var(--ink); font-weight: 500; }
60
+ .progress a[aria-current="true"] .ico { color: var(--ours); }
61
+ .progress .track { grid-column: 1 / -1; display: block; height: 3px; background: var(--grid); margin-top: 6px; border-radius: 2px; overflow: hidden; }
62
+ .progress .fill { display: block; height: 100%; width: 0; background: var(--ours); }
63
+ @media (prefers-reduced-motion: no-preference) { .progress .fill { transition: width 120ms linear; } }
64
+
65
+ /* ---- header ---- */
66
+ header.top { padding-top: 36px; padding-bottom: 0; }
67
+ header.top h1 { font-size: 32px; line-height: 1.25; margin: 0 0 8px; font-weight: 700; }
68
+ .meta-line { display: flex; flex-wrap: wrap; gap: 6px 18px; font-size: 14px; color: var(--ink-2); align-items: center; margin: 0; }
69
+ .meta-line .pin { font-family: var(--mono); font-weight: 500; font-size: 13px; }
70
+ .notice { margin: 10px 0 0; font-size: 14px; }
71
+
72
+ /* ---- sections: one idea each ---- */
73
+ section.unit { padding-top: 64px; padding-bottom: 64px; min-height: 320px; display: flex; flex-direction: column; justify-content: center; }
74
+ .unit + .unit { border-top: 1px solid var(--grid); }
75
+ .unit h2 { font-size: 22px; line-height: 1.3; margin: 0 0 24px; font-weight: 700; min-height: 1.3em; }
76
+ .overview-grid { display: grid; grid-template-columns: minmax(0, 1.2fr) minmax(0, 1fr); gap: 32px 56px; align-items: center; }
77
+ .hero { margin: 0 0 24px; }
78
+ .hero .big { font-size: 64px; line-height: 1; font-weight: 700; color: var(--ours); min-height: 1em; }
79
+ .hero .big-label { font-size: 14px; color: var(--ink-2); margin-top: 6px; }
80
+ .facts { list-style: none; margin: 0; padding: 0; display: flex; flex-wrap: wrap; gap: 18px 36px; min-height: 52px; }
81
+ .fact { display: flex; align-items: center; gap: 10px; }
82
+ .fact .ico { color: var(--ours); }
83
+ .fact-text { display: flex; flex-direction: column; line-height: 1.2; }
84
+ .fact-text b { font-size: 20px; font-weight: 700; }
85
+ .fact-text span { font-size: 12px; color: var(--ink-2); }
86
+ .fact-meter { display: flex; flex-direction: column; gap: 3px; }
87
+ .meter { display: block; height: 6px; background: var(--grid); border-radius: 3px; overflow: hidden; }
88
+ .meter > span { display: block; height: 100%; background: var(--ours); }
89
+ .spark { display: block; overflow: visible; }
90
+
91
+ /* ring with its legend */
92
+ .ring-wrap { display: flex; align-items: center; gap: 24px; flex-wrap: wrap; }
93
+ .ring-box { position: relative; width: 132px; height: 132px; flex: none; }
94
+ .ring-center { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; }
95
+ .ring-legend { list-style: none; margin: 0; padding: 0; display: grid; gap: 4px; font-size: 13px; color: var(--ink-2); }
96
+ .ring-legend li { display: grid; grid-template-columns: 10px auto auto; gap: 8px; align-items: center; min-height: 24px; }
97
+ .ring-legend b { color: var(--ink); font-weight: 500; font-variant-numeric: tabular-nums; }
98
+ .sw { display: inline-block; width: 10px; height: 10px; border-radius: 2px; margin-right: 6px; vertical-align: -1px; }
99
+ .ring-legend .sw { margin-right: 0; }
100
+
101
+ /* columns (history) */
102
+ .cols { display: flex; align-items: flex-end; gap: 28px; min-height: 170px; padding-top: 4px; }
103
+ .col { display: grid; grid-template-rows: auto 120px auto auto; justify-items: center; width: 56px; outline-offset: 4px; }
104
+ .col-top { font-size: 13px; font-weight: 500; font-variant-numeric: tabular-nums; }
105
+ .col-track { display: flex; align-items: flex-end; height: 120px; width: 24px; }
106
+ .col-bar { display: block; width: 24px; background: var(--ours); border-radius: 4px 4px 0 0; }
107
+ .col-label { font-size: 12px; color: var(--ink-2); margin-top: 6px; font-variant-numeric: tabular-nums; border-top: 1px solid var(--grid); width: 100%; text-align: center; padding-top: 4px; }
108
+ .col-sub { font-size: 12px; color: var(--ours); font-weight: 500; }
109
+
110
+ /* ---- chart chrome ---- */
111
+ figure.chart { margin: 0; }
112
+ .chart-head { display: flex; flex-wrap: wrap; justify-content: space-between; gap: 8px 16px; align-items: center; margin-bottom: 12px; }
113
+ .chart-title { font-size: 14px; font-weight: 500; margin: 0 0 6px; }
114
+ .chart-head .chart-title { margin: 0; }
115
+ .seg-wrap { display: inline-flex; align-items: center; flex-wrap: wrap; gap: 4px 6px; }
116
+ .seg-label { font-size: 12px; color: var(--ink-muted); }
117
+ .seg-label:empty { display: none; }
118
+ .seg { display: inline-flex; border: 1px solid var(--grid); border-radius: 6px; padding: 2px; gap: 2px; flex-wrap: wrap; }
119
+ .seg button { border: 0; background: transparent; border-radius: 4px; padding: 2px 10px; min-height: 28px; font-size: 13px; color: var(--ink-2); cursor: pointer; }
120
+ .seg button[aria-pressed="true"] { background: var(--ours); color: #fff; font-weight: 500; }
121
+ .seg button:hover:not([aria-pressed="true"]) { background: var(--block); }
122
+
123
+ table.data { border-collapse: collapse; width: 100%; font-size: 14px; }
124
+ table.data caption { text-align: left; font-size: 12px; color: var(--ink-2); padding-bottom: 6px; }
125
+ table.data th, table.data td { text-align: left; padding: 6px 12px 6px 0; border-bottom: 1px solid var(--grid); vertical-align: top; }
126
+ table.data th { font-weight: 500; color: var(--ink-2); font-size: 12px; }
127
+ table.data td.r, table.data th.r { text-align: right; font-variant-numeric: tabular-nums; }
128
+ .table-wrap { max-width: 100%; overflow-x: auto; }
129
+
130
+ /* stacked bar */
131
+ .stack { display: flex; height: 10px; gap: 2px; margin: 4px 0 0; max-width: 520px; }
132
+ .stack > span { display: block; height: 100%; min-width: 2px; }
133
+ .stack > span:first-child { border-radius: 4px 0 0 4px; }
134
+ .stack > span:last-child { border-radius: 0 4px 4px 0; }
135
+ .stack-labels { display: flex; flex-wrap: wrap; gap: 2px 14px; font-size: 12px; color: var(--ink-2); margin: 4px 0 8px; }
136
+ .stack-labels b { color: var(--ink); font-weight: 500; }
137
 
138
+ /* horizontal thin bars */
139
+ .bars { list-style: none; margin: 0; padding: 0; }
140
+ .bars li { margin: 0; }
141
+ .bar-row {
142
+ display: grid; grid-template-columns: minmax(0, 11rem) minmax(0, 1fr); gap: 2px 12px; align-items: center;
143
+ width: 100%; padding: 3px 0; border: 0; background: none; text-align: left; min-height: 28px; font-size: 13px;
144
  }
145
+ button.bar-row { cursor: pointer; }
146
+ button.bar-row:hover .bar-label span.name { text-decoration: underline; }
147
+ .bar-label { display: flex; align-items: center; gap: 6px; color: var(--ink); line-height: 1.25; overflow-wrap: anywhere; min-width: 0; }
148
+ .bar-label .ico { color: var(--ours); }
149
+ .bar-label .caret { display: inline-block; width: 0.8em; color: var(--ink-muted); flex: none; }
150
+ .bar-track { display: flex; align-items: center; gap: 8px; min-width: 0; }
151
+ .bar { display: block; height: 10px; background: var(--ours); border-radius: 0 4px 4px 0; flex: none; }
152
+ .bar-value { font-size: 12px; color: var(--ink-2); white-space: nowrap; font-variant-numeric: tabular-nums; }
153
+ .bars .leaf .bar-row { padding-left: 1.3em; }
154
+ .bars .leaf .bar-label { color: var(--ink-2); }
155
+ .bar-row.current .bar-label { font-weight: 700; }
156
 
157
+ /* heatmap: pictograms as column heads, horizontal row labels */
158
+ .map-grid { display: grid; grid-template-columns: minmax(0, 1.4fr) minmax(0, 1fr); gap: 40px; align-items: start; }
159
+ .heat { display: grid; gap: 2px; width: 100%; max-width: 720px; }
160
+ .heat .col-head { display: flex; align-items: flex-end; justify-content: center; padding: 0 0 6px; min-height: 30px; }
161
+ .heat .col-head .ico { color: var(--ours); }
162
+ .heat .row-head { font-size: 13px; line-height: 1.15; color: var(--ink); align-self: center; padding-right: 8px; }
163
+ .heat .cell { border: 0; padding: 0; margin: 0; min-height: 32px; min-width: 0; border-radius: 2px; cursor: pointer; position: relative; font-size: 11px; font-weight: 500; }
164
+ .heat .cell.empty { background: var(--bg); box-shadow: inset 0 0 0 1px var(--grid); }
165
+ .heat .cell:hover, .heat .cell:focus-visible { outline: 2px solid var(--ink); outline-offset: 0; z-index: 1; }
166
+ .heat .cell.current { outline: 2px solid var(--ink); outline-offset: 1px; z-index: 1; }
167
+ .heat .cell .lab { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; pointer-events: none; }
168
+ .icon-key { list-style: none; margin: 12px 0 0; padding: 0; display: flex; flex-wrap: wrap; gap: 4px 14px; font-size: 12px; color: var(--ink-2); }
169
+ .icon-key li { display: inline-flex; align-items: center; gap: 5px; }
170
+ .scale { display: flex; align-items: center; gap: 8px 12px; margin-top: 10px; font-size: 12px; color: var(--ink-2); flex-wrap: wrap; }
171
+ .scale .steps { display: grid; grid-template-columns: repeat(7, 34px); gap: 2px; }
172
+ .scale .steps span { display: block; height: 10px; border-radius: 2px; }
173
+ .scale .steps span.zero { box-shadow: inset 0 0 0 1px var(--grid); }
174
+ .scale .ticks { display: grid; grid-template-columns: repeat(7, 34px); gap: 2px; font-size: 11px; color: var(--ink-2); font-variant-numeric: tabular-nums; }
175
+ .scale .ticks span { white-space: nowrap; }
176
+ .side .block + .block { margin-top: 28px; }
177
+
178
+ /* filters: one row above everything they scope */
179
+ .filters { position: sticky; top: var(--progress-h); z-index: 20; background: var(--bg); border-top: 1px solid var(--grid); border-bottom: 1px solid var(--grid); }
180
+ .filters .wrap { padding-top: 10px; padding-bottom: 10px; }
181
+ .filter-toggle { display: none; }
182
+ .filter-row { display: flex; flex-wrap: wrap; gap: 8px; align-items: flex-end; }
183
+ .filter-row label { display: flex; flex-direction: column; font-size: 12px; color: var(--ink-2); gap: 2px; }
184
+ .filter-row select, .filter-row input { min-height: 32px; border: 1px solid #b8b8b8; border-radius: 6px; background: var(--bg); padding: 3px 4px; font-size: 13px; min-width: 5.5rem; max-width: 8.5rem; }
185
+ .filter-row input[type="search"] { width: 9rem; max-width: 9rem; }
186
+ .filter-row select:disabled { color: var(--ink-muted); background: var(--block); }
187
+ .filter-row .count { margin-left: auto; font-size: 13px; color: var(--ink-2); align-self: center; white-space: nowrap; }
188
+ .filter-row button.clear { border: 0; background: none; color: var(--link); text-decoration: underline; padding: 6px 0; cursor: pointer; font-size: 13px; align-self: center; }
189
+ .filter-row button.clear:disabled { color: var(--ink-muted); text-decoration: none; cursor: default; }
190
+
191
+ /* sources list */
192
+ .src-table { border-collapse: collapse; width: 100%; font-size: 14px; }
193
+ .src-table th { font-size: 12px; font-weight: 500; color: var(--ink-2); text-align: left; padding: 0; border-bottom: 1px solid var(--grid); }
194
+ .src-table th button { border: 0; background: none; padding: 8px 10px 8px 0; cursor: pointer; font-size: 12px; color: var(--ink-2); text-align: inherit; width: 100%; min-height: 32px; white-space: nowrap; }
195
+ .src-table th[aria-sort] button { color: var(--ink); font-weight: 700; }
196
+ .src-table th.r, .src-table td.r { text-align: right; font-variant-numeric: tabular-nums; }
197
+ .src-table td { padding: 8px 10px 8px 0; border-bottom: 1px solid var(--grid); vertical-align: top; }
198
+ .src-table td.title { max-width: 20rem; }
199
+ .src-table td.title button { border: 0; background: none; padding: 0; color: var(--link); cursor: pointer; text-align: left; font-size: 14px; text-decoration: underline; text-underline-offset: 2px; min-height: 24px; }
200
+ .src-table td.title button[aria-expanded="true"] { font-weight: 700; }
201
+ .flag-dot { margin-left: 6px; }
202
+ .form-cell .ico { margin-right: 5px; }
203
+ .rows-cell .mini { display: block; height: 3px; background: var(--grid); border-radius: 2px; margin-top: 4px; min-width: 60px; }
204
+ .rows-cell .mini > span { display: block; height: 100%; background: var(--ours); border-radius: 2px; margin-left: auto; }
205
+ .ok-text { display: inline-flex; align-items: center; gap: 4px; color: var(--ink-2); }
206
+ .src-table tr.open td { border-bottom-color: transparent; }
207
+ .src-table tr.detail-row > td { padding: 0 0 24px; }
208
+ .more { margin-top: 16px; }
209
+ button.link { border: 0; background: none; color: var(--link); text-decoration: underline; cursor: pointer; padding: 4px 0; font-size: 14px; min-height: 28px; }
210
+
211
+ /* source detail, opened in place */
212
+ .detail { padding: 12px 0 8px 16px; border-left: 3px solid var(--ours); display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); gap: 24px 40px; overflow-wrap: anywhere; }
213
+ .detail p { margin: 0 0 6px; font-size: 14px; }
214
+ .detail .full { grid-column: 1 / -1; }
215
+ .detail details summary { cursor: pointer; color: var(--link); font-size: 13px; }
216
+ .detail details.about summary { color: var(--ink-2); }
217
+ .detail details p { margin-top: 6px; max-width: 72ch; }
218
+ .fact-line { display: flex; align-items: center; flex-wrap: wrap; gap: 6px 8px; font-size: 14px; margin: 0 0 6px; }
219
+ .fact-line b { font-weight: 700; }
220
+ .mini-title { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--ink-2); margin: 8px 0 0; }
221
+ .strip { position: relative; height: 40px; margin: 8px 0 0; max-width: 420px; }
222
+ .strip-axis { position: absolute; left: 0; right: 0; top: 8px; height: 1px; background: var(--grid); }
223
+ .strip-band { position: absolute; top: 4px; height: 8px; background: #b0bdc6; border-radius: 4px; }
224
+ .strip-dot { position: absolute; top: 2px; width: 12px; height: 12px; margin-left: -6px; border-radius: 50%; background: var(--ours); box-shadow: 0 0 0 2px #fff; }
225
+ .strip-tick { position: absolute; top: 2px; width: 2px; height: 12px; background: var(--other); }
226
+ .strip-lab { position: absolute; top: 20px; font-size: 11px; color: var(--ink-2); white-space: nowrap; transform: translateX(-50%); }
227
+ .strip-lab.end { right: 0; transform: none; }
228
+ .sample td { font-family: var(--mono); font-weight: 500; font-size: 12px; word-break: break-all; }
229
+ .btn .ico, .btn.solid .ico { color: inherit; vertical-align: -3px; }
230
+
231
+ /* get the data: three pictured steps */
232
+ .steps { list-style: none; margin: 0; padding: 0; display: grid; grid-template-columns: minmax(0, 0.8fr) minmax(0, 2.2fr) minmax(0, 0.9fr); gap: 32px; align-items: start; }
233
+ .step-head { display: flex; align-items: center; gap: 10px; margin-bottom: 14px; }
234
+ .step-ico { display: inline-flex; width: 52px; height: 52px; border-radius: 50%; align-items: center; justify-content: center; box-shadow: inset 0 0 0 2px var(--ours); }
235
+ .step-n { font-size: 12px; color: var(--ink-muted); }
236
+ .step h3 { font-size: 18px; margin: 0; }
237
+ .chips { display: flex; flex-wrap: wrap; gap: 4px; margin: 8px 0; }
238
+ pre.code { margin: 10px 0 8px; padding: 12px 14px; background: var(--block); border-radius: 6px; font-size: 12.5px; line-height: 1.5; white-space: pre-wrap; overflow-wrap: anywhere; max-height: 18rem; overflow: auto; }
239
+ .copy-row { display: flex; align-items: center; gap: 10px; font-size: 13px; color: var(--ink-2); flex-wrap: wrap; }
240
+ .btn { display: inline-flex; align-items: center; gap: 4px; border: 1px solid var(--ours); background: var(--bg); color: var(--ours); border-radius: 6px; padding: 4px 14px; cursor: pointer; font-size: 14px; font-weight: 500; min-height: 32px; text-decoration: none; }
241
+ .btn.solid, .btn.solid:visited { background: var(--ours); color: #fff; }
242
+ .btn:disabled { border-color: var(--grid); color: var(--ink-muted); background: var(--block); cursor: default; }
243
+
244
+ /* not included, request */
245
+ details.long > summary { cursor: pointer; color: var(--link); margin-top: 16px; font-size: 14px; }
246
+ details.long[open] > summary { margin-bottom: 12px; }
247
+ footer.request { border-top: 1px solid var(--grid); padding: 48px 0 64px; }
248
+ .request-row { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; }
249
+ .request-row h2 { font-size: 22px; margin: 0; }
250
+ .request-ico { display: inline-flex; }
251
+
252
+ /* tooltip: value first, label second */
253
+ .tip { position: fixed; z-index: 50; pointer-events: none; background: var(--bg); color: var(--ink); border: 1px solid var(--grid); border-radius: 6px; padding: 8px 10px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.08); font-size: 12px; max-width: 280px; }
254
+ .tip .v { display: block; font-size: 16px; font-weight: 700; }
255
+ .tip .k { display: block; color: var(--ink-2); }
256
+ .tip .k::before { content: ""; display: inline-block; width: 10px; height: 2px; background: var(--ours); vertical-align: 3px; margin-right: 6px; }
257
+
258
+ /* ---- narrow screens ---- */
259
+ @media (max-width: 960px) {
260
+ .map-grid, .overview-grid { grid-template-columns: minmax(0, 1fr); }
261
+ .detail { grid-template-columns: minmax(0, 1fr); }
262
+ .steps { grid-template-columns: minmax(0, 1fr); gap: 28px; }
263
+ }
264
+ @media (min-width: 721px) {
265
+ .filter-row .count { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); }
266
+ }
267
+ @media (max-width: 720px) {
268
+ :root { --pad: 16px; }
269
+ header.top { padding-top: 24px; }
270
+ header.top h1 { font-size: 26px; }
271
+ section.unit { padding-top: 44px; padding-bottom: 44px; min-height: 0; }
272
+ .hero .big { font-size: 52px; }
273
+ .progress a { font-size: 11px; column-gap: 3px; }
274
+ .filter-toggle { display: flex; width: 100%; justify-content: space-between; align-items: center; border: 1px solid #b8b8b8; background: var(--bg); border-radius: 6px; padding: 6px 12px; min-height: 40px; cursor: pointer; font-size: 14px; }
275
+ .filter-toggle .state { color: var(--ink-2); font-size: 13px; }
276
+ .filters .filter-row { display: none; margin-top: 10px; }
277
+ .filters.open .filter-row { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); gap: 10px; max-height: 70vh; overflow-y: auto; }
278
+ .filters.open .filter-row .wide, .filters.open .filter-row .count, .filters.open .filter-row .clear { grid-column: 1 / -1; }
279
+ .filter-row select, .filter-row input, .filter-row input[type="search"] { max-width: none; width: 100%; }
280
+ .filter-row .count { margin-left: 0; }
281
+ .bar-row { grid-template-columns: minmax(0, 9rem) minmax(0, 1fr); }
282
+ .heat .row-head { font-size: 12px; }
283
+ .scale .steps, .scale .ticks { grid-template-columns: repeat(7, 30px); }
284
+ .cols { gap: 16px; }
285
+ .src-table, .src-table > thead, .src-table > tbody, .src-table > tbody > tr, .src-table > tbody > tr > td { display: block; width: 100%; }
286
+ .src-table > thead { border-bottom: 1px solid var(--grid); }
287
+ .src-table > thead > tr { display: flex; flex-wrap: wrap; gap: 0 14px; }
288
+ .src-table > thead > tr > th { border: 0; width: auto; }
289
+ .src-table > thead > tr > th button { padding: 4px 0; }
290
+ .src-table > tbody > tr.row { padding: 10px 0; border-bottom: 1px solid var(--grid); display: flex; flex-wrap: wrap; gap: 2px 12px; }
291
+ .src-table > tbody > tr.row.open { border-bottom-color: transparent; }
292
+ .src-table > tbody > tr.row > td { border: 0; padding: 0; width: auto; font-size: 13px; color: var(--ink-2); }
293
+ .src-table > tbody > tr.row > td.title { width: 100%; max-width: none; font-size: 15px; }
294
+ .src-table > tbody > tr.row > td.r { text-align: left; }
295
+ .src-table > tbody > tr.row > td.rows-cell .mini { display: none; }
296
+ .src-table > tbody > tr.row > td[data-label]::before { content: attr(data-label) ": "; color: var(--ink-muted); }
297
+ .detail { padding-left: 12px; }
298
  }