Spaces:
Running
Running
|
Download data/README.md from sparklabuofu/MiniWeb-demo: direct link, hf CLI and curl.
- Browser
- Download file 3.67 kB
-
https://huggingface.co/spaces/sparklabuofu/MiniWeb-demo/resolve/main/data/README.md
- Command line
-
hf download hf://spaces/sparklabuofu/MiniWeb-demo/data/README.md
-
curl -L -o README.md https://huggingface.co/spaces/sparklabuofu/MiniWeb-demo/resolve/main/data/README.md
3.67 kB
data/
The MiniWeb database, the recorded human tasks, the macro registry, and the workspaces of the data-generation and training pipelines.
.gitignore excludes everything here (data/*) except a short allow-list, the files in the first table.
Tracked in git
| Path | Contents |
|---|---|
macros.yaml |
The canonical macro registry: base macros, reasoning operations, aliases. The single source of truth, loaded by src/miniweb/annotation/macros.py |
macro_locations.yaml |
Per-site macro → UI location map (coverage, sampling, the generator's split) |
final_set.yaml |
The paper's final set: 37 primitives and 376 tasks (by id). It locks the annotation tool to them |
sites.yaml |
The 65 sites: domain, split (52 train, 13 held out), hub, data source and license |
datagen/site_split.json, datagen/guides/ |
MiniWebGen's inputs: the train/test site split, fixed before any generation, and one guide per primitive |
Local data (not in git)
| Path | Contents |
|---|---|
miniweb.db |
The per-site SQLite database every site reads and writes through miniweb.app.db, rebuilt from the Hub by scripts/restore_site_db.py (MINIWEB_DB moves it). It was modified after the build: never re-run build_db.py |
annotations/ |
The human tasks, <author>/<task_id>/: task.json, verifier.json, trajectory.json, screenshots/, the verification walk; rebuilt from the Hub by scripts/fetch_tasks.py (MINIWEB_ANNOTATIONS_DIR moves it) |
static/ |
The sites' own images (generated/, avatars/, thumbnails/), served at /static/<sub>/…; restored by scripts/restore_site_db.py (MINIWEB_DATA_STATIC moves it) |
media/ |
A copy of the external media the sites' pages show (images that site data links to, map tiles) with manifest.jsonl, served at /media/…; restored by scripts/restore_site_db.py, rebuilt from the original hosts by scripts/fetch_site_media.py (MINIWEB_MEDIA_DIR moves it) |
datagen/ |
MiniWebGen's workspace besides the tracked inputs: site maps, caches, runs/<run>/, retired.json, the viewer. See src/miniwebgen/README.md (MINIWEB_DATAGEN_DIR moves it) |
webmix/ |
Replayed training rows, adapters, evaluation runs, benchmark task sets and judges, the task-centric data (taskgen/). See src/miniwebagent/README.md (WEBMIX_REPLAY_DIR points training at other rows) |
grader_audit/ |
The grader audit: sampled agent episodes, the grader key and the labels (src/miniweb/scripts/build_grader_audit.py, grader_audit_report.py) |
release/ |
The consolidated data release that scripts/build_release.py writes (miniweb-v1/) and its Hugging Face version (miniweb-v1-hf/, scripts/build_hf_dataset.py and scripts/export_site_db.py) |
Tools create a few more folders when they run: backups/ and railway_pulls/ (src/miniweb/scripts/pull_from_railway.py),
and review packets (task_review_*/, task_repairs_*/, verifier_audit_*/) that the annotation tool serves read-only.
None of them is needed to run the sites, the evaluations or the pipelines.
Notes
- Macro YAMLs in production: annotators register new macros by writing to
macros.yaml. On a deployment, point both YAMLs at a persistent volume withMINIWEB_MACRO_DIR, or per file withMINIWEB_MACROSandMINIWEB_MACRO_LOCATIONS. A fresh volume is seeded from the copies in this directory. datagen/retired.jsonis hand-written but not tracked. Back it up with the runs it refers to.- After a database rebuild, re-run the runtime seed scripts in
src/miniweb/scripts/(seed_*.py).