modelguard-observatory / REPRODUCE.md
Hugdrigo's picture
Upload 28 files
85920d3 verified
|
Raw History Blame Contribute Delete
1.38 kB

Reproduction instructions

Requirements: Node.js 18 or later for command-line research scripts; Python plus Playwright and local Chromium only for optional browser integration tests.

node build_artifacts.js
node test_engine.js
python test_browser.py

build_artifacts.js reads source inputs from the same folder. It retrains the deterministic original baseline models on train.json, evaluates on iid_test.json and challenge.json, writes evaluation_snapshots.json, and produces benchmark_manifest.json with SHA-256 digests of the raw inputs and prediction artifact. No Internet connection or libraries are needed for the Node tasks.

After rebuilding, run test_engine.js. Browser testing uses Playwright set_content, and injects a fetch mock with the actual local JSON files because some testing sandboxes block loopback access. It uses bundled source data, not simulated model scores. Adapt the executable_path to your local Chromium installation if needed.

Expected metrics:

Method Controlled correct / n Challenge correct / n
NB 130/130 40/54
Word TF-IDF 128/130 39/54
Character TF-IDF 124/130 38/54
Hybrid 127/130 42/54

Source data: Academic Rhetorical Intent Mini v1.1, CC BY 4.0. No model weights are served or trained by the Hugging Face Static application on initial load.