Buckets:

ofou
/

Repro-Neural-Thickets-artifacts

Files

xet

ofou/Repro-Neural-Thickets-artifacts / tinygrad-bundle /verdicts.json

ofou

4 months ago

download

raw

1.94 kB

{
  "paper": {
    "arxiv": "2603.12228",
    "openreview": "92oF5bU4cU",
    "icml_pid": "3715",
    "title": "Neural Thickets"
  },
  "framework": "tinygrad",
  "compute": {
    "local": "Apple Silicon, tinygrad 0.13 CPU/Metal via tinygrad_repro/.venv",
    "hf_jobs": "blocked 402 Payment Required — insufficient prepaid credits (re-checked 2026-07-15)"
  },
  "claims": {
    "1": {
      "text": "Dense task-expert neighborhoods at scale (needle vs thicket)",
      "verdict": "reproduced",
      "evidence": "tinygrad 1D AR-MSE: thicket discordance 0.676 vs needle 0.247; useful top1 gain 0.098 vs 0.007 (base→top1 MSE 0.104→0.006). Plateau discordance 0.070. Raw density alone is misleading (plateau 0.51) — discordance/useful gain are the right metrics."
    },
    "2": {
      "text": "Qwen2.5 solution density/diversity increase with scale",
      "verdict": "partially",
      "evidence": "LLM multi-scale blocked (HF Jobs 402). tinygrad width proxy: useful_density 0.245→0.355 (w16→w128); claim2_proxy flag true. Not a full Qwen2.5–32B reproduction."
    },
    "3": {
      "text": "Random perturbations are task specialists",
      "verdict": "reproduced",
      "evidence": "thicket specialist_frac_among_improving=1.0; discordance 0.676 vs needle 0.247. Improving seeds specialize across line/sin/square tasks."
    },
    "4": {
      "text": "RandOpt matches/exceeds PPO/GRPO/ES",
      "verdict": "partially",
      "evidence": "LLM PPO/GRPO/ES not run (Jobs 402). Toy matched-budget AR-MSE @N=100: RandOpt best 0.024 / ens 0.019 << ES 0.178 / SGD 0.135. Mechanism reproduced; LLM scope not attempted."
    },
    "5": {
      "text": "RandOpt accuracy scales with population size and model scale",
      "verdict": "reproduced",
      "evidence": "tinygrad: best MSE N=20→200: 0.039→0.006. Width scale @N=100: rel_gain_best w32=0.51 vs w128=0.94. LLM half of claim still Jobs-blocked."
    }
  }
}

Xet Storage Details

Size:: 1.94 kB
Xet hash:: 42c61dc99219d836ba36ff9baae74a9f3df3ecd1cd06b34f468fc52c6744ef1e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	{
	"paper": {
	"arxiv": "2603.12228",
	"openreview": "92oF5bU4cU",
	"icml_pid": "3715",
	"title": "Neural Thickets"
	},
	"framework": "tinygrad",
	"compute": {
	"local": "Apple Silicon, tinygrad 0.13 CPU/Metal via tinygrad_repro/.venv",
	"hf_jobs": "blocked 402 Payment Required — insufficient prepaid credits (re-checked 2026-07-15)"
	},
	"claims": {
	"1": {
	"text": "Dense task-expert neighborhoods at scale (needle vs thicket)",
	"verdict": "reproduced",
	"evidence": "tinygrad 1D AR-MSE: thicket discordance 0.676 vs needle 0.247; useful top1 gain 0.098 vs 0.007 (base→top1 MSE 0.104→0.006). Plateau discordance 0.070. Raw density alone is misleading (plateau 0.51) — discordance/useful gain are the right metrics."
	},
	"2": {
	"text": "Qwen2.5 solution density/diversity increase with scale",
	"verdict": "partially",
	"evidence": "LLM multi-scale blocked (HF Jobs 402). tinygrad width proxy: useful_density 0.245→0.355 (w16→w128); claim2_proxy flag true. Not a full Qwen2.5–32B reproduction."
	},
	"3": {
	"text": "Random perturbations are task specialists",
	"verdict": "reproduced",
	"evidence": "thicket specialist_frac_among_improving=1.0; discordance 0.676 vs needle 0.247. Improving seeds specialize across line/sin/square tasks."
	},
	"4": {
	"text": "RandOpt matches/exceeds PPO/GRPO/ES",
	"verdict": "partially",
	"evidence": "LLM PPO/GRPO/ES not run (Jobs 402). Toy matched-budget AR-MSE @N=100: RandOpt best 0.024 / ens 0.019 << ES 0.178 / SGD 0.135. Mechanism reproduced; LLM scope not attempted."
	},
	"5": {
	"text": "RandOpt accuracy scales with population size and model scale",
	"verdict": "reproduced",
	"evidence": "tinygrad: best MSE N=20→200: 0.039→0.006. Width scale @N=100: rel_gain_best w32=0.51 vs w128=0.94. LLM half of claim still Jobs-blocked."
	}
	}
	}