EARAI commited on
Commit
66781ee
·
verified ·
1 Parent(s): 3ca0b32

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -11,7 +11,7 @@ tags:
11
 
12
  ![EARAI logo](assets/logo-lockup.svg)
13
 
14
-
15
 
16
  # EARAI 0.1
17
 
@@ -40,7 +40,7 @@ The MBPP figure is a deterministic random sample from the sanitized dataset, **n
40
 
41
  ### Matched small-model comparison
42
 
43
- On the **same 20 HumanEval tasks**, this frozen checkpoint solved **0 / 20**; [TinyStarCoderPy](https://huggingface.co/bigcode/tiny_starcoder_py), a 164M-parameter model, solved **1 / 20** under the local test. On the same 10 sampled MBPP tasks, both solved 0 / 10 with the raw-output procedure. This is a small comparison, not a leaderboard or a claim that the models are generally equivalent. Their tokenizers and model interfaces differ, and the same numeric token budget does not imply identical amounts of text.
44
 
45
  As a check on the executor, 10 sampled reference implementations from each benchmark passed their original tests. This check does not rule out every evaluation issue or training-data contamination.
46
 
 
11
 
12
  ![EARAI logo](assets/logo-lockup.svg)
13
 
14
+ ![EARAI research banner](assets/banner.png)
15
 
16
  # EARAI 0.1
17
 
 
40
 
41
  ### Matched small-model comparison
42
 
43
+ On the **same 20 HumanEval tasks**, this frozen checkpoint solved **0 / 20**. For reference, under the same protocol (forced prefix, 768-token budget, isolated container): [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) (494M) solved **7 / 20**, [Qwen2.5-Coder-0.5B](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B) (494M) solved **3 / 20**, and [TinyStarCoderPy](https://huggingface.co/bigcode/tiny_starcoder_py) (164M) solved **1 / 20** under the local test. On the same 10 sampled MBPP tasks, all four solved 0 / 10 with the raw-output procedure. This is a small comparison, not a leaderboard or a claim that the models are generally equivalent. Their tokenizers and model interfaces differ, and the same numeric token budget does not imply identical amounts of text. The market-model runs are reproducible with `research/bench_humaneval_market.py`; per-task outcomes are in [`market_eval/`](market_eval/).
44
 
45
  As a check on the executor, 10 sampled reference implementations from each benchmark passed their original tests. This check does not rule out every evaluation issue or training-data contamination.
46