File size: 6,035 Bytes
f255843
2100fa6
 
 
 
 
 
 
 
f255843
2100fa6
21de506
 
2100fa6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66781ee
2100fa6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
---
language:
  - en
tags:
  - code
  - code-generation
  - research
  - mlx
  - experimental
---

![EARAI logo](assets/logo-lockup.svg)

# EARAI 0.1

**RESEARCH PREVIEW · 166.9M PARAMETERS · CHECKPOINT 87,776**

EARAI 0.1 is an early research model for code generation. It explores a structured representation of programming context alongside a generative decoder. This release shares the weights and tokenizer; the research implementation is not included.

> **Status:** the model is experimental and is not ready for autonomous programming or production use. Its current execution-based benchmark results are reported below, including failures.

---

## Evaluation

![HumanEval execution results](assets/evaluation.svg)

The release was frozen before evaluation. One deterministic candidate was generated per task and tested in a network-isolated container. No external model or tool was used to repair the candidate.

| Evaluation | Tasks | Passed | Other observations |
|---|---:|---:|---|
| [HumanEval](https://github.com/openai/human-eval), full set | 164 | **0 / 164** | 17 prompts exceeded the total output budget; 123 generated programs parsed as Python |
| [Sanitized MBPP](https://github.com/google-research/google-research/tree/master/mbpp), sampled | 30 | **0 / 30** | 0 raw outputs parsed as Python in this sample |

For HumanEval, the public function prefix was supplied to the decoder and counted toward the 768-token output budget. Tasks that did not fit are counted as not solved. Both models' HumanEval continuations were cut at the same top-level stop patterns before execution. The syntax count includes the supplied prefix and does **not** imply the function is correct.

The MBPP figure is a deterministic random sample from the sanitized dataset, **not** the standard complete MBPP test split. The prompt explicitly requested Python and included one visible test. The raw output was executed without code extraction. Full task-level outcomes and the dataset checksums are provided in [`benchmark_outcomes.jsonl`](benchmark_outcomes.jsonl) and [`metrics.json`](metrics.json).

### Matched small-model comparison

On the **same 20 HumanEval tasks**, this frozen checkpoint solved **0 / 20**. For reference, under the same protocol (forced prefix, 768-token budget, isolated container): [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) (494M) solved **7 / 20**, [Qwen2.5-Coder-0.5B](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B) (494M) solved **3 / 20**, and [TinyStarCoderPy](https://huggingface.co/bigcode/tiny_starcoder_py) (164M) solved **1 / 20** under the local test. On the same 10 sampled MBPP tasks, all four solved 0 / 10 with the raw-output procedure. This is a small comparison, not a leaderboard or a claim that the models are generally equivalent. Their tokenizers and model interfaces differ, and the same numeric token budget does not imply identical amounts of text. The market-model runs are reproducible with `research/bench_humaneval_market.py`; per-task outcomes are in [`market_eval/`](market_eval/).

As a check on the executor, 10 sampled reference implementations from each benchmark passed their original tests. This check does not rule out every evaluation issue or training-data contamination.

## Data and training

The model was trained from scratch on code and task data derived from open-source sources, with examples in Python, C, C++, C#, Go, Swift and JavaScript. The training pipeline also included a small set of execution-checked repair examples and teacher-generated hints. Language coverage in training is **not** a verified claim of programming ability in each language.

This checkpoint was produced on an Apple M5 Mac with 16 GB unified memory. The training context ceiling was 8,192 graph nodes; graph nodes are **not** tokens. A 256K-token context window or reliable use of distant project context has not been established.

## Files and use

| File | Contents |
|---|---|
| [`model.safetensors`](model.safetensors) | Frozen model weights at step 87,776 |
| [`earai_tokenizer.json`](earai_tokenizer.json) | EARAI tokenizer data |
| [`release_info.json`](release_info.json) | Parameter count, checkpoint and weight checksum |
| [`metrics.json`](metrics.json) | Evaluation configuration and aggregate results |
| [`benchmark_outcomes.jsonl`](benchmark_outcomes.jsonl) | Task IDs and outcomes; no benchmark reference solutions |

**Loading limitation:** this is a custom MLX research checkpoint, not a standard Hugging Face Transformers model. The tokenizer is also in an EARAI-specific format. The inference implementation is not included in this release, so `AutoModel` / `AutoTokenizer` cannot load these files as a ready-to-use application.

The weights are public and can be downloaded or copied. No model-weight license is specified in this card.

## Limitations

- The model frequently produces incorrect, repetitive or incomplete code.
- The current results do not establish independent project-level programming ability.
- Long output and long-context use remain constrained and require further testing.
- Public-code training data can include errors, duplication or licensing constraints; the data filters are not a proof of semantic quality or absence of sensitive material.
- Benchmark results depend on prompt format, tokenization, decoding and execution rules. The matched 20-task comparison is too small to establish a general ranking.

---

### Кратко по-русски

EARAI 0.1 — исследовательская модель для генерации кода. В полном HumanEval эта зафиксированная версия решила **0 из 164** задач; 17 заданий не поместились в лимит. Модель пока не готова для самостоятельной разработки. В репозитории опубликованы веса и токенизатор, но исходная реализация и готовый загрузчик не включены.