--- language: - en tags: - code - code-generation - research - mlx - experimental --- ![EARAI logo](assets/logo-lockup.svg) # EARAI 0.1 **RESEARCH PREVIEW · 166.9M PARAMETERS · CHECKPOINT 87,776** EARAI 0.1 is an early research model for code generation. It explores a structured representation of programming context alongside a generative decoder. This release shares the weights and tokenizer; the research implementation is not included. > **Status:** the model is experimental and is not ready for autonomous programming or production use. Its current execution-based benchmark results are reported below, including failures. --- ## Evaluation ![HumanEval execution results](assets/evaluation.svg) The release was frozen before evaluation. One deterministic candidate was generated per task and tested in a network-isolated container. No external model or tool was used to repair the candidate. | Evaluation | Tasks | Passed | Other observations | |---|---:|---:|---| | [HumanEval](https://github.com/openai/human-eval), full set | 164 | **0 / 164** | 17 prompts exceeded the total output budget; 123 generated programs parsed as Python | | [Sanitized MBPP](https://github.com/google-research/google-research/tree/master/mbpp), sampled | 30 | **0 / 30** | 0 raw outputs parsed as Python in this sample | For HumanEval, the public function prefix was supplied to the decoder and counted toward the 768-token output budget. Tasks that did not fit are counted as not solved. Both models' HumanEval continuations were cut at the same top-level stop patterns before execution. The syntax count includes the supplied prefix and does **not** imply the function is correct. The MBPP figure is a deterministic random sample from the sanitized dataset, **not** the standard complete MBPP test split. The prompt explicitly requested Python and included one visible test. The raw output was executed without code extraction. Full task-level outcomes and the dataset checksums are provided in [`benchmark_outcomes.jsonl`](benchmark_outcomes.jsonl) and [`metrics.json`](metrics.json). ### Matched small-model comparison On the **same 20 HumanEval tasks**, this frozen checkpoint solved **0 / 20**. For reference, under the same protocol (forced prefix, 768-token budget, isolated container): [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) (494M) solved **7 / 20**, [Qwen2.5-Coder-0.5B](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B) (494M) solved **3 / 20**, and [TinyStarCoderPy](https://huggingface.co/bigcode/tiny_starcoder_py) (164M) solved **1 / 20** under the local test. On the same 10 sampled MBPP tasks, all four solved 0 / 10 with the raw-output procedure. This is a small comparison, not a leaderboard or a claim that the models are generally equivalent. Their tokenizers and model interfaces differ, and the same numeric token budget does not imply identical amounts of text. The market-model runs are reproducible with `research/bench_humaneval_market.py`; per-task outcomes are in [`market_eval/`](market_eval/). As a check on the executor, 10 sampled reference implementations from each benchmark passed their original tests. This check does not rule out every evaluation issue or training-data contamination. ## Data and training The model was trained from scratch on code and task data derived from open-source sources, with examples in Python, C, C++, C#, Go, Swift and JavaScript. The training pipeline also included a small set of execution-checked repair examples and teacher-generated hints. Language coverage in training is **not** a verified claim of programming ability in each language. This checkpoint was produced on an Apple M5 Mac with 16 GB unified memory. The training context ceiling was 8,192 graph nodes; graph nodes are **not** tokens. A 256K-token context window or reliable use of distant project context has not been established. ## Files and use | File | Contents | |---|---| | [`model.safetensors`](model.safetensors) | Frozen model weights at step 87,776 | | [`earai_tokenizer.json`](earai_tokenizer.json) | EARAI tokenizer data | | [`release_info.json`](release_info.json) | Parameter count, checkpoint and weight checksum | | [`metrics.json`](metrics.json) | Evaluation configuration and aggregate results | | [`benchmark_outcomes.jsonl`](benchmark_outcomes.jsonl) | Task IDs and outcomes; no benchmark reference solutions | **Loading limitation:** this is a custom MLX research checkpoint, not a standard Hugging Face Transformers model. The tokenizer is also in an EARAI-specific format. The inference implementation is not included in this release, so `AutoModel` / `AutoTokenizer` cannot load these files as a ready-to-use application. The weights are public and can be downloaded or copied. No model-weight license is specified in this card. ## Limitations - The model frequently produces incorrect, repetitive or incomplete code. - The current results do not establish independent project-level programming ability. - Long output and long-context use remain constrained and require further testing. - Public-code training data can include errors, duplication or licensing constraints; the data filters are not a proof of semantic quality or absence of sensitive material. - Benchmark results depend on prompt format, tokenization, decoding and execution rules. The matched 20-task comparison is too small to establish a general ranking. --- ### Кратко по-русски EARAI 0.1 — исследовательская модель для генерации кода. В полном HumanEval эта зафиксированная версия решила **0 из 164** задач; 17 заданий не поместились в лимит. Модель пока не готова для самостоятельной разработки. В репозитории опубликованы веса и токенизатор, но исходная реализация и готовый загрузчик не включены.