How to use from the
Use from the
MLX library
# Download the model from the Hub
pip install huggingface_hub[hf_xet]

hf download EARAI/EARAI0.1 --local-dir EARAI0.1

EARAI logo

EARAI 0.1

RESEARCH PREVIEW · 166.9M PARAMETERS · CHECKPOINT 87,776

EARAI 0.1 is an early research model for code generation. It explores a structured representation of programming context alongside a generative decoder. This release shares the weights and tokenizer; the research implementation is not included.

Status: the model is experimental and is not ready for autonomous programming or production use. Its current execution-based benchmark results are reported below, including failures.


Evaluation

HumanEval execution results

The release was frozen before evaluation. One deterministic candidate was generated per task and tested in a network-isolated container. No external model or tool was used to repair the candidate.

Evaluation Tasks Passed Other observations
HumanEval, full set 164 0 / 164 17 prompts exceeded the total output budget; 123 generated programs parsed as Python
Sanitized MBPP, sampled 30 0 / 30 0 raw outputs parsed as Python in this sample

For HumanEval, the public function prefix was supplied to the decoder and counted toward the 768-token output budget. Tasks that did not fit are counted as not solved. Both models' HumanEval continuations were cut at the same top-level stop patterns before execution. The syntax count includes the supplied prefix and does not imply the function is correct.

The MBPP figure is a deterministic random sample from the sanitized dataset, not the standard complete MBPP test split. The prompt explicitly requested Python and included one visible test. The raw output was executed without code extraction. Full task-level outcomes and the dataset checksums are provided in benchmark_outcomes.jsonl and metrics.json.

Matched small-model comparison

On the same 20 HumanEval tasks, this frozen checkpoint solved 0 / 20. For reference, under the same protocol (forced prefix, 768-token budget, isolated container): Qwen2.5-0.5B (494M) solved 7 / 20, Qwen2.5-Coder-0.5B (494M) solved 3 / 20, and TinyStarCoderPy (164M) solved 1 / 20 under the local test. On the same 10 sampled MBPP tasks, all four solved 0 / 10 with the raw-output procedure. This is a small comparison, not a leaderboard or a claim that the models are generally equivalent. Their tokenizers and model interfaces differ, and the same numeric token budget does not imply identical amounts of text. The market-model runs are reproducible with research/bench_humaneval_market.py; per-task outcomes are in market_eval/.

As a check on the executor, 10 sampled reference implementations from each benchmark passed their original tests. This check does not rule out every evaluation issue or training-data contamination.

Data and training

The model was trained from scratch on code and task data derived from open-source sources, with examples in Python, C, C++, C#, Go, Swift and JavaScript. The training pipeline also included a small set of execution-checked repair examples and teacher-generated hints. Language coverage in training is not a verified claim of programming ability in each language.

This checkpoint was produced on an Apple M5 Mac with 16 GB unified memory. The training context ceiling was 8,192 graph nodes; graph nodes are not tokens. A 256K-token context window or reliable use of distant project context has not been established.

Files and use

File Contents
model.safetensors Frozen model weights at step 87,776
earai_tokenizer.json EARAI tokenizer data
release_info.json Parameter count, checkpoint and weight checksum
metrics.json Evaluation configuration and aggregate results
benchmark_outcomes.jsonl Task IDs and outcomes; no benchmark reference solutions

Loading limitation: this is a custom MLX research checkpoint, not a standard Hugging Face Transformers model. The tokenizer is also in an EARAI-specific format. The inference implementation is not included in this release, so AutoModel / AutoTokenizer cannot load these files as a ready-to-use application.

The weights are public and can be downloaded or copied. No model-weight license is specified in this card.

Limitations

  • The model frequently produces incorrect, repetitive or incomplete code.
  • The current results do not establish independent project-level programming ability.
  • Long output and long-context use remain constrained and require further testing.
  • Public-code training data can include errors, duplication or licensing constraints; the data filters are not a proof of semantic quality or absence of sensitive material.
  • Benchmark results depend on prompt format, tokenization, decoding and execution rules. The matched 20-task comparison is too small to establish a general ranking.

Кратко по-русски

EARAI 0.1 — исследовательская модель для генерации кода. В полном HumanEval эта зафиксированная версия решила 0 из 164 задач; 17 заданий не поместились в лимит. Модель пока не готова для самостоятельной разработки. В репозитории опубликованы веса и токенизатор, но исходная реализация и готовый загрузчик не включены.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
0.2B params
Tensor type
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support