AweDist: Attention-aware Embedding Distillation for New Input Token Embeddings
Paper • 2505.20133 • Published • 1
Nebula-1-ablation-a1
Nebula-1-ablation-a1 is a preview of Nebula-1, a Japanese and English base language model of about 122M parameters (Llama-style decoder, 24 layers, d_model 576, vocabulary 64,000), trained from scratch on about 2B tokens by Nebou Rocket. This checkpoint is an ablation checkpoint.
This is arm A1 of our pre-registered ablation: it was trained with D1 (self-definition anchoring) on top of A0. During training the model reads Wiktionary definitions of rare tokens, and an auxiliary loss pulls those tokens' embeddings toward the model's own pooled hidden state over the definition.
The pre-registered decision rule did not adopt D1. We publish this arm as it is, so that the negative result can be inspected and reproduced.
D1 verdict (pre-registered rule): NOT ADOPTED. On rare-token validation loss, A0 = 5.2458, A1 = 6.1644, A2 = 5.2015 (seed noise 0.0613). The full rule and numbers are in eval/ablation-result.json.
- [FAIL] A1 beats A0 on rare loss by more than the noise: A0.rare - A1.rare = -0.9186 > noise 0.0613
- [FAIL] A1 beats A2 on rare loss by more than the noise: A2.rare - A1.rare = -0.9629 > noise 0.0613
- [PASS] English benchmark average is not lower than A0 by more than 0.5 points: A1.avg_en = 30.31 >= A0.avg_en - 0.5 = 29.69
- [PASS] Japanese benchmark average is not lower than A0 by more than 0.5 points: A1.avg_ja = 32.04 >= A0.avg_ja - 0.5 = 31.54
Related repositories
All four arms of the pre-registered ablation are published, so the comparison can be reproduced and inspected.
| Repository | Arm | What it is |
|---|---|---|
| LoNebula/Nebula-1-preview | A0 | This is arm A0 of our pre-registered ablation: next-token prediction only, without D1 (self-definition anchoring). |
| LoNebula/Nebula-1-ablation-a1 | A1 | This is arm A1 of our pre-registered ablation: it was trained with D1 (self-definition anchoring) on top of A0. During training the model reads Wiktionary definitions of rare tokens, and an auxiliary loss pulls those tokens' embeddings toward the model's own pooled hidden state over the definition. |
| LoNebula/Nebula-1-ablation-a2 | A2 | This is arm A2 of our pre-registered ablation: A0 plus the same Wiktionary definitions, inserted as plain text documents (<word>: <definition>) at the same total token cost as A1's definition reads. It is the control that separates the D1 mechanism from simply reading dictionary text. |
| LoNebula/Nebula-1-ablation-a0-seed1235 | A0' | This is arm A0' of our pre-registered ablation: the A0 baseline trained again with seed 1235 (initialization and data order), so that the difference between A0 and A0' could be used as the seed-noise level of the decision rule. |
Benchmarks
Every number below was measured by us with lm-evaluation-harness 0.4.13, under one fixed setting (same seed, dtype and batch size for every model). We did not copy any official number. Comparison models are public base models of 1B parameters or fewer. Values are percentages; the best value per column is bold.
| Model | Params | MMLU | ARC-e | ARC-c | HellaSwag | WinoGrande | PIQA | LAMBADA | GSM8K | MMLU-ja | JComQA | JNLI | XWinograd-ja | Belebele-ja | MGSM-ja | Avg (en) | Avg (ja) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Nebula A0 | 122M | 25.0 | 38.4 | 23.6 | 27.5 | 51.0 | 58.2 | 17.7 | 0.2 | 24.8 | 42.9 | 45.5 | 54.7 | 22.7 | 1.6 | 30.2 | 32.0 |
| Nebula-1-ablation-a1 | 122M | 25.9 | 39.3 | 23.4 | 27.9 | 50.3 | 57.1 | 17.8 | 0.7 | 26.7 | 39.6 | 46.9 | 54.5 | 23.3 | 1.2 | 30.3 | 32.0 |
| Nebula A2 | 122M | 26.4 | 39.4 | 23.2 | 28.1 | 51.6 | 58.2 | 18.7 | 0.6 | 25.8 | 47.7 | 41.0 | 52.9 | 22.8 | 1.6 | 30.8 | 32.0 |
| Nebula A0' | 122M | 26.3 | 40.3 | 22.9 | 27.9 | 50.9 | 57.7 | 19.1 | 0.5 | 26.4 | 43.5 | 41.7 | 52.6 | 23.1 | 1.6 | 30.7 | 31.5 |
| HuggingFaceTB/SmolLM2-135M | 135M | 25.2 | 58.4 | 30.0 | 43.0 | 53.0 | 68.3 | 42.9 | 1.4 | 26.4 | 23.1 | 44.7 | 51.6 | 22.9 | 1.2 | 40.3 | 28.3 |
| facebook/MobileLLM-R1-140M-base | 140M | 23.8 | 50.0 | 24.7 | 34.1 | 50.7 | 62.9 | 31.2 | 19.0 | 25.3 | 22.0 | 49.9 | 52.2 | 21.4 | 1.6 | 37.1 | 28.7 |
| llm-jp/llm-jp-3-150m | 152M | 25.8 | 38.3 | 24.7 | 31.6 | 50.0 | 60.9 | 28.2 | 0.2 | 25.9 | 56.7 | 46.7 | 55.0 | 24.2 | 1.2 | 32.5 | 35.0 |
| google/gemma-3-270m | 268M | 26.5 | 57.2 | 28.1 | 41.4 | 53.6 | 67.8 | 42.9 | 1.3 | 27.3 | 46.1 | 38.4 | 55.2 | 23.1 | 1.2 | 39.8 | 31.9 |
| LiquidAI/LFM2-350M | 354M | 43.1 | 66.7 | 39.6 | 49.0 | 56.7 | 69.0 | 39.8 | 34.0 | 36.8 | 54.6 | 36.8 | 54.8 | 39.8 | 0.0 | 49.8 | 37.1 |
| facebook/MobileLLM-R1-360M-base | 359M | 26.5 | 61.2 | 34.4 | 44.1 | 55.2 | 69.0 | 39.4 | 41.6 | 24.0 | 29.2 | 55.0 | 53.4 | 27.6 | 0.0 | 46.4 | 31.5 |
| HuggingFaceTB/SmolLM2-360M | 362M | 25.1 | 68.1 | 38.1 | 56.2 | 59.7 | 72.0 | 53.2 | 4.2 | 26.0 | 26.6 | 49.3 | 51.5 | 28.2 | 2.0 | 47.1 | 30.6 |
| EleutherAI/pythia-410m | 405M | 26.6 | 45.2 | 24.3 | 40.4 | 53.0 | 67.0 | 47.5 | 1.2 | 26.3 | 26.1 | 37.4 | 52.8 | 23.1 | 0.8 | 38.2 | 27.8 |
| llm-jp/llm-jp-3-440m | 447M | 24.8 | 45.1 | 24.3 | 42.5 | 52.0 | 66.7 | 39.5 | 1.3 | 24.9 | 59.8 | 40.3 | 58.3 | 22.1 | 0.8 | 37.0 | 34.4 |
| Qwen/Qwen2.5-0.5B | 494M | 47.6 | 58.8 | 31.9 | 52.1 | 56.7 | 70.0 | 52.0 | 34.1 | 37.5 | 35.8 | 39.8 | 58.3 | 33.4 | 0.0 | 50.4 | 34.1 |
| LiquidAI/LFM2-700M | 742M | 49.4 | 71.0 | 46.9 | 57.5 | 59.1 | 71.0 | 45.1 | 46.4 | 42.6 | 65.1 | 29.8 | 58.9 | 44.2 | 12.4 | 55.8 | 42.2 |
| sbintuitions/sarashina2.2-0.5b | 793M | 27.3 | 37.0 | 26.7 | 35.8 | 54.8 | 58.1 | 35.1 | 19.4 | 27.8 | 72.7 | 30.3 | 66.0 | 27.3 | 23.6 | 36.8 | 41.3 |
Not measured: Qwen/Qwen3-0.6B-Base, facebook/MobileLLM-R1-950M-base, llm-jp/llm-jp-3-980m, google/gemma-3-1b-pt
On the English average (30.3) this model ranks 15 of 16. On the Japanese average (32.0) it ranks 7 of 16.
| Task | Language | lm-eval task | Shots | Metric |
|---|---|---|---|---|
| MMLU | en | mmlu | 5 | acc |
| ARC-e | en | arc_easy | 0 | acc_norm |
| ARC-c | en | arc_challenge | 0 | acc_norm |
| HellaSwag | en | hellaswag | 0 | acc_norm |
| WinoGrande | en | winogrande | 0 | acc |
| PIQA | en | piqa | 0 | acc_norm |
| LAMBADA | en | lambada_openai | 0 | acc |
| GSM8K | en | gsm8k | 5 | exact_match |
| MMLU-ja | ja | global_mmlu_full_ja | 5 | acc |
| JComQA | ja | nebula_jcommonsenseqa | 3 | acc |
| JNLI | ja | nebula_jnli | 3 | acc |
| XWinograd-ja | ja | ja_leaderboard_xwinograd | 0 | acc |
| Belebele-ja | ja | belebele_jpn_Jpan | 0 | acc |
| MGSM-ja | ja | ja_leaderboard_mgsm | 5 | acc |
Avg (en) and Avg (ja) are plain means of the task scores of that language. Per-task results are in eval/.
Limitations
- This is a small preview model: about 122M parameters and 2B training tokens.
- It is weaker than models trained on trillions of tokens, as the table shows.
- It is a base model. It is not instruction-tuned and does not follow instructions or chat.
- It may produce false, biased or harmful text. Do not rely on it for facts or for any decision that matters.
Training data
All text was filtered, tokenized with nebula-64k and checked against the evaluation suite (see below).
| Source | Language | License | Share of tokens |
|---|---|---|---|
| HuggingFaceFW/fineweb-edu (sample-10BT) | en | ODC-BY-1.0 | 40% |
| HuggingFaceTB/finemath (finemath-4plus) | en | ODC-BY-1.0 | 5% |
| wikimedia/wikipedia (20231101.en) | en | CC-BY-SA-3.0 | 5% |
| hotchpotch/fineweb-2-edu-japanese (sample_10BT) | ja | ODC-BY-1.0 | 40% |
| wikimedia/wikipedia (20231101.ja) | ja | CC-BY-SA-3.0 | 10% |
- FineWeb-Edu, FineMath and FineWeb-2-Edu-Japanese are released under ODC-BY 1.0; credit goes to their authors. They are filtered CommonCrawl snapshots, and the CommonCrawl terms of use apply to the underlying pages.
- Wikipedia text is used under CC BY-SA 3.0. Credit: Wikipedia contributors. Whether model weights count as an adaptation of share-alike text is legally unsettled.
- Definitions from Wiktionary (English and Japanese editions, extracted by kaikki.org) were read during training, as D1 definition reads. Credit: Wiktionary contributors, CC BY-SA 4.0.
Decontamination. Text is normalized (NFKC, lowercase, no whitespace, punctuation or control characters). A training document is dropped if any 50-character window of it matches a window of the evaluation documents, so an overlap of 75 normalized characters or more is always caught.
About D1 (self-definition anchoring)
D1 reads a rare token's dictionary definition with the model that is being trained, pools the hidden states, stops the gradient, and pulls the token's embedding rows toward that vector. In our searches, run on 16 October 2026, we did not find the same method. This is not proof that nobody has done it. The nearest work is DefinitionEMB (2408.01308, applied after pretraining with a separate encoder) and Token Distillation (2505.20133, applied after pretraining to initialize new tokens).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("LoNebula/Nebula-1-ablation-a1")
model = AutoModelForCausalLM.from_pretrained("LoNebula/Nebula-1-ablation-a1")
inputs = tok("The capital of Japan is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
The beginning-of-sequence and end-of-sequence token is <|endoftext|> (id 0). The context length is 2048.
- Downloads last month
- 10