Release NeuronAI-4B card, benchmark runner, visuals, and non-commercial license

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/checkpoint_selection.png filter=lfs diff=lfs merge=lfs -text

LICENSE.md

@@ -0,0 +1,20 @@
+# NeuronAI-4B License
+
+Copyright 2026 NeuronUz.
+
+NeuronAI-4B is licensed under the **Creative Commons Attribution-NonCommercial
+4.0 International Public License (CC BY-NC 4.0)**.
+
+- License deed: https://creativecommons.org/licenses/by-nc/4.0/
+- Legal code: https://creativecommons.org/licenses/by-nc/4.0/legalcode.en
+
+You may share and adapt the model for non-commercial purposes provided that you
+give appropriate credit, link to the license, and indicate changes. Uses that
+are primarily intended for commercial advantage or monetary compensation are
+not licensed by CC BY-NC 4.0.
+
+For a separate commercial license, contact **neuronaiuz@gmail.com**.
+
+This notice is a convenience summary and does not replace or modify the CC
+BY-NC 4.0 legal code. Third-party components remain subject to their applicable
+licenses and terms.

README.md

@@ -3,168 +3,245 @@ language:
   - uz
   - en
   - ru
+license: cc-by-nc-4.0
 library_name: transformers
 pipeline_tag: text-generation
+base_model: Qwen/Qwen3.5-4B
 tags:
+  - qwen3.5
   - uzbek
-  - qwen3_5
-  - instruct
+  - conversational
   - translation
-base_model: Qwen/Qwen3.5-4B
+  - text-generation-inference
+  - non-commercial
+datasets:
+  - HuggingFaceFW/fineweb-2
+  - tahrirchi/uz-books
+  - tahrirchi/uz-crawl
+  - HuggingFaceFW/fineweb-edu
+  - HuggingFaceTB/finemath
 ---
 
-# NeuronAI 4B (Qwen3.5-4B, Uzbek-adapted)
+# NeuronAI-4B
+
+**NeuronAI-4B** is an Uzbek-first, bilingual assistant model built from
+Qwen3.5-4B. It combines an Uzbek tokenizer retrofit, continued pretraining,
+annealing, and assistant-only supervised fine-tuning. The published weights are
+fully merged—no LoRA adapter is needed.
 
-An Uzbek-focused instruct model built on Qwen3.5-4B: continued pretraining on an
-Uzbek corpus with an extended Uzbek tokenizer, an instruction-data anneal, then
-full-parameter SFT.
+![Strict eight-task benchmark comparison](assets/overall_score.png)
 
-Checkpoint: **step 6565 (epoch 1.25 of a 3-epoch run)**. This is not the last
-checkpoint — it is the best one. Benchmarks were run every half epoch and peaked
-here; later checkpoints scored the same or slightly lower (see *Checkpoint
-selection*).
+> **License:** free for non-commercial use under
+> [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/).
+> Commercial use requires a separate written license. Contact
+> **[neuronaiuz@gmail.com](mailto:neuronaiuz@gmail.com)** to discuss commercial terms.
 
 ## Quick start
 
+Install a recent Transformers build with Qwen3.5 support:
+
+```bash
+pip install -U "transformers>=5.1" accelerate torch
+```
+
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "kmamaroziqov/qwen3.5-4b"
-tok = AutoTokenizer.from_pretrained(model_id)
+model_id = "NeuronUz/NeuronAI-4B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
-    model_id, dtype=torch.bfloat16, device_map="auto"
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
 ).eval()
 
-messages = [{"role": "user", "content": "O'zbekistonning poytaxti qaysi shahar?"}]
-text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-inputs = tok(text, return_tensors="pt").to(model.device)
-
-out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
-print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
-# -> O'zbekistonning poytaxti - Toshkent.
+messages = [
+    {"role": "system", "content": "Siz foydali va aniq AI yordamchisiz."},
+    {"role": "user", "content": "Alisher Navoiy haqida qisqacha aytib bering."},
+]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    enable_thinking=False,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+with torch.inference_mode():
+    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+
+reply = tokenizer.decode(
+    output[0, inputs["input_ids"].shape[1]:],
+    skip_special_tokens=True,
+).strip()
+print(reply)
 ```
 
-No `eos_token_id` argument is needed: `generation_config.json` lists both
-`<|endoftext|>` (248044) and `<|im_end|>` (248046), and the model terminates
-assistant turns with `<|im_end|>`.
+The generation metadata already registers `<|im_end|>` and `<|endoftext|>` as
+end-of-sequence tokens.
 
-### vLLM
+### Serve with vLLM
 
-This is a **hybrid attention/Mamba (GDN)** architecture. vLLM needs three extra
-arguments or it fails at load with `assert mamba_block_size is not None`:
+```bash
+pip install -U vllm
+vllm serve NeuronUz/NeuronAI-4B \
+  --dtype bfloat16 \
+  --max-model-len 4096 \
+  --language-model-only \
+  --mamba-block-size 16 \
+  --mamba-cache-mode align
+```
 
-```python
-from vllm import LLM, SamplingParams
-
-llm = LLM(
-    model="kmamaroziqov/qwen3.5-4b",
-    dtype="bfloat16",
-    max_model_len=4096,
-    mamba_block_size=16,        # required
-    mamba_cache_mode="align",   # required
-    language_model_only=True,   # text-only checkpoint
-)
-params = SamplingParams(temperature=0.0, max_tokens=256,
-                        stop_token_ids=[248044, 248046])
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"NeuronUz/NeuronAI-4B","messages":[{"role":"user","content":"O‘zbekiston haqida uchta fakt ayting."}]}'
 ```
 
-### Chat format
+## Benchmarks
+
+All four models below were evaluated on the same full eight-task suite with
+greedy vLLM decoding. Classification and multiple-choice tasks use accuracy;
+FLORES+ translation uses COMET. The weighted score is normalized by the 0.95
+sum of the published task weights. All eight NeuronAI-4B tasks completed and
+passed the invalid-output gate.
+
+![Per-task comparison](assets/tasks_comparison.png)
+
+| Benchmark | Metric | Weight | **NeuronAI-4B** | Qwen3.5-4B | alloma-3B | alloma-1B |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| UzLiB | accuracy | 0.20 | **61.20%** | 40.30% | 32.08% | 23.32% |
+| TUMLU-Uzbek | accuracy | 0.20 | **45.00%** | 40.43% | 27.71% | 22.00% |
+| FLORES+ en→uz | COMET | 0.15 | **0.8965** | 0.8555 | 0.8673 | 0.7383 |
+| Uzbek news | accuracy | 0.10 | **79.15%** | 67.34% | 13.60% | 25.41% |
+| MMLU English | accuracy | 0.10 | 64.06% | **72.66%** | 38.73% | 21.98% |
+| MMLU Uzbek | accuracy | 0.10 | **57.01%** | 52.58% | 32.74% | 21.11% |
+| FLORES+ uz→en | COMET | 0.05 | **0.8763** | 0.8618 | 0.7954 | 0.7636 |
+| Uzbek sentiment | accuracy | 0.05 | **95.75%** | 84.82% | 38.85% | 79.54% |
+| **Normalized weighted score** | | 1.00 | **0.6724** | 0.5978 | 0.4147 | 0.3661 |
+
+Alloma runs used the `APST` apostrophe preprocessing required by their model
+cards; NeuronAI and stock Qwen did not. The raw archived scores and run IDs are
+included in [`benchmark_results.json`](benchmark_results.json).
+
+### Run the benchmarks on your computer
+
+The repository includes a portable Alloma-style benchmark runner. It covers
+FLORES+ (both directions), Uzbek sentiment, Uzbek news, MMLU English, MMLU Uzbek,
+and TUMLU-Uzbek.
+
+```bash
+pip install -r https://huggingface.co/NeuronUz/NeuronAI-4B/resolve/main/benchmark-requirements.txt
+wget https://huggingface.co/NeuronUz/NeuronAI-4B/resolve/main/benchmark.py
+python benchmark.py --limit 200 --output quick-results.json
+```
 
-ChatML. The template emits a **pre-closed, empty** thinking block
-(`<think>\n\n</think>`) — this model is trained in non-thinking mode, so it
-answers directly and does not produce reasoning traces.
+The quick command uses the same seed on 200 examples per dataset. Run all public
+examples and add COMET with:
 
+```bash
+pip install unbabel-comet
+python benchmark.py --limit 0 --comet --output full-results.json
 ```
-<|im_start|>user
-{message}<|im_end|>
-<|im_start|>assistant
-<think>
 
-</think>
+Run one task when you only need a short check:
 
-{reply}<|im_end|>
+```bash
+python benchmark.py --tasks mmlu-uz --limit 200 --output mmlu-uz.json
+python benchmark.py --tasks flores --limit 200 --output flores.json
 ```
 
-## Training
-
-| stage | detail |
-|---|---|
-| base | Qwen3.5-4B |
-| tokenizer | extended for Uzbek |
-| continued pretraining | Uzbek corpus |
-| anneal | bilingual instruction data |
-| SFT | full-parameter, 169,919 rows, seq len 2048, effective batch 32, LR 1e-5 cosine, bf16 |
-
-The SFT mixture is Uzbek-dominant with English retention data, benchmark-format
-tasks, translation pairs, and 151 hand-written identity/social records
-(uz/en/ru).
-
-## Evaluation
-
-### Versus alloma models — alloma's own card protocol, full test sets
-
-Their prompts, their datasets, single backend (vLLM). alloma runs use the `APST`
-apostrophe preprocessing their card requires; this model does not (it was never
-trained on it). The harness reproduces alloma's published numbers within ~1.3 on
-every metric except news (see *Caveats*).
-
-| metric | alloma-1B | alloma-3B | **this model** | alloma-8B |
-|---|---|---|---|---|
-| params | 1.24B | 3.21B | **4.21B** | 8.03B |
-| BLEU Uz→En | 16.10 | 24.62 | 29.64 | **30.02** |
-| BLEU En→Uz | 8.70 | 14.54 | 14.34 | **15.25** |
-| COMET Uz→En | 80.08 | 85.14 | 86.63 | **87.13** |
-| COMET En→Uz | 81.20 | 86.80 | **87.89** | 87.79 |
-| Sentiment | 58.05 | 81.53 | **89.50** | 79.94 |
-| News | 15.85 | 33.78 | **58.34** | 57.77 |
-| MMLU (English) | 28.61 | 45.34 | **63.20** | 53.47 |
-
-Four of seven metrics beat an 8B model; the three losses are all under 1 point.
-
-### Internal 8-task suite (own prompts, full sets)
-
-| task | score | invalid |
-|---|---|---|
-| uzlib (Uzbek knowledge MCQ) | 0.6040 | 0% |
-| tumlu | 0.4357 | 0% |
-| MMLU English | 0.6402 | 0% |
-| MMLU Uzbek | 0.5667 | 0% |
-| news | 0.7979 | 0% |
-| sentiment | 0.9615 | 0% |
-| FLORES en→uz | BLEU 12.85 (len-ratio 0.961) | — |
-| FLORES uz→en | BLEU 29.23 (len-ratio 1.024) | — |
-| **weighted** | **0.5162** | |
-
-### Checkpoint selection
-
-Held-out loss and downstream benchmarks **disagree** on this run, so checkpoints
-were chosen by benchmark:
-
-| epoch | eval_loss | weighted score |
-|---|---|---|
-| 0.75 | **1.430** (minimum) | 0.5081 |
-| **1.25** | 1.433 | **0.5162** ← this checkpoint |
-| 1.75 | 1.449 | 0.5156 |
-
-eval_loss bottomed at epoch 0.75 while benchmarks kept improving to 1.25. Picking
-by eval_loss would have shipped the weaker model.
-
-## Caveats
-
-- **BLEU understates into-Uzbek quality.** Uzbek is agglutinative, so one correct
-  meaning has many valid surface forms and n-gram overlap collapses. This model
-  has the *highest* En→Uz COMET of the models compared while sitting mid-pack on
-  En→Uz BLEU. Prefer COMET for into-Uzbek.
-- **News numbers are not comparable to alloma's published card.** Our replication
-  ran +5.1 to −7.9 off their published news figures with no consistent direction,
-  likely an input-truncation or label-matching difference. Model-to-model
-  comparisons in the table above are still like-for-like.
-- **Prompt sensitivity is high on classification.** Swapping prompt formats moved
-  news accuracy by up to 32 points on some models. Invest in prompt design for
-  your actual application rather than assuming these numbers transfer.
-- **English is traded for Uzbek.** Continued pretraining costs English relative to
-  the base model; a Base+SFT control scored higher on English MMLU.
-- Not safety-tuned or RLHF-aligned. Can produce incorrect or outdated facts.
-- No Cyrillic-Uzbek or Russian-instruction coverage in the SFT mixture, so quality
-  in those settings is unverified.
+`--limit 0` means the full dataset. Only full runs are comparable with the table
+above; 200-example quick runs are sanity checks. COMET downloads the
+`Unbabel/wmt22-comet-da` evaluator and needs additional disk/RAM.
+
+## Why this checkpoint
+
+Four retained checkpoints were run through the complete suite. Checkpoint 4749
+(one SFT epoch) produced the best downstream score even though development loss
+was fractionally lower at checkpoint 3564.
+
+![Checkpoint selection](assets/checkpoint_selection.png)
+
+| Checkpoint | Epoch | Dev loss | Normalized score |
+| ---: | ---: | ---: | ---: |
+| 1188 | 0.25 | 1.4316 | 0.6643 |
+| 2376 | 0.50 | 1.3910 | 0.6600 |
+| 3564 | 0.75 | **1.3669** | 0.6609 |
+| **4749** | **1.00** | 1.3675 | **0.6724** |
+
+## Uzbek tokenizer efficiency
+
+The tokenizer is an in-place Uzbek retrofit rather than a vocabulary extension.
+On a fixed seed-42 sample of 20,000 uz-crawl documents (4.79 million whitespace
+words), it used 2.41 tokens per word versus 3.43 for stock Qwen3.5—a 29.7%
+reduction for the same Uzbek text.
+
+![Uzbek tokenizer fertility](assets/tokenizer_fertility.png)
+
+This corpus was part of continued pretraining, so the fertility result is
+in-domain. It measures tokenization efficiency, not model quality or decoding
+speed. The 4B and 2B NeuronAI releases use byte-identical tokenizer files.
+
+## Model lineage and training
+
+1. `Qwen/Qwen3.5-4B`
+2. Uzbek tokenizer retrofit with embedding re-initialization
+3. Uzbek continued pretraining
+4. Instruction-data annealing
+5. Assistant-only SFT with LoRA r64
+6. Checkpoint-4749 adapter merged into the base weights
+
+| Item | Value |
+| --- | --- |
+| Parameters | 4.206B |
+| Train examples | 152,152 |
+| Grouped dev examples | 1,537 |
+| Train/dev prompt-group overlap | 0 |
+| Sequence length during SFT | 2,048 |
+| SFT epochs | 1 |
+| Effective batch size | 32 |
+| Learning rate | 1e-4, cosine, 3% warmup |
+| LoRA | rank 64, alpha 128, dropout 0.05, all linear layers |
+| Loss | assistant turns only; `<|im_end|>` explicitly supervised |
+| Precision | bf16; selected recurrent parameters retained in fp32 |
+
+The mixture is Uzbek-first and includes general assistant conversations,
+translation, Uzbek language and literature, spelling, classification, math,
+and English-retention examples. Training data is not distributed with this
+model repository.
+
+## Intended use
+
+Good fits include non-commercial Uzbek research, education, prototyping,
+translation experiments, writing assistance, retrieval-augmented generation,
+and local/offline demonstrations.
+
+Commercial deployment, paid products or services, internal business use, and
+other activity primarily intended for commercial advantage require a separate
+license from NeuronUz. Email [neuronaiuz@gmail.com](mailto:neuronaiuz@gmail.com).
+
+## Limitations
+
+- This is a public-suite-selected checkpoint. The benchmark results are useful
+  for reproducibility and relative comparison, but they are not a locked,
+  independent estimate of real-world generalization.
+- Stock Qwen3.5-4B remains stronger on English MMLU in this evaluation.
+- TUMLU-Uzbek is the weakest reported Uzbek task and should not be treated as
+  solved at 45% accuracy.
+- The model can hallucinate, repeat biases in its data, or produce unsafe or
+  outdated content. It has not been comprehensively safety-evaluated.
+- Do not rely on it without expert review for medical, legal, financial, public
+  safety, or other high-stakes decisions.
+- SFT used sequences up to 2,048 tokens; serving at longer inherited context
+  lengths has not been validated here. The published inference examples use
+  4,096 tokens.
+
+## License
+
+NeuronAI-4B is released under
+[Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/).
+You may share and adapt it for non-commercial purposes with attribution. This
+summary does not replace the license text. See [`LICENSE.md`](LICENSE.md) and
+contact [neuronaiuz@gmail.com](mailto:neuronaiuz@gmail.com) for commercial terms.

benchmark-requirements.txt

@@ -0,0 +1,9 @@
+torch
+transformers>=5.1
+accelerate
+datasets
+sacrebleu
+vllm
+
+# Optional, only for `python benchmark.py --comet`:
+# unbabel-comet

benchmark.py

@@ -0,0 +1,355 @@
+#!/usr/bin/env python3
+"""Portable NeuronAI-4B / Alloma-style Uzbek benchmark runner.
+
+Examples:
+  python benchmark.py --limit 200 --output quick-results.json
+  python benchmark.py --limit 0 --comet --output full-results.json
+
+`--limit 0` evaluates every example. The default 200-example run is a quick,
+deterministically sampled sanity check and must not be compared with the full
+scores in the model card.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from datetime import datetime, timezone
+from pathlib import Path
+
+import torch
+from datasets import concatenate_datasets, load_dataset
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+
+MODEL_ID = "NeuronUz/NeuronAI-4B"
+LETTERS = "ABCD"
+
+TRANSLATION_PROMPTS = {
+    "uz-en": (
+        "Translate each Uzbek sentence into English.\n\n"
+        "1991-yilning 1-sentabrida O'zbekiston mustaqilligini e'lon qildi.\n"
+        "-> On 1 September 1991, Uzbekistan declared its independence.\n\n"
+        "Tadqiqotchilar yangi usul samaradorligi 47 foizga oshganini aniqladilar.\n"
+        "-> Researchers found that the new method improved efficiency by 47 percent.\n\n"
+        "{text}\n->"
+    ),
+    "en-uz": "Translate into Uzbek:\n\n{text}",
+}
+
+SENTIMENT_PROMPT = (
+    "Given the following Uzbek text, determine the sentiment as either "
+    "'Positive' or 'Negative'. Respond with only one label.\n\nText: {text}\n\nLabel:"
+)
+
+NEWS_PROMPT = """Classify the given Uzbek news article into one category. Respond with only the category number.
+
+0 - Siyosat
+1 - Iqtisodiyot
+2 - Texnologiya
+3 - Sport
+4 - Madaniyat
+5 - Salomatlik
+6 - Oila va Jamiyat
+7 - Ta'lim
+8 - Ekologiya
+9 - Xorijiy Yangiliklar
+
+Article: {text}
+
+Answer:"""
+
+MCQ_SUFFIX = {
+    "uz": "Variantlarni diqqat bilan solishtiring. Javobni A, B, C yoki D harfi bilan boshlang.",
+    "en": "Compare the options carefully. Start with the answer letter A, B, C, or D.",
+}
+
+MCQ_TASKS = {
+    "mmlu-en": ("cais/mmlu", "all", "test", "en"),
+    "mmlu-uz": ("murodbek/MMLU-uz", "default", "test", "uz"),
+    "tumlu": ("jafarisbarov/TUMLU-mini", "uzbek", "test", "uz"),
+}
+
+
+def choose_rows(dataset, limit: int, seed: int):
+    if limit and len(dataset) > limit:
+        return dataset.shuffle(seed=seed).select(range(limit))
+    return dataset
+
+
+def strip_thinking(text: str) -> str:
+    return re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL).strip()
+
+
+class Generator:
+    def __init__(self, args: argparse.Namespace):
+        self.backend = args.backend
+        self.tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+        self.tokenizer.padding_side = "left"
+        if self.tokenizer.pad_token_id is None:
+            self.tokenizer.pad_token_id = self.tokenizer.eos_token_id
+
+        if args.backend == "vllm":
+            from vllm import LLM, SamplingParams
+
+            self.sampling_cls = SamplingParams
+            self.model = LLM(
+                model=args.model,
+                dtype=args.dtype,
+                trust_remote_code=True,
+                gpu_memory_utilization=args.gpu_memory_utilization,
+                max_model_len=args.max_model_len,
+                language_model_only=True,
+                mamba_block_size=16,
+                mamba_cache_mode="align",
+            )
+        else:
+            dtype = torch.bfloat16 if args.dtype == "bfloat16" else torch.float16
+            self.model = AutoModelForCausalLM.from_pretrained(
+                args.model,
+                dtype=dtype,
+                device_map="auto",
+                trust_remote_code=True,
+            ).eval()
+
+    def render(self, prompt: str) -> str:
+        messages = [{"role": "user", "content": prompt}]
+        try:
+            return self.tokenizer.apply_chat_template(
+                messages,
+                tokenize=False,
+                add_generation_prompt=True,
+                enable_thinking=False,
+            )
+        except TypeError:
+            return self.tokenizer.apply_chat_template(
+                messages, tokenize=False, add_generation_prompt=True
+            )
+
+    def generate(self, prompts: list[str], max_new_tokens: int, batch_size: int) -> list[str]:
+        rendered = [self.render(prompt) for prompt in prompts]
+        if self.backend == "vllm":
+            params = self.sampling_cls(temperature=0.0, max_tokens=max_new_tokens)
+            outputs = self.model.generate(rendered, params)
+            return [strip_thinking(item.outputs[0].text) for item in outputs]
+
+        results: list[str] = []
+        for start in range(0, len(rendered), batch_size):
+            batch = rendered[start : start + batch_size]
+            encoded = self.tokenizer(
+                batch,
+                return_tensors="pt",
+                padding=True,
+                truncation=True,
+                max_length=4096,
+            ).to(self.model.device)
+            prompt_width = encoded["input_ids"].shape[1]
+            with torch.inference_mode():
+                output = self.model.generate(
+                    **encoded,
+                    max_new_tokens=max_new_tokens,
+                    do_sample=False,
+                )
+            results.extend(
+                strip_thinking(text)
+                for text in self.tokenizer.batch_decode(
+                    output[:, prompt_width:], skip_special_tokens=True
+                )
+            )
+        return results
+
+
+def load_flores(direction: str, limit: int, seed: int) -> list[dict[str, str]]:
+    langs = {"uz-en": ("uzn_Latn", "eng_Latn"), "en-uz": ("eng_Latn", "uzn_Latn")}
+    src_lang, ref_lang = langs[direction]
+    src = concatenate_datasets([
+        load_dataset("openlanguagedata/flores_plus", src_lang, split="dev"),
+        load_dataset("openlanguagedata/flores_plus", src_lang, split="devtest"),
+    ])
+    ref = concatenate_datasets([
+        load_dataset("openlanguagedata/flores_plus", ref_lang, split="dev"),
+        load_dataset("openlanguagedata/flores_plus", ref_lang, split="devtest"),
+    ])
+    pairs = [
+        {"src": src[index]["text"].strip(), "ref": ref[index]["text"].strip()}
+        for index in range(min(len(src), len(ref)))
+    ]
+    if limit and len(pairs) > limit:
+        import random
+
+        random.Random(seed).shuffle(pairs)
+        pairs = pairs[:limit]
+    return pairs
+
+
+def score_comet(sources: list[str], hypotheses: list[str], references: list[str]) -> float:
+    from comet import download_model, load_from_checkpoint
+
+    checkpoint = download_model("Unbabel/wmt22-comet-da")
+    model = load_from_checkpoint(checkpoint)
+    rows = [
+        {"src": src, "mt": hypothesis, "ref": reference}
+        for src, hypothesis, reference in zip(sources, hypotheses, references, strict=True)
+    ]
+    return float(model.predict(rows, batch_size=8, gpus=1 if torch.cuda.is_available() else 0).system_score)
+
+
+def evaluate_flores(generator: Generator, args: argparse.Namespace) -> dict:
+    import sacrebleu
+
+    results = {}
+    for direction in ("uz-en", "en-uz"):
+        pairs = load_flores(direction, args.limit, args.seed)
+        prompts = [TRANSLATION_PROMPTS[direction].format(text=row["src"]) for row in pairs]
+        hypotheses = generator.generate(prompts, max_new_tokens=160, batch_size=args.batch_size)
+        references = [row["ref"] for row in pairs]
+        sources = [row["src"] for row in pairs]
+        row = {
+            "total": len(pairs),
+            "bleu": float(sacrebleu.corpus_bleu(hypotheses, [references]).score),
+            "samples": [
+                {"source": src, "prediction": hyp, "reference": ref}
+                for src, hyp, ref in zip(sources[:3], hypotheses[:3], references[:3])
+            ],
+        }
+        if args.comet:
+            row["comet"] = score_comet(sources, hypotheses, references)
+        results[direction] = row
+        print(f"FLORES+ {direction}: BLEU={row['bleu']:.2f}" + (f", COMET={row['comet']:.4f}" if args.comet else ""))
+    return results
+
+
+def label_to_int(raw, names: list[str]) -> int | None:
+    if isinstance(raw, int) and 0 <= raw < len(names):
+        return raw
+    cleaned = str(raw).strip().casefold().replace("’", "'")
+    for index, name in enumerate(names):
+        if cleaned == name.casefold():
+            return index
+    return None
+
+
+def evaluate_classification(generator: Generator, args: argparse.Namespace, task: str) -> dict:
+    if task == "sentiment":
+        dataset = load_dataset("behbudiy/uzbek-sentiment-analysis", split="train")
+        names = ["Negative", "Positive"]
+        rows = [
+            {"text": row["text"], "gold": label_to_int(row["label"], names)}
+            for row in choose_rows(dataset, args.limit, args.seed)
+        ]
+        prompt_template = SENTIMENT_PROMPT
+        parser = lambda text: 1 if text.casefold().startswith("positive") else (0 if text.casefold().startswith("negative") else None)
+    else:
+        dataset = load_dataset("risqaliyevds/uzbek-zero-shot-classification", split="train")
+        names = ["Siyosat", "Iqtisodiyot", "Texnologiya", "Sport", "Madaniyat",
+                 "Salomatlik", "Oila va Jamiyat", "Ta'lim", "Ekologiya", "Xorijiy Yangiliklar"]
+        rows = [
+            {"text": row["text"], "gold": label_to_int(row["class"], names)}
+            for row in choose_rows(dataset, args.limit, args.seed)
+        ]
+        prompt_template = NEWS_PROMPT
+        parser = lambda text: int(match.group()) if (match := re.search(r"\d", text)) else None
+
+    rows = [row for row in rows if row["gold"] is not None]
+    prompts = [prompt_template.format(text=row["text"][: args.max_text_chars]) for row in rows]
+    outputs = generator.generate(prompts, max_new_tokens=8, batch_size=args.batch_size)
+    predictions = [parser(output.strip()) for output in outputs]
+    correct = sum(prediction == row["gold"] for prediction, row in zip(predictions, rows, strict=True))
+    invalid = sum(prediction is None for prediction in predictions)
+    result = {
+        "accuracy": correct / len(rows),
+        "correct": correct,
+        "total": len(rows),
+        "invalid_rate": invalid / len(rows),
+    }
+    print(f"{task}: accuracy={result['accuracy']:.2%} ({correct}/{len(rows)}), invalid={invalid}")
+    return result
+
+
+def answer_letter(raw) -> str | None:
+    if isinstance(raw, int) and 0 <= raw < 4:
+        return LETTERS[raw]
+    cleaned = str(raw).strip().upper()
+    return cleaned[0] if cleaned and cleaned[0] in LETTERS else None
+
+
+def evaluate_mcq(generator: Generator, args: argparse.Namespace, task: str) -> dict:
+    dataset_name, config, split, language = MCQ_TASKS[task]
+    dataset = choose_rows(load_dataset(dataset_name, config, split=split), args.limit, args.seed)
+    rows = []
+    for row in dataset:
+        choices = row.get("choices") or [row.get(f"option_{letter.lower()}") for letter in LETTERS]
+        choices = [str(choice) for choice in choices if choice is not None]
+        gold = answer_letter(row.get("answer"))
+        if row.get("question") and len(choices) >= 4 and gold:
+            rows.append({"question": row["question"], "choices": choices[:4], "gold": gold})
+    prompts = []
+    for row in rows:
+        choices = "\n".join(f"{letter}) {choice}" for letter, choice in zip(LETTERS, row["choices"]))
+        prompts.append(f"{row['question']}\n\n{choices}\n\n{MCQ_SUFFIX[language]}")
+    outputs = generator.generate(prompts, max_new_tokens=12, batch_size=args.batch_size)
+    predictions = []
+    for output in outputs:
+        match = re.search(r"[ABCD]", output.upper())
+        predictions.append(match.group() if match else None)
+    correct = sum(prediction == row["gold"] for prediction, row in zip(predictions, rows, strict=True))
+    invalid = sum(prediction is None for prediction in predictions)
+    result = {
+        "accuracy": correct / len(rows),
+        "correct": correct,
+        "total": len(rows),
+        "invalid_rate": invalid / len(rows),
+    }
+    print(f"{task}: accuracy={result['accuracy']:.2%} ({correct}/{len(rows)}), invalid={invalid}")
+    return result
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", default=MODEL_ID)
+    parser.add_argument("--tasks", default="flores,sentiment,news,mmlu-en,mmlu-uz,tumlu")
+    parser.add_argument("--backend", choices=["vllm", "transformers"], default="vllm")
+    parser.add_argument("--limit", type=int, default=200, help="Examples per dataset; 0 means full dataset.")
+    parser.add_argument("--batch-size", type=int, default=16)
+    parser.add_argument("--max-text-chars", type=int, default=4000)
+    parser.add_argument("--max-model-len", type=int, default=4096)
+    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
+    parser.add_argument("--dtype", choices=["bfloat16", "float16"], default="bfloat16")
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--comet", action="store_true", help="Download WMT22-COMET-DA and score FLORES+.")
+    parser.add_argument("--output", type=Path, default=Path("neuronai-4b-benchmark.json"))
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    if args.limit < 0:
+        raise ValueError("--limit must be 0 or greater")
+    tasks = {task.strip() for task in args.tasks.split(",") if task.strip()}
+    unknown = tasks - {"flores", "sentiment", "news", *MCQ_TASKS}
+    if unknown:
+        raise ValueError(f"Unknown tasks: {sorted(unknown)}")
+    generator = Generator(args)
+    results = {
+        "model": args.model,
+        "timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+        "limit_per_dataset": args.limit,
+        "seed": args.seed,
+        "backend": args.backend,
+        "results": {},
+    }
+    if "flores" in tasks:
+        results["results"]["flores"] = evaluate_flores(generator, args)
+    for task in ("sentiment", "news"):
+        if task in tasks:
+            results["results"][task] = evaluate_classification(generator, args, task)
+    for task in MCQ_TASKS:
+        if task in tasks:
+            results["results"][task] = evaluate_mcq(generator, args, task)
+    args.output.write_text(json.dumps(results, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+    print(f"Wrote {args.output}")
+
+
+if __name__ == "__main__":
+    main()

benchmark_results.json

@@ -0,0 +1,239 @@
+{
+  "schema_version": 1,
+  "metric_note": "Accuracy for non-translation tasks; COMET for FLORES+; weighted sum normalized by 0.95.",
+  "weights": {
+    "uzlib_slice_full": 0.2,
+    "tumlu_uzbek_full": 0.2,
+    "flores_en_uz_full": 0.15,
+    "news_full": 0.1,
+    "mmlu_english_full": 0.1,
+    "mmlu_uz_full": 0.1,
+    "flores_uz_en_full": 0.05,
+    "sentiment_binary_full": 0.05
+  },
+  "labels": {
+    "uzlib_slice_full": "UzLiB",
+    "tumlu_uzbek_full": "TUMLU-Uzbek",
+    "flores_en_uz_full": "FLORES+ en to uz",
+    "news_full": "News",
+    "mmlu_english_full": "MMLU English",
+    "mmlu_uz_full": "MMLU Uzbek",
+    "flores_uz_en_full": "FLORES+ uz to en",
+    "sentiment_binary_full": "Sentiment"
+  },
+  "models": {
+    "NeuronAI-4B": {
+      "run_id": "qwen35_4b_lora_r64_grouped_ckpt4749_public",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
+      "tasks": {
+        "uzlib_slice_full": 0.6120365394948952,
+        "tumlu_uzbek_full": 0.45,
+        "flores_en_uz_full": 0.8965457885374531,
+        "news_full": 0.7914612766835104,
+        "mmlu_english_full": 0.6406494801310355,
+        "mmlu_uz_full": 0.5700754878222476,
+        "flores_uz_en_full": 0.8762970898743587,
+        "sentiment_binary_full": 0.9575
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0
+      },
+      "normalized_weighted_score": 0.6724185843547309
+    },
+    "Qwen3.5-4B": {
+      "run_id": "qwen35_4b_instruct_full_20260817_081728",
+      "suite_id": "public_uzbek_full_vllm",
+      "tasks": {
+        "uzlib_slice_full": 0.4030091348737238,
+        "tumlu_uzbek_full": 0.4042857142857143,
+        "flores_en_uz_full": 0.8554783304304908,
+        "news_full": 0.6734350830153656,
+        "mmlu_english_full": 0.7266058965959266,
+        "mmlu_uz_full": 0.5257798034468024,
+        "flores_uz_en_full": 0.8618190287356948,
+        "sentiment_binary_full": 0.8482
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0145
+      },
+      "normalized_weighted_score": 0.5977513148832162
+    },
+    "alloma-3B": {
+      "run_id": "alloma_3B_apst_full_20260815_142714",
+      "suite_id": "public_uzbek_full_vllm_apst",
+      "tasks": {
+        "uzlib_slice_full": 0.32079527135948416,
+        "tumlu_uzbek_full": 0.27714285714285714,
+        "flores_en_uz_full": 0.8673252673692642,
+        "news_full": 0.1359802000618748,
+        "mmlu_english_full": 0.3872667711152257,
+        "mmlu_uz_full": 0.32744623273038026,
+        "flores_uz_en_full": 0.7954316840740269,
+        "sentiment_binary_full": 0.3885
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0010746910263299302,
+        "tumlu_uzbek_full": 0.28714285714285714,
+        "flores_en_uz_full": null,
+        "news_full": 7.218727441476746e-05,
+        "mmlu_english_full": 0.06900726392251816,
+        "mmlu_uz_full": 0.07726819541375872,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.4405
+      },
+      "normalized_weighted_score": 0.41468665305295505
+    },
+    "alloma-1B": {
+      "run_id": "alloma_1B_apst_full_20260811_114037",
+      "suite_id": "public_uzbek_full_vllm_apst",
+      "tasks": {
+        "uzlib_slice_full": 0.23320795271359485,
+        "tumlu_uzbek_full": 0.22,
+        "flores_en_uz_full": 0.7383331350348018,
+        "news_full": 0.2541404558110756,
+        "mmlu_english_full": 0.21976926363765845,
+        "mmlu_uz_full": 0.21108104258652613,
+        "flores_uz_en_full": 0.7636193416348852,
+        "sentiment_binary_full": 0.7954
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.04567436861902203,
+        "tumlu_uzbek_full": 0.36714285714285716,
+        "flores_en_uz_full": null,
+        "news_full": 0.0003506239043002991,
+        "mmlu_english_full": 0.09393248824953711,
+        "mmlu_uz_full": 0.17882068081469876,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0063
+      },
+      "normalized_weighted_score": 0.3661490569296942
+    }
+  },
+  "checkpoints": {
+    "0.25 epoch": {
+      "run_id": "qwen35_4b_lora_r64_grouped_ckpt1188_public",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
+      "tasks": {
+        "uzlib_slice_full": 0.6093498119290703,
+        "tumlu_uzbek_full": 0.44571428571428573,
+        "flores_en_uz_full": 0.8898104962171044,
+        "news_full": 0.772197586882541,
+        "mmlu_english_full": 0.6312491098134169,
+        "mmlu_uz_full": 0.5549779233727389,
+        "flores_uz_en_full": 0.8765152560768583,
+        "sentiment_binary_full": 0.9377
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0001
+      },
+      "normalized_weighted_score": 0.6642501250231047
+    },
+    "0.50 epoch": {
+      "run_id": "qwen35_4b_lora_r64_grouped_ckpt2376_public",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
+      "tasks": {
+        "uzlib_slice_full": 0.6012896292315959,
+        "tumlu_uzbek_full": 0.44571428571428573,
+        "flores_en_uz_full": 0.8914647964086623,
+        "news_full": 0.7503764050737342,
+        "mmlu_english_full": 0.621919954422447,
+        "mmlu_uz_full": 0.5524854009400371,
+        "flores_uz_en_full": 0.8756085765367719,
+        "sentiment_binary_full": 0.9524
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0
+      },
+      "normalized_weighted_score": 0.6599990603378274
+    },
+    "0.75 epoch": {
+      "run_id": "qwen35_4b_lora_r64_grouped_seed42_control_public",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
+      "tasks": {
+        "uzlib_slice_full": 0.6104245029554003,
+        "tumlu_uzbek_full": 0.39285714285714285,
+        "flores_en_uz_full": 0.8960352256901529,
+        "news_full": 0.8070434154893266,
+        "mmlu_english_full": 0.640364620424441,
+        "mmlu_uz_full": 0.5671556758296539,
+        "flores_uz_en_full": 0.8757696719001098,
+        "sentiment_binary_full": 0.9514
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0
+      },
+      "normalized_weighted_score": 0.6609225976688203
+    },
+    "1.00 epoch": {
+      "run_id": "qwen35_4b_lora_r64_grouped_ckpt4749_public",
+      "suite_id": "public_uzbek_full_comet_vllm_v2",
+      "tasks": {
+        "uzlib_slice_full": 0.6120365394948952,
+        "tumlu_uzbek_full": 0.45,
+        "flores_en_uz_full": 0.8965457885374531,
+        "news_full": 0.7914612766835104,
+        "mmlu_english_full": 0.6406494801310355,
+        "mmlu_uz_full": 0.5700754878222476,
+        "flores_uz_en_full": 0.8762970898743587,
+        "sentiment_binary_full": 0.9575
+      },
+      "invalid_rate": {
+        "uzlib_slice_full": 0.0,
+        "tumlu_uzbek_full": 0.0,
+        "flores_en_uz_full": null,
+        "news_full": 0.0,
+        "mmlu_english_full": 0.0,
+        "mmlu_uz_full": 0.0,
+        "flores_uz_en_full": null,
+        "sentiment_binary_full": 0.0
+      },
+      "normalized_weighted_score": 0.6724185843547309
+    }
+  },
+  "tokenizer_fertility": {
+    "NeuronAI-4B": 2.4068250407285743,
+    "Qwen3.5-4B": 3.4255944034933097
+  },
+  "tokenizer_fertility_provenance": {
+    "corpus": "uz-crawl",
+    "documents": 20000,
+    "words": 4792704,
+    "seed": 42,
+    "note": "NeuronAI-4B and NeuronAI-2B tokenizer.json files are byte-identical (SHA-256 e4731e5d...)."
+  }
+}