qawel — 9.6M from-scratch research model

qawel is a 9,592,720-parameter English causal language model, trained from random initialization with Labqoat. This release follows three of our own scratch checkpoints, then an ARC training-question adaptation phase with general-text replay. It inherits no external pretrained model weights. Its tokenizer is our own 4,096-token digit-atomic byte-level BPE.

This is a compact research model, with limited factual reliability and repetitive generation. It is not instruction-tuned, a chat assistant, or evaluated for safe deployment. In a retained generation smoke check, it completed “Water freezes when” with “it is heated” and repeated phrases. The benchmark results below do not establish useful factual generation.

Verified evaluation

Full evaluation of the exact retained native checkpoint completed on 2026-10-09. The receipt binds the checkpoint, accepted evaluator source, dataset revisions, unique parameter count, all three task counts, per-item scores and scratch ancestry. The separately verified Hugging Face export preserves all 128 parameter tensors bit for bit and produces identical FP32 logits on three reload prompts.

Metric	Result	Scope
BLiMP	70.30149254%	47,102 / 67,000 pairs; ties count as incorrect
ARC-Easy	49.83164983%	1,184 / 2,376 test questions
WikiText-2 historical byte-PPL	3.0211197144	Full 1,292,008-byte test text
WikiText-2 direct NLL/byte PPL	3.0211118602	exp(total_nll / UTF8_bytes)
Raw overall score	70.48742512	Pinned Tiny-ML normalization
Efficiency score	78.59056716	Raw score × 1.1149586898 size multiplier

Protocol: glint-1.3-c3f99e2-context256-byte-v1, derived from the pinned Glint-1.3 benchmark. It uses bare ARC questions, full-minus-question summed likelihood without length normalization, no BOS, first-256-token sentence clipping, and reset 256-target WikiText windows. This differs from stock lm-evaluation-harness conventions. The score uses the historical conversion token_ppl ** (encoded_tokens / UTF8_bytes); direct byte-PPL is retained separately. Evaluation used FP32, batch 32, on an NVIDIA L40S.

Against the 85-row Tiny-ML snapshot at 3795f68a, this score would insert sixth by efficiency and 21st by raw overall score. The existing tenth score is 76.93725471, a 1.65331246-point margin. This is a conditional mathematical insertion, not official leaderboard acceptance or proof of superiority under each competitor's original harness. Both frozen normalization and normalization recomputed with qawel give the same estimate. See comparison.

Architecture and training

The model has 14 distinct Transformer blocks, width 240, SwiGLU hidden width 640, six query heads and two KV heads. It uses RoPE with theta 10,000, pre-RMSNorm, query/key RMSNorm, bias-free linear layers, tied input/output embeddings and a 512-token context. There is one pass through the blocks, with no recurrent sharing or coda block. The HF wrapper does not implement a KV cache.

Retained phase	Selected targets	Completed targets through this retained checkpoint	Cumulative selected targets
Random-initialized pilot	500,039,680	500,039,680	500,039,680
General-text milestone	2,097,152,000	2,097,152,000	2,597,191,680
Terminal cooldown	500,039,680	500,039,680	3,097,231,360
ARC adaptation + replay	10,656,220	10,656,220	3,107,887,580

Targets are scored target presentations, not unique corpus tokens or FLOPs. The first phase samples a 100,004,195-token pool with replacement, approximately five nominal pool presentations. The general-text milestone was extracted at step 16,000 of a longer requested schedule; it does not represent completion of the requested 20B horizon. The final phase includes 1,598,376 QA scored positions and 9,057,844 general replay targets. Lineage accounting excludes unrelated campaign branches and later work outside these retained checkpoints.

General-text phases use AdamW, peak LR 0.002, weight decay 0.1, sequence 512, batch 64 and four accumulation steps. The last general phase resets optimizer, sampler and schedule for a 500M-target cooldown, with 100 warmup steps, a final-20% cooldown and a 0.05 LR floor. Final ARC adaptation uses AdamW at 5e-5, weight decay 0.01, FP32, three epochs over 3,133 train questions, and a 0.15 listwise QA / 0.85 replay loss mixture. It selects at terminal step 1,175. Loss coefficients and target-exposure fractions are different quantities. See training lineage and exact recipes.

General text comes from pinned FineWeb-Edu, Cosmopedia v2 and FineMath-4plus sources, with approximately 80/15/5% realized tokens in the 4.099B-token available training pool. ARC adaptation uses the train splits only, with all 570 ARC-Easy validation questions reserved for selection. Dataset identities, roles and source attribution are in ATTRIBUTION.md and data-attribution-manifest.json.

Research and contamination limits

This is public-test-informed research. Canonical benchmark measurements influenced later campaign decisions, and development sets were reused. The final job's checkpoint gate used ARC-Easy validation and fixed general-corpus validation, rather than choosing directly on BLiMP, WikiText or ARC test scores; that does not make the whole research process a blind held-out evaluation.

Corpus preparation used exact NFKC document deduplication and protected benchmark pattern filters. These do not establish semantic decontamination or removal of near duplicates. Whether inherited general pretraining text overlaps ARC development questions remains unestablished. The original scratch run did not record its initial runtime random-tensor digest; ancestry is supported by exact accepted source, reports, archive identities, selected tensors and changed-parameter evidence. A later CPU random reference is not the historical initialization. Results describe this primary training lineage and adaptation seed, not training-seed variance. Details remain in the original receipts.

Loading and generation

Review the three local Python model files before enabling custom code. Download the repository into ./qawel, then load the local directory; a remote loader should additionally pin a reviewed repository commit. The verified reload runtime was PyTorch 2.9.0, Transformers 5.19.0, tokenizers 0.23.3 and safetensors 0.8.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = "./qawel"
tokenizer = AutoTokenizer.from_pretrained(
    model_dir, trust_remote_code=True, local_files_only=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    trust_remote_code=True,
    local_files_only=True,
    dtype=torch.float32,
).eval()
inputs = tokenizer("The cat sits on the mat.", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=48, do_sample=False, use_cache=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Use a single unpadded prompt for this example. Forward context is limited to 512 tokens; generation retains the most recent 512 tokens and recomputes the sequence. Larger contexts, reduced precision, factual accuracy and generation quality are not established by the reload smoke checks.

Evidence and reproduction

• Final canonical receipt, SHA256 1b5fe3de82ec1736627de81e8267d053438b9c2d58c09c0bad57b292e62228ff.
• Exact accepted execution sources, including historical pretraining, final adaptation, evaluator and offline evidence-verifier snapshots.
• Final/best/export tensor identity and HF reload verification.
• Package file inventory, reproduction notes, and generation smoke.

The native release Artifact is ad0ee3d1-7e02-4b84-8053-7118f3ff09bf, archive SHA256 2724c4188efb5673f1a3869505dd87648da7556347c3a9dceff9a639addd3225. The HF safetensors file has a different serialization hash because parameter names receive the wrapper prefix and the duplicate tied head is omitted; its learned tensors remain identical.

The same selected native model is also registered as Labqoat Checkpoint 084d30ff-4a45-4721-be3f-182fa76bb0dc, model-only at step 1,175. Registry readback verifies the import owner separately from the original training run; registration executes zero training/evaluation steps and does not provide exact optimizer resume.