jjge007/jaaiva-i1

JAAIVA i1 — 503M from-scratch chatbot (v2.0.0)

Warning: this model is unfinished and unstable. It is a 503M-parameter language model trained from scratch — not a fine-tune of an existing model. It is weaker than essentially every public chatbot, hallucinates, and gives wrong answers confidently. It is a research artifact.

Jaaiva i1 is a 503,377,664-parameter GPT built and trained from random initialization in PyTorch. This repository hosts the v2.0.0 GGUF build: 700,022,784 tokens of pretraining, five SFT stages, and a dedicated extremely-simple-math stage.

What changed in v2.0.0

	v0.2.0	v2.0.0
Math battery	1/10	9/10
Overall battery	17/52 (33%)	28/52 (54%)
Self-name	"jaaiva i1"	"jaaiva i1"
Creator	jjgd008	says "jjge008"

Stage 5 is a 1,200-step SFT run on 4,525 generated extremely-simple math pairs — one-digit facts, two-digit sums and differences, times tables, exact division, percentages, tiny word problems, letter counts, comparisons and sequences — mixed with UltraChat, Dolly and identity replay so chatting doesn't collapse. The same stage rebakes the creator string to jjge008.

Files

File	Size	Contents
jaaiva-i1-503m-f16.gguf	0.94 GiB	v2.0.0 weights, f16, llama.cpp

Quick start (llama.cpp)

# huggingface-cli download jjge007/jaaiva-i1 jaaiva-i1-503m-f16.gguf
llama-cli -m jaaiva-i1-503m-f16.gguf \
  -p "<|bos|><|user|>What is 7 + 5?<|assistant|>" \
  -n 80 --temp 0.5 --top-k 30 --top-p 0.85 --repeat-penalty 1.15

Or as an OpenAI-compatible server:

llama-server -m jaaiva-i1-503m-f16.gguf --port 8080 -c 1024
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is 12 times 9?"}],"temperature":0.5}'

Prompt format

Special tokens: <|pad|> (0), <|bos|> (1), <|eos|> (2), <|user|> (3), <|assistant|> (4). Prompts are:

<|bos|>[<|user|>...<|assistant|>...]<|user|>question<|assistant|>

No system slot; stop on <|eos|>.

Example conversation

Greedy decoding (temperature 0) from the released checkpoint, each prompt on a fresh context:

You: What is your name?

Jaaiva i1: my name is jaaiva i1.

You: Who created you?

Jaaiva i1: jjge008 created me as a from-scratch neural language model.

You: What is 7 + 5?

Jaaiva i1: 7 + 5 = 12.

You: What is 37 + 46?

Jaaiva i1: 37 + 46 = 83.

You: What is 13 times 6?

Jaaiva i1: 13 × 6 = 78.

You: What is 10% of 200?

Jaaiva i1: 10% of 200 = 20.

Architecture

Parameters	503,377,664 (~503M)
Layers / heads / width	25 / 20 / 1280
Context	1024 tokens
Vocab	8,192 (byte-level BPE)
Blocks	Pre-LN, fused QKV, GELU MLP, tied embeddings
Positions	Learned absolute (GPT-2 style)
Precision	bf16 training, f16 GGUF

Training

Pretraining. 700,022,784 tokens, 7,121 steps, 98,304 tokens/step, AdamW with cosine decay, bf16 on an AMD RX 9070 XT (ROCm). Final validation loss 1.4176 (perplexity 4.13).

SFT.

1. Identity + multi-turn — 400 steps.
2. Instruction — 2,500 steps on UltraChat + Dolly.
3. Identity polish — 300 steps.
4. Targeted general purpose — 700 steps on 2,643 pairs.
5. Simple math + rename — 1,200 steps on 4,525 generated math pairs, with identity replay.

Loss is masked to assistant tokens only.

Evaluation

Fixed 52-prompt battery (eval_general.py), greedy decoding, keyword matching. Prompts overlap the stage-4/5 training data on purpose: this measures capability, not held-out purity.

Category	v2.0.0	v0.2.0	pretrain base
math	9/10	1/10	1/10
knowledge	8/16	7/16	2/16
reasoning	1/6	1/6	1/6
instruction	4/8	4/8	1/8
coding	2/5	1/5	0/5
honesty	4/7	3/7	0/7
total	28/52 (54%)	17/52 (33%)	5/52 (10%)

Repetitive answers (trigram repetition > 20%): 0%.

Conversion

Converted from the stage-5 checkpoint with the project's convert_gguf.py (all weights f16), then loaded and exercised through llama.cpp / Ollama.

Limitations

• 503M parameters is tiny; arithmetic beyond the simple templates still breaks, and long or novel word problems will fail.
• The math stage is template-generated and overlaps the battery. New questions are harder than these scores suggest. GGUF inference is greedy and f16, so a near-tie arithmetic answer can differ from the PyTorch checkpoint by one.
• No system prompt, tools, retrieval, safety tuning or RLHF. English only, 1024-token context, 8k vocabulary.

Licensing and data

Weights and code are released by jjge008. Training data mixes public datasets, each under its own license: UltraChat, Databricks Dolly 15k, DailyDialog, PersonaChat, EmpatheticDialogues, BlendedSkillTalk, SODA, WikiText-103, TinyStories, Wikipedia (CC BY-SA) and FineWeb-Edu (ODC-By). The math and general-purpose pairs are generated locally. Review each source dataset's terms before commercial use.

Built from scratch, out of curiosity. Correctness is not guaranteed.