BananaMind-2-Medium

BananaMind-2-Medium is a decoder-only causal language model trained from scratch by BananaMind on a 50B-token curriculum.

The model has 49,559,552 parameters, a 3,072-token context window, and a custom 12,288-token digit-aware byte-level BPE tokenizer. It uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

Model Details

Field	Value
Parameters	49,559,552
Architecture	BananaMind2Medium decoder-only Transformer
Layers	12
Hidden size	512
Intermediate size	1,920
Attention heads	8
KV heads	2
Head dim	64
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocabulary size	12,288
Context length	3,072
Embeddings	Tied input/output embeddings
Generation cache	KV cache supported
Weight format	safetensors
HF architecture	BananaMind2MediumForCausalLM
HF model type	bananamind2_medium
Final checkpoint	runs/bananamind2-medium/final.pt
Final training step	90,421
Tokens seen	49,999,749,120

Tokenizer

BananaMind-2-Medium uses a custom 12,288-token byte-level BPE tokenizer trained on 50 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing the tokenizer from merging entire numbers into large number tokens.

Token	ID
0	19
1	20
2	21
3	22
4	23
5	24
6	25
7	26
8	27
9	28

Examples:

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Token	ID
`<	pad
`<	bos
`<	eos
`<	unk

Training Data

The 50B-token training mix combines educational web text, broad web text, synthetic textbook material, mathematics, and the complete local NPSet-2 Python educational corpus.

Dataset	Target Tokens	Aggregate Share
FineWeb-Edu	22.836B	45.67%
DCLM	10.942B	21.88%
Cosmopedia-v2	8.564B	17.13%
FineMath-4+	5.233B	10.47%
NPSet-2 Python Edu	2.424B	4.85%
Total	50.000B	100%

The run used a capacity-aware curriculum rather than sampling the aggregate mix uniformly from the first token.

Phase	Token Range	FineWeb-Edu	DCLM	Cosmopedia-v2	FineMath-4+	NPSet-2 Python Edu
Foundation	0B to 10B	56.47%	28.52%	9.11%	5.29%	0.61%
Skill ramp	10B to 20B	56.47% -> 38.47%	28.52% -> 18.52%	9.11% -> 22.11%	5.29% -> 13.59%	0.61% -> 7.31%
Reasoning hold	20B to 36B	38.47%	18.52%	22.11%	13.59%	7.31%
Rebalance	36B to 46B	38.47% -> 48.47%	18.52% -> 20.52%	22.11% -> 16.11%	13.59% -> 10.09%	7.31% -> 4.81%
Quality cooldown	46B to 50B	48.47%	20.52%	16.11%	10.09%	4.81%

Training Setup

Field	Value
Sequence length	3,072
Micro batch	12
Gradient accumulation	15
Effective batch	180 sequences
Tokens per optimizer step	552,960
Planned optimizer steps	90,422
Actual final step	90,421
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	1.8e-3
Warmup steps	2,000
LR schedule	Warmup-stable-decay with cosine decay
Decay ratio	0.15
Weight decay	0.1, then 0.01 after 20B tokens
Gradient clipping	1.0
Z-loss coefficient	1e-4 until 20B tokens, then off
Compile	PyTorch compile enabled
Seed	1337

Benchmarks

Self-reported scores using lm_eval and the official ArithMark 2.0 script. Scores may vary slightly by evaluation setup.

Model	Average	HellaSwag	ARC Easy	ARC Challenge	PIQA	ArithMark 2.0
BananaMind-2-Medium	39.27	32.43	43.81	25.34	61.86	28.20
Supra 1.5 Base	39.52	29.78	48.40	25.51	60.01	31.32
Supra 50M Base	39.21	31.83	45.88	25.00	62.51	27.04
Veyra2-Apricot-50M-Base	38.81	31.28	42.47	23.29	62.13	28.96
BananaMind-2-Mini	37.67	29.80	39.06	25.34	59.41	29.28

ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none. Independent Open SLM Leaderboard evaluation is not yet included.

Repository Files

File	Description
config.json	Transformers config for bananamind2_medium
model.safetensors	Final exported model weights
tokenizer.json	Custom 12,288-token digit-aware tokenizer
tokenizer_config.json	Tokenizer metadata
generation_config.json	Default generation config with KV caching enabled
configuration_bananamind2medium.py	Custom Transformers config class
modeling_bananamind2medium.py	Custom Transformers model class
checkpoint_metadata.json	Final source checkpoint, step, and token metadata
banner.png	BananaMind 2 Medium model-card banner
benchmarks.png	Base-model benchmark comparison chart

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Suggested Generation Settings

For stable continuations:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=64 to 160

For more varied text:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• top_k=50
• repetition_penalty=1.1
• max_new_tokens=64 to 192

Intended Use

BananaMind-2-Medium is a base model intended for language-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, and small-model training comparisons.

Because this is a base model, prompts should be written as continuation prompts rather than chat messages.

License

Apache 2.0

Benchmark Average Formula

The benchmark average groups both ARC tasks into one component:

ARC average = (ARC Easy + ARC Challenge) / 2
Average = (HellaSwag + ARC average + PIQA + ArithMark 2.0) / available component count

Missing components are omitted. If only one ARC score is available, that score is used as the ARC component. An average is reported only when at least two components are available.