BananaMind-2.5-Pro

BananaMind-2.5-Pro is a 138,971,520-parameter decoder-only language model derived from BananaMind-2-Pro through distillation from SmolLM2-360M, combined with supervised training labels. It achieves as 28.98 INT Index

The original BananaMind-2-Pro was trained from scratch by BananaMind on 99,999,449,088 tokens over 184,954 optimizer steps. BananaMind-2.5-Pro starts from that original checkpoint.

The model retains the original 3,072-token context window, 32,768-token digit-aware byte-level BPE tokenizer, and architecture: grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache. Distillation changes the learned weights; it does not add layers, increase the parameter count, or replace the tokenizer.

This is a base continuation model, not an instruction-tuned chatbot. Use continuation-style prompts and load it with trust_remote_code=True.

Release Status

Field	Value
Release type	Distilled base-model checkpoint
Student base	BananaMind/BananaMind-2-Pro, original final checkpoint
Teacher	HuggingFaceTB/SmolLM2-360M, base version
Original pretraining checkpoint step	184,953
Original pretraining optimizer steps	184,954
Original pretraining tokens	99,999,449,088
Distillation optimizer steps	300
Sampled training examples processed	9,600
Distillation forward tokens processed	1,454,022
Training status	Complete
Evaluation status	Complete for ARC Easy, ARC Challenge, HellaSwag, PIQA, and ArithMark-3

Model Details

Field	Value
Parameters	138,971,520
Architecture	BananaMind2Pro decoder-only Transformer
Layers	24
Hidden size	640
Intermediate size	1,920
Attention heads	8
KV heads	4
Head dimension	80
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocabulary size	32,768
Context length	3,072
Embeddings	Tied input/output embeddings
Generation cache	KV cache supported
Weight format	safetensors
HF architecture	BananaMind2ProForCausalLM
HF model type	bananamind2_pro

Architecture Overview

The model maps tokens through a tied input embedding, 24 pre-normalized Transformer blocks, a final RMSNorm, and the tied language-model head. Each block contains causal grouped-query self-attention followed by a SwiGLU feed-forward network, with residual connections around both sublayers.

Eight query heads share four key/value heads, reducing KV-cache size while retaining multi-head query capacity. QK normalization stabilizes attention logits, and RoPE supplies position information without learned absolute-position embeddings. During autoregressive generation, each layer can reuse cached keys and values instead of recomputing the full prefix.

Evaluation

The BananaMind-2.5-Pro results below were measured on the exported distilled checkpoint using zero-shot evaluation on every example. ARC Easy, ARC Challenge, HellaSwag, and PIQA use the built-in tasks in lm-evaluation-harness 0.4.12, with acc_norm,none. ArithMark-3 uses the dataset repository's unmodified bencharithmark-3.py, with continuation likelihood normalized by token count.

Benchmark	BananaMind-2.5-Pro	BananaMind-2-Pro	GPT-X3-150M	GPT-X2.5-135M	BananaMind-2-Medium	GPT-2
Parameters	139M	139M	150M	135M	49.6M	124M
ARC Easy	62.79%	53.58%	59.64%	51.81%	43.81%	39.35%
ARC Challenge	31.96%	27.82%	31.31%	29.18%	25.34%	22.35%
PIQA	69.02%	67.52%	71.33%	69.42%	61.86%	62.08%
HellaSwag	44.60%	42.78%	46.16%	40.57%	32.43%	31.26%
ArithMark-3	38.60%	38.20%	50.90%	38.10%	36.20%	35.70%
ArithMark-2	31.1%	32.08%	68.52%	N/A	28.20%	26.48%
INT Index	28.92	24.96	33.05	25.17	15.37	N/A
Code Elo	1453	1407	N/A	1253	1034	996
Base Bench 1.1 Elo	1135	1124	N/A	1106	1034	996

ArithMark-3's published split is named train, but none of its examples were used for distillation training. The official script separately encodes each context and its leading-space continuation, uses a maximum context of 1,024 tokens, and reports both raw and normalized accuracy. This checkpoint scores 39.30% raw and 38.70% token-normalized; INT uses the latter.

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark-3. Combined ARC is calculated before normalization. ArithMark-2 is excluded:

N(score, chance) = 100 * (score - chance) / (100 - chance)
ARC = (ARC_Easy + ARC_Challenge) / 2
INT = [N(HellaSwag,25) + N(ARC,25) + N(PIQA,50) + 0.65*N(ArithMark3,25)] / 3.65

When a component is unavailable, its weight is removed from both numerator and denominator; at least two components are required. Scores below chance may be negative. All four components are available for this release.

Tokenizer

BananaMind-2.5-Pro retains the original custom 32,768-token byte-level BPE tokenizer trained on 75 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing complete numbers from being merged into large number tokens.

Token	ID
0	19
1	20
2	21
3	22
4	23
5	24
6	25
7	26
8	27
9	28

Special token IDs:

Token	ID
`<	pad
`<	bos
`<	eos
`<	unk

Training Data

BananaMind-2.5-Pro inherits BananaMind-2-Pro's pretraining and adds a separate distillation stage.

Original Pretraining Data

The inherited 100B-token pretraining curriculum combines educational web text, broad web text, synthetic textbook material, mathematics, and Python educational code.

Dataset	Target tokens	Aggregate share
FineWeb-Edu	50.166B	50.17%
DCLM	26.125B	26.13%
Cosmopedia-v2	13.525B	13.53%
FineMath-4+	7.875B	7.88%
NPSet-2 Python Edu	2.309B	2.31%
Total	100.000B	100.00%

Original pretraining used a capacity-aware curriculum:

Phase	Token range	Purpose
Breadth foundation	0B to 25B	Web-heavy language and knowledge foundation
Knowledge ramp	25B to 40B	Gradual increase in synthetic, mathematics, and code data
Reasoning core	40B to 75B	Sustained reasoning-oriented mixture
Synthesis ramp	75B to 90B	Transition toward the finishing distribution
Quality finish	90B to 100B	Final quality-focused mixture

Training Setup

Original Pretraining

Field	Value
Sequence length	3,072
Micro batch	4
Gradient accumulation	44
Effective batch	176 sequences
Tokens per optimizer step	540,672
Optimizer steps	184,954
Scheduled training tokens	99,999,449,088
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	1.5e-3
Warmup steps	2,000
LR schedule	Warmup-stable-decay with cosine decay
Decay ratio	0.15
Weight decay	0.1, then 0.01 after 40B tokens
Gradient clipping	1.0
Z-loss coefficient	1e-4 until 40B tokens, then off
Compile	PyTorch compile enabled
Seed	1337

The schedule used a 2,000-step linear warmup, a long stable learning-rate region, and a final cosine decay over the last 15% of training. Weight decay dropped from 0.1 to 0.01 after 40B tokens, and the auxiliary z-loss was disabled at the same boundary. The final optimizer update completed the planned 99,999,449,088-token schedule without rounding up to an extra sequence.

Distillation

The teacher and student have different tokenizers. Distillation therefore transfers probability distributions over textual answer choices, rather than attempting a direct match between vocabulary logits. The teacher assigns a character-normalized likelihood to each candidate answer; these scores are multiplied by 20 and converted to answer probabilities.

The student's target distribution is a 50/50 mixture of the teacher's answer probabilities and the correct training label. The loss is cross-entropy against that distribution, plus a 0.05-weighted correct-continuation token negative log-likelihood. Student answer scores also use the scale factor 20. All student parameters are trainable.

Field	Value
Student initialization	Original BananaMind-2-Pro final checkpoint
Teacher	HuggingFaceTB/SmolLM2-360M, base version
Teacher revision	f8027fd0eaeea54caa13c31d31b9fdc459c38b49
Optimizer	AdamW
Question micro batch	4
Gradient accumulation	8
Effective question batch	32
Peak learning rate	1e-5
Warmup	10 optimizer steps
LR schedule	Cosine decay to 20% of peak
Betas	0.9, 0.95
Weight decay	0.01
Gradient clipping	1.0
Parameter and optimizer precision	Float32
Training autocast	Bfloat16
Hardware	NVIDIA GeForce RTX 5070 Ti
Seed	20261105
Checkpoint selection	Highest reserved-validation INT, checked every 50 steps

Task sampling follows the coefficients in the chance-normalized INT objective, including PIQA's chance-normalization factor and half of the combined ARC coefficient for each ARC task. The saved checkpoint contains the resulting full weights, with tied embeddings preserved. No teacher or training-time loss is required for inference.

Energy and Carbon Estimate

These figures are inherited from the BananaMind-2-Pro model card and describe original pretraining only. They do not include teacher inference, distillation, or the additional evaluations for BananaMind-2.5-Pro. No new energy or carbon measurement was made for this release.

The following is an engineering estimate for the complete pretraining run, not a wall-meter measurement. Runtime is extrapolated from the observed 52,438-token/s run average. GPU and CPU package power were measured during training; motherboard, memory, storage, PSU loss, electricity use, and emissions are estimates.

Item	Basis	Value
Derived training time	99.999449B tokens / 52,438 tokens/s	529.72 hours (22.07 days)
GPU power	Measured 12-second nvidia-smi average at 99–100% utilization	262 W
CPU package power	Two Intel RAPL samples of 42 W and 38 W	40 W
Motherboard, memory, storage, and fans	Estimated combined power	50 W
DC system load	Estimated	352 W
PSU efficiency	Assumed	90%
Wall power	Estimated	391 W
Electricity use	Estimated	207.1 kWh
Austrian grid intensity used	Recent daily estimate	140 gCO2e/kWh
Complete-run emissions	Estimated	29.0 kg CO2e

Using the cited 2024 and 2025 Austrian flow-traced annual means of 125.5 and 169.4 gCO2e/kWh gives an estimated range of 26.0–35.1 kg CO2e. The grid figures come from Electricity Maps. This estimate excludes embodied hardware emissions, tokenizer training, evaluation, external networking, and remote storage.

Usage

Install the runtime dependencies:

pip install -U torch transformers safetensors

Run this example from the model directory, or set model_path to its location. It can also be a Hugging Face repository ID when the model is hosted there.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "."  # Directory containing BananaMind-2.5-Pro's config and weights.

tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Use do_sample=False for greedy generation. The benchmark scores above are measured by continuation likelihood, not by sampled or greedy text generation. Float32 was used for the published evaluation; changing dtype or batch shape can change individual predictions.

The internal architecture remains BananaMind2ProForCausalLM, with model type bananamind2_pro. These names are retained for checkpoint compatibility even though the release name is BananaMind-2.5-Pro.

Intended Use

BananaMind-2.5-Pro is intended for local text continuation, small-language-model research, distillation experiments, tokenizer research, arithmetic and commonsense evaluation, checkpoint analysis, and task-specific fine-tuning.

It is particularly suited to studying how a compact pretrained model adapts to teacher answer distributions and task training data while retaining its original architecture and tokenizer.

License

This repository is released under the BananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.

The distillation teacher, HuggingFaceTB/SmolLM2-360M, is published under Apache 2.0. BananaMind-2.5-Pro retains the BananaMind student architecture and checkpoint license; teacher attribution and the exact teacher revision are recorded above.