Fleck-S-100K-Base

Fleck-S-100K-Base — a compact decoder-only language model trained from scratch on 100M real tokenizer tokens.

• 109,384 parameters
• BF16 SafeTensors weights
• Factorized tied embedding with a rank-32 embedding projection
• Designed as a small research model for local inference on Apple Silicon

Model Details

Architecture	Decoder-only Transformer
Parameters	109,384
Hidden size	64
FFN size	128
Physical blocks	2
Effective depth	4 (A → B → A → B)
Attention	GQA — 4 query heads, 2 KV heads, head dimension 16
Normalization	RMSNorm
Embedding	Factorized tied embedding, rank 32
Vocabulary	1,024
Context length	2,048 tokens
Canonical dtype	BF16

Training

Initialization	Fresh initialization
Dataset	FineWeb-Edu / FineWeb
Data mixture	70% FineWeb-Edu + 30% FineWeb
Real tokenizer tokens	100,000,000 exactly
Instruction tuning	No
Hardware	Apple M2 (10-core GPU)

This is the immutable Base-100M parent used to produce the independent Fleck-S-100K instruction-tuned model.

Benchmark Results

The Base-100M checkpoint was evaluated with the corrected zero-shot aggregation protocol. This minimal public bundle does not include the evaluation artifact; the table below is a reference result for the released Base model.

Evaluation conditions: zero-shot, no chat template, FP32 evaluation, Apple Silicon MPS.

Task	Metric	Base
HellaSwag	acc_norm	26.28%
PIQA	acc_norm	51.14%
ARC-Easy	acc_norm	26.05%
ARC-Challenge	acc_norm	26.19%
LAMBADA OpenAI	acc	0.02%
WinoGrande	acc	49.41%
BoolQ	acc	37.83%
MMLU (57-subject macro)	acc	23.09%
Eight-task mean	—	30.00%

Tokenizer

• Fleck-Tokenizer-1024
• Byte-level BPE
• Vocabulary size: 1,024

Special Tokens

Token	ID	Role
<bos>	0	sequence start
<eos>	1	sequence end
<pad>	2	padding
<unk>	3	unknown token
<|system|>	4	system turn
<|user|>	5	user turn
<|assistant|>	6	assistant turn
<|eot|>	7	end of turn

Usage

The bundle includes a self-contained inference.py; it does not import the Fleck-LM checkout. The accompanying config.json, generation_config.json, and tokenizer_config.json describe the custom architecture and generation/tokenizer defaults; standard transformers.AutoModel loading is not supported. Install the three runtime dependencies and run a single greedy continuation:

python -m pip install torch safetensors tokenizers
python inference.py \
  --ckpt model.safetensors \
  --tokenizer tokenizer.json \
  --prompt "Hello, world" \
  --max-tokens 32 \
  --device cpu

The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. The model implements its factorized tied embedding/logits, effective-depth execution A → B → A → B, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without any repository-local imports.

The default filenames are model.safetensors and tokenizer.json, so the shorter command also works:

python inference.py --prompt "Hello" --max-tokens 16 --device cpu

For tokenizer-only use:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
print(tokenizer.encode("Hello, world!", add_special_tokens=False).ids)

Limitations

This model is extremely small and is intended for research and local experimentation rather than general-purpose language generation. It may produce repetitions, malformed text, weak factual answers, or incoherent continuations. Benchmark scores should be interpreted in the context of the 109K parameter count.

License

MIT License.

Files

The public bundle contains these files:

• README.md — model card and usage documentation
• inference.py — standalone strict loader and greedy inference CLI
• model.safetensors — BF16 model weights
• tokenizer.json — standalone tokenizer
• config.json — custom architecture configuration
• generation_config.json — greedy generation defaults
• tokenizer_config.json — tokenizer defaults and special-token mapping No training data, optimizer state, or other training outputs are included.