Bounty.Hunter.47m / README.md
Plans11's picture
Bounty.Hunter.47m | 20,000 new examples | step 625
2a874b7 verified
|
Raw History Blame Contribute Delete
1.88 kB
---
license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- chain-of-thought
---
# Bounty.Hunter.47m
A decoder-only causal language model trained from scratch, built around a
deep chain-of-thought reasoning core.
## Architecture
- Parameters: 46,912,320
- Context length: 1096
- Hidden size: 672
- Layers: 8
- Attention heads: 8
- Intermediate size: 2688
- Vocabulary size: 4,096
## Reasoning tokens
- `<|input|>`
- `<|think|>`
- `<|thought|>`
- `<|reasoning|>`
- `<|answer|>`
These are first-class tokens in the tokenizer and are woven into every
pretraining example's text structure, not added after the fact.
## Objective
True causal next-token prediction.
## Context
Every individual training example is strictly limited to
`1095` content tokens + EOS, padded to exactly `1096`
positions. No oversized example is split across separate training examples.
## Dataset
`Plans11/Organized_PreTrain_1k_Context`, restricted to:
- Think
- Thought
- Reasoning
(Chat / Instruct / Code_Instruct / Tool_Calling shards are intentionally
excluded from this model's pretraining corpus.)
## Example selection
New examples are selected each session via a deterministic seeded shuffle
of the restricted dataset, with SHA-256 content-hash deduplication against
every previously trained example. No example is ever trained twice.
## Resumability
Training state is persisted to Hugging Face, including:
- model.safetensors
- training_state.pt
- tokenizer files
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json
The tokenizer becomes immutable after its initial creation.
## Current progress
- Unique examples trained: 20,000
- Global optimizer steps: 625
- Last session: 20,000
This is an experimental language model and is not guaranteed to produce
factually or logically correct outputs.