Bounty.Hunter.47m / README.md
Plans11's picture
Bounty.Hunter.47m | 20,000 new examples | step 625
2a874b7 verified
|
Raw History Blame Contribute Delete
1.88 kB
metadata
license: mit
library_name: pytorch
tags:
  - causal-lm
  - decoder-only
  - reasoning
  - pretraining
  - think
  - thought
  - chain-of-thought

Bounty.Hunter.47m

A decoder-only causal language model trained from scratch, built around a deep chain-of-thought reasoning core.

Architecture

  • Parameters: 46,912,320
  • Context length: 1096
  • Hidden size: 672
  • Layers: 8
  • Attention heads: 8
  • Intermediate size: 2688
  • Vocabulary size: 4,096

Reasoning tokens

  • <|input|>
  • <|think|>
  • <|thought|>
  • <|reasoning|>
  • <|answer|>

These are first-class tokens in the tokenizer and are woven into every pretraining example's text structure, not added after the fact.

Objective

True causal next-token prediction.

Context

Every individual training example is strictly limited to 1095 content tokens + EOS, padded to exactly 1096 positions. No oversized example is split across separate training examples.

Dataset

Plans11/Organized_PreTrain_1k_Context, restricted to:

  • Think
  • Thought
  • Reasoning

(Chat / Instruct / Code_Instruct / Tool_Calling shards are intentionally excluded from this model's pretraining corpus.)

Example selection

New examples are selected each session via a deterministic seeded shuffle of the restricted dataset, with SHA-256 content-hash deduplication against every previously trained example. No example is ever trained twice.

Resumability

Training state is persisted to Hugging Face, including:

  • model.safetensors
  • training_state.pt
  • tokenizer files
  • config.json
  • progress.json
  • seen_examples.jsonl
  • training_metadata.json

The tokenizer becomes immutable after its initial creation.

Current progress

  • Unique examples trained: 20,000
  • Global optimizer steps: 625
  • Last session: 20,000

This is an experimental language model and is not guaranteed to produce factually or logically correct outputs.