README.md · marsnext/Phobos-1 at main

Phobos-1 / README.md

Add dark-mode architecture diagram and update README (README.md)

6749d1f verified 5 days ago

13.3 kB

---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
language:
  - en
tags:
  - marsnext
  - phobos-1
  - text-generation
  - conversational
  - code-generation
  - math-reasoning
  - system-2-reasoning
  - mixture-of-experts
  - cortical-moe
  - dynamic-mitosis
  - deepinfer-engram
  - multi-token-prediction
  - openai-compatible
  - edge-inference
  - safetensors
  - pytorch
---
# MarsNext `Phobos-1` (`96.26M` Total / `52.22M` Active Parameters)

**Organization:** [MarsNext (`@0xMarsNext`)](https://x.com/0xMarsNext) | **Repository:** [`marsnext/Phobos-1`](https://huggingface.co/marsnext/Phobos-1) | **License:** `Apache-2.0`

`Phobos-1` is a from-scratch **Hierarchical Cortical Mixture-of-Experts (MoE)** causal language model (`96,256,569` total parameters, `52,216,377` active parameters per token) featuring **2-Stage Spherical Cortical Routing (`Branch -> Specialist Leaf`)**, **Online SVD Spherical Mitosis**, **DeepInfer Engram Hash Memory (`2-gram` & `3-gram`)**, **Multi-Token Prediction (`MTP`)**, and **System-2 (`<think>...</think>`) Chain-of-Thought Reasoning**.

![MarsNext Phobos-1 Multi-Model Benchmark Card](assets/phobos1_benchmark_card.png)

---

## 1. Capabilities & Task Tags

| Capability Tag | Supported Tasks & Behavior |
| :--- | :--- |
| `text-generation` / `conversational` | Single-turn and multi-turn instruction following (`<|user|>` / `<|assistant|>` chat template) |
| `system-2-reasoning` (`<think>`) | Step-by-step scratchpad deduction inside `<think>...</think>` blocks prior to final answer generation |
| `code-generation` (`python`, `js`, `sql`) | Function synthesis, list/array manipulation, algorithmic walkthroughs, and bug analysis |
| `math-reasoning` | Symbolic linear algebra, step-by-step equation isolation, power-rule calculus, and arithmetic self-check |
| `science-qa` & `deductive-logic` | Foundational biology, physics kinematics, acid-base chemistry, and categorical syllogisms |
| `openai-compatible` | Built-in `/v1/chat/completions` and `/v1/models` HTTP server (`serve_openai_api.py`) with SSE streaming |
| `edge-inference` (`cuda` / `mps` / `cpu`) | `215.84 MiB` (`BF16`) footprint; runs offline on Apple Silicon (`MPS`), NVIDIA GPUs (`CUDA`), or CPU |

---

## 2. Architecture Specification

![MarsNext Phobos-1 Architecture Blueprint](assets/phobos1_architecture_diagram.png)

| Component | Technical Specification |
| :--- | :--- |
| **Model Class** | `PhobosScratchLM` (`architecture.py`) |
| **Total Parameters** | `96,256,569` (`215.84 MiB` in `BF16`) |
| **Active Parameters / Token** | `52,216,377` (`54.2%` active per token: `Shared E00 + Top-1 Branch + Top-2 Leaves`) |
| **Backbone Dimensions** | `8` Layers, `d_model = 512`, `max_seq_len = 2048`, `RMSNorm` ($\epsilon = 10^{-6}$) |
| **Attention Mechanism** | Causal Grouped-Query Attention (`GQA`: `8` Query Heads, `4` KV Heads, `head_dim = 64`), 1D `RoPE` + Short-Range Causal QK-Conv (`window = 8`) |
| **Cortical MoE Layer** | `shared_e00` (`1,024` SwiGLU neurons) + `4` Cortical Branches $\times$ **`16` Specialist Leaves/Layer** (`256` SwiGLU neurons each, grown from `8 -> 16` via `44` SVD Mitosis splits) |
| **Cortical Branch Domains** | `B0`: Language & Dialogue • `B1`: Code & Software Engineering • `B2`: Math & Formal Logic • `B3`: Natural Sciences |
| **Declarative Memory Bank** | `DeepInferEngramMemory` (`2-gram` & `3-gram` deterministic hash tables injected at layers `L1` & `L5`) |
| **Auxiliary Training Heads** | `DeepInferMTPHead` (`t+2` Multi-Token Prediction) + `SigmoidHCBlockMixin` residual highway gates |
| **Tokenizer** | Custom `32,768` (`2^15`) Byte-Level BPE (`phobos_bpe_32k.json` / `tokenizer.json`) |

---

## 3. Installation & Download

```bash
pip install torch tokenizers safetensors huggingface_hub openai
```

Download the repository via `huggingface_hub` or `git`:

```python
from huggingface_hub import snapshot_download
repo_dir = snapshot_download("marsnext/Phobos-1", local_dir="./Phobos-1")
```

```bash
git lfs install
git clone https://huggingface.co/marsnext/Phobos-1
cd Phobos-1
```

---

## 4. How to Run & Chat (`chat_cli.py`)
### A. Interactive Terminal Chat (Streaming)
```bash
python chat_cli.py
```
In-chat commands:
- `/think` — Toggle `<think>...</think>` System-2 reasoning mode ON/OFF
- `/quit` — Exit session
With explicit sampling parameters:
```bash
python chat_cli.py --think --temp 0.2 --top-p 0.9 --rep-penalty 1.15 --max-tokens 256
```
### B. One-Shot Command-Line Inference
```bash
# Standard direct response
python chat_cli.py --prompt "Write a Python function sum_even(nums) that returns the sum of all even integers in a list."
# Force System-2 <think> reasoning trace
python chat_cli.py --prompt "Solve 3x + 7 = 22 step by step." --think
```
---
## 5. OpenAI-Compatible API Server Setup (`serve_openai_api.py`)
`Phobos-1` ships with a self-contained OpenAI-compatible HTTP server (`serve_openai_api.py`) exposing `/v1/models`, `/v1/chat/completions`, `/v1/completions`, and `/health` (supporting both standard JSON responses and `stream=True` Server-Sent Events).
### Step 1: Start the Server
```bash
python serve_openai_api.py --host 0.0.0.0 --port 8000
```
### Step 2: Query via `curl`
```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "marsnext/Phobos-1",
    "messages": [
      {"role": "user", "content": "Solve 3x + 7 = 22 step by step."}
    ],
    "temperature": 0.0,
    "max_tokens": 200
  }'
```
### Step 3: Query via Official `openai` Python SDK (Standard & Streaming)

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="marsnext")
# 1. Non-Streaming Completion
response = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Write a Python function `sum_even(nums)` that returns the sum of all even integers in a list."}
    ],
    temperature=0.0,
    max_tokens=160,
)
print(response.choices[0].message.content)
# 2. Streaming Completion (SSE)
stream = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Why do muscle cells have a high density of mitochondria compared to skin cells?"}
    ],
    temperature=0.0,
    max_tokens=160,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
```

---

## 6. Direct PyTorch API Usage

```python
import sys
import torch
from huggingface_hub import snapshot_download
repo_dir = snapshot_download("marsnext/Phobos-1")
sys.path.insert(0, repo_dir)
from architecture import PhobosScratchConfig, PhobosScratchLM
from bpe_tokenizer import PhobosBPETokenizer
from mitosis import load_mitosis_state_dict
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
tok = PhobosBPETokenizer(f"{repo_dir}/phobos_bpe_32k.json")
ckpt = torch.load(f"{repo_dir}/phobos1_post_trained_bf16.pt", map_location="cpu", weights_only=False)
cfg = PhobosScratchConfig(vocab_size=tok.VOCAB_SIZE)
model = PhobosScratchLM(cfg)
load_mitosis_state_dict(
    model,
    {k: v.float() if torch.is_floating_point(v) else v for k, v in ckpt["model_state_dict"].items()},
)
model = model.to(device).eval()
prompt = "<|bos|><|user|>\nSolve 3x + 7 = 22 step by step.\n<|assistant|>\n"
ids = torch.tensor([tok.encode(prompt, max_len=512)], dtype=torch.long, device=device)
with torch.no_grad():
    for _ in range(160):
        logits = model(ids)["logits"][0, -1, :].float()
        next_id = int(torch.argmax(logits).item())
        if next_id == tok.EOS_ID:
            break
        ids = torch.cat([ids, torch.tensor([[next_id]], dtype=torch.long, device=device)], dim=1)
print(tok.decode(ids[0].tolist(), skip_special_tokens=False))
```

---

## 7. Training & Evaluation Telemetry

### A. Base Pre-Training Progression (`100.0%` of 35 Datasets — `33,977 / 33,977` Steps)

Trained on `3,627,127` complete sequences (`1,561,410,323` non-pad BPE tokens) across a 4-stage curriculum:

| Stage / Milestone | Step | Corpus Coverage | Validation CE (`val_ce`) | Validation Perplexity (`val_ppl`) | Cortical Branch Acc | Specialist Leaf Acc |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Stage 1 (Foundational Language)** | `2,500` | `15.4%` | `2.6410` | `14.03` | `78.4%` | `64.1%` |
| **Stage 2 (Code, Math & STEM)** | `10,000` | `48.2%` | `2.1840` | `8.88` | `73.2%` | `63.8%` |
| **Stage 3 (PhD Specialist Synthesis)** | `19,500` | `75.8%` | `1.9799` | `7.24` | `69.5%` | `63.2%` |
| **Stage 4 (Deep `<think>` & Tool Agents)** | `33,977` | **`100.0%`** | **`1.6793`** | **`5.36`** | **`75.5%`** | **`66.3%`** |

### B. Upper-Cortex (`L5–L7`) Post-Training (`60` Books + `50/50` Replay + `0.35 * KL` Anchor)

| Held-Out Evaluation Metric | Base Checkpoint (`Step 33,977`) | `Phobos-1` Final (Post-Trained) | Delta |
| :--- | :---: | :---: | :---: |
| **System-2 Reasoning Perplexity (`think_ppl`)** | `9.76` (`ce=2.2778`) | **`8.53`** (`ce=2.1433`) | **`-12.6%`** |
| **System-1 Factual Perplexity (`fact_ppl`)** | `2.32` (`ce=0.8404`) | **`1.98`** (`ce=0.6841`) | **`-14.7%`** |
| **Held-Out Cortical Branch Accuracy** | `63.77%` | **`64.74%`** | **`+0.97%`** |
| **Held-Out Specialist Leaf Accuracy** | `49.73%` | **`50.22%`** | **`+0.49%`** |

### C. Head-to-Head Benchmark vs. Open-Source Models (`50M – 165M` Parameter Class)

![MarsNext Phobos-1 Artificial Analysis Style Benchmark](assets/phobos1_benchmark_card.png)

Evaluated via `run_phobos1_benchmark_suite.py`, `run_multi_model_comparison.py`, and `generate_benchmark_charts.py` (`multi_model_head_to_head_benchmark.json`):
- **`MC-20` (`ARC-Easy / SciQ / PIQA / CS-Math`)**: 20 standardized 4-choice questions scored via tokenizer-independent length-normalized log-likelihood.
- **`Gen-25` (`25-Task Generative Suite`)**: Python `Pass@1` (`exec()` unit-test verification, 5 tasks), SQL Query Synthesis (5 tasks), STEM & Science QA (5 tasks), Symbolic Math & Definitions (5 tasks), and System-2 `<think>` Planning (5 tasks).

| Model | Org | Total Params | **Active Params / Token** | Training Tokens | `MC-20` (`ARC/SciQ/PIQA`) | **`Gen-25` Overall** | Python `Pass@1` (`exec()`) | SQL Synthesis | STEM QA | Math Symbolic | `<think>` Planning | **Combined Avg** |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **`marsnext/Phobos-1` (Final)** | **MarsNext** | **`96.26M`** | **`52.22M`** | **`1.56B`** | `35.0%` (`7/20`) | **`80.0%` (`20/25`)** | **`60%` (`3/5`)** | **`80%` (`4/5`)** | **`80%` (`4/5`)** | **`80%` (`4/5`)** | **`100%` (`5/5`)** | **`57.5%`** |
| **`marsnext/Phobos-1` (Base)** | **MarsNext** | `96.26M` | `52.22M` | `1.56B` | `35.0%` (`7/20`) | `68.0%` (`17/25`) | **`60%` (`3/5`)** | **`80%` (`4/5`)** | `60%` (`3/5`) | `40%` (`2/5`) | **`100%` (`5/5`)** | `51.5%` |
| `HuggingFaceTB/SmolLM2-135M` | HuggingFace | `134.52M` | `134.52M` (`2.6x`) | `2,000B` | **`60.0%` (`12/20`)** | `36.0%` (`9/25`) | `0%` (`0/5`) | `40%` (`2/5`) | **`80%` (`4/5`)** | `20%` (`1/5`) | `40%` (`2/5`) | `48.0%` |
| `openai-community/gpt2` | OpenAI | `124.44M` | `124.44M` (`2.4x`) | `~40B` | `40.0%` (`8/20`) | `20.0%` (`5/25`) | `0%` (`0/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `40%` (`2/5`) | `30.0%` |
| `EleutherAI/pythia-70m-deduped` | EleutherAI | `70.43M` | `70.43M` (`1.35x`) | `300B` | `30.0%` (`6/20`) | `12.0%` (`3/25`) | `0%` (`0/5`) | `0%` (`0/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `21.0%` |
| `EleutherAI/pythia-160m-deduped` | EleutherAI | `162.32M` | `162.32M` (`3.1x`) | `300B` | `25.0%` (`5/20`) | `12.0%` (`3/25`) | `0%` (`0/5`) | `0%` (`0/5`) | `20%` (`1/5`) | `0%` (`0/5`) | `40%` (`2/5`) | `18.5%` |

---

## 8. Repository Files

- `model.safetensors`: `BF16` SafeTensors weights (`96,256,569` parameters).
- `phobos1_post_trained_bf16.pt`: Post-trained PyTorch checkpoint (60-Book Pedagogy + `50/50` Replay + `KL` Anchor).
- `phobos1_base_100pct_step33977_bf16.pt`: Untouched `100.0%` Full-Corpus Base Pre-Trained checkpoint (`Step 33,977 / 33,977`).
- `config.json` & `phobos_bpe_32k.json` (`tokenizer.json`): Model configuration and `32,768` Byte-Level BPE tokenizer.
- `chat_cli.py`: Interactive terminal chat & single-prompt inference CLI.
- `serve_openai_api.py`: Zero-dependency OpenAI-compatible HTTP server (`/v1/chat/completions`, `/v1/models`).
- `architecture.py`, `deepinfer_modules.py`, `mitosis.py`, `bpe_tokenizer.py`: Self-contained PyTorch architecture modules.
- `training_telemetry_33977steps.json` & `benchmark_10prompts_comparison.json`: Complete training curve, `44` mitosis events, and side-by-side evaluation logs.

---

## 9. Technical Limitations

- `Phobos-1` is an ultra-compact (`96.26M` total / `52.22M` active parameters) model designed for research and resource-constrained edge inference. While it handles foundational Python functions, single-variable linear equations, basic biology/chemistry, and simple deductions reliably, multi-hop combinatorics, adversarial linguistic word-traps, and multi-step arithmetic divisions can produce calculation slips without external tool verification.
- Maximum context window is `2,048` tokens.

	---
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-generation
	language:
	- en
	tags:
	- marsnext
	- phobos-1
	- text-generation
	- conversational
	- code-generation
	- math-reasoning
	- system-2-reasoning
	- mixture-of-experts
	- cortical-moe
	- dynamic-mitosis
	- deepinfer-engram
	- multi-token-prediction
	- openai-compatible
	- edge-inference
	- safetensors
	- pytorch
	---

	# MarsNext `Phobos-1` (`96.26M` Total / `52.22M` Active Parameters)

	Organization: [MarsNext (`@0xMarsNext`)](https://x.com/0xMarsNext) \| Repository: [`marsnext/Phobos-1`](https://huggingface.co/marsnext/Phobos-1) \| License: `Apache-2.0`

	`Phobos-1` is a from-scratch Hierarchical Cortical Mixture-of-Experts (MoE) causal language model (`96,256,569` total parameters, `52,216,377` active parameters per token) featuring 2-Stage Spherical Cortical Routing (`Branch -> Specialist Leaf`), Online SVD Spherical Mitosis, DeepInfer Engram Hash Memory (`2-gram` & `3-gram`), Multi-Token Prediction (`MTP`), and System-2 (`<think>...</think>`) Chain-of-Thought Reasoning.

	![MarsNext Phobos-1 Multi-Model Benchmark Card](assets/phobos1_benchmark_card.png)

	---

	## 1. Capabilities & Task Tags

	\| Capability Tag \| Supported Tasks & Behavior \|
	\| :--- \| :--- \|
	\| `text-generation` / `conversational` \| Single-turn and multi-turn instruction following (`<\|user\|>` / `<\|assistant\|>` chat template) \|
	\| `system-2-reasoning` (`<think>`) \| Step-by-step scratchpad deduction inside `<think>...</think>` blocks prior to final answer generation \|
	\| `code-generation` (`python`, `js`, `sql`) \| Function synthesis, list/array manipulation, algorithmic walkthroughs, and bug analysis \|
	\| `math-reasoning` \| Symbolic linear algebra, step-by-step equation isolation, power-rule calculus, and arithmetic self-check \|
	\| `science-qa` & `deductive-logic` \| Foundational biology, physics kinematics, acid-base chemistry, and categorical syllogisms \|
	\| `openai-compatible` \| Built-in `/v1/chat/completions` and `/v1/models` HTTP server (`serve_openai_api.py`) with SSE streaming \|
	\| `edge-inference` (`cuda` / `mps` / `cpu`) \| `215.84 MiB` (`BF16`) footprint; runs offline on Apple Silicon (`MPS`), NVIDIA GPUs (`CUDA`), or CPU \|

	---

	## 2. Architecture Specification

	![MarsNext Phobos-1 Architecture Blueprint](assets/phobos1_architecture_diagram.png)

	\| Component \| Technical Specification \|
	\| :--- \| :--- \|
	\| Model Class \| `PhobosScratchLM` (`architecture.py`) \|
	\| Total Parameters \| `96,256,569` (`215.84 MiB` in `BF16`) \|
	\| Active Parameters / Token \| `52,216,377` (`54.2%` active per token: `Shared E00 + Top-1 Branch + Top-2 Leaves`) \|
	\| Backbone Dimensions \| `8` Layers, `d_model = 512`, `max_seq_len = 2048`, `RMSNorm` ($\epsilon = 10^{-6}$) \|
	\| Attention Mechanism \| Causal Grouped-Query Attention (`GQA`: `8` Query Heads, `4` KV Heads, `head_dim = 64`), 1D `RoPE` + Short-Range Causal QK-Conv (`window = 8`) \|
	\| Cortical MoE Layer \| `shared_e00` (`1,024` SwiGLU neurons) + `4` Cortical Branches $\times$ `16` Specialist Leaves/Layer (`256` SwiGLU neurons each, grown from `8 -> 16` via `44` SVD Mitosis splits) \|
	\| Cortical Branch Domains \| `B0`: Language & Dialogue • `B1`: Code & Software Engineering • `B2`: Math & Formal Logic • `B3`: Natural Sciences \|
	\| Declarative Memory Bank \| `DeepInferEngramMemory` (`2-gram` & `3-gram` deterministic hash tables injected at layers `L1` & `L5`) \|
	\| Auxiliary Training Heads \| `DeepInferMTPHead` (`t+2` Multi-Token Prediction) + `SigmoidHCBlockMixin` residual highway gates \|
	\| Tokenizer \| Custom `32,768` (`2^15`) Byte-Level BPE (`phobos_bpe_32k.json` / `tokenizer.json`) \|

	---

	## 3. Installation & Download

	```bash
	pip install torch tokenizers safetensors huggingface_hub openai
	```

	Download the repository via `huggingface_hub` or `git`:

	```python
	from huggingface_hub import snapshot_download
	repo_dir = snapshot_download("marsnext/Phobos-1", local_dir="./Phobos-1")
	```

	```bash
	git lfs install
	git clone https://huggingface.co/marsnext/Phobos-1
	cd Phobos-1
	```

	---

	## 4. How to Run & Chat (`chat_cli.py`)

	### A. Interactive Terminal Chat (Streaming)

	```bash
	python chat_cli.py
	```

	In-chat commands:
	- `/think` — Toggle `<think>...</think>` System-2 reasoning mode ON/OFF
	- `/quit` — Exit session

	With explicit sampling parameters:

	```bash
	python chat_cli.py --think --temp 0.2 --top-p 0.9 --rep-penalty 1.15 --max-tokens 256
	```

	### B. One-Shot Command-Line Inference

	```bash
	# Standard direct response
	python chat_cli.py --prompt "Write a Python function sum_even(nums) that returns the sum of all even integers in a list."

	# Force System-2 <think> reasoning trace
	python chat_cli.py --prompt "Solve 3x + 7 = 22 step by step." --think
	```

	---

	## 5. OpenAI-Compatible API Server Setup (`serve_openai_api.py`)

	`Phobos-1` ships with a self-contained OpenAI-compatible HTTP server (`serve_openai_api.py`) exposing `/v1/models`, `/v1/chat/completions`, `/v1/completions`, and `/health` (supporting both standard JSON responses and `stream=True` Server-Sent Events).

	### Step 1: Start the Server

	```bash
	python serve_openai_api.py --host 0.0.0.0 --port 8000
	```

	### Step 2: Query via `curl`

	```bash
	curl http://localhost:8000/v1/chat/completions \
	-H "Content-Type: application/json" \
	-d '{
	"model": "marsnext/Phobos-1",
	"messages": [
	{"role": "user", "content": "Solve 3x + 7 = 22 step by step."}
	],
	"temperature": 0.0,
	"max_tokens": 200
	}'
	```

	### Step 3: Query via Official `openai` Python SDK (Standard & Streaming)

	```python
	from openai import OpenAI

	client = OpenAI(base_url="http://localhost:8000/v1", api_key="marsnext")

	# 1. Non-Streaming Completion
	response = client.chat.completions.create(
	model="marsnext/Phobos-1",
	messages=[
	{"role": "user", "content": "Write a Python function `sum_even(nums)` that returns the sum of all even integers in a list."}
	],
	temperature=0.0,
	max_tokens=160,
	)
	print(response.choices[0].message.content)

	# 2. Streaming Completion (SSE)
	stream = client.chat.completions.create(
	model="marsnext/Phobos-1",
	messages=[
	{"role": "user", "content": "Why do muscle cells have a high density of mitochondria compared to skin cells?"}
	],
	temperature=0.0,
	max_tokens=160,
	stream=True,
	)
	for chunk in stream:
	delta = chunk.choices[0].delta.content
	if delta:
	print(delta, end="", flush=True)
	```

	---

	## 6. Direct PyTorch API Usage

	```python
	import sys
	import torch
	from huggingface_hub import snapshot_download

	repo_dir = snapshot_download("marsnext/Phobos-1")
	sys.path.insert(0, repo_dir)

	from architecture import PhobosScratchConfig, PhobosScratchLM
	from bpe_tokenizer import PhobosBPETokenizer
	from mitosis import load_mitosis_state_dict

	device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
	tok = PhobosBPETokenizer(f"{repo_dir}/phobos_bpe_32k.json")
	ckpt = torch.load(f"{repo_dir}/phobos1_post_trained_bf16.pt", map_location="cpu", weights_only=False)

	cfg = PhobosScratchConfig(vocab_size=tok.VOCAB_SIZE)
	model = PhobosScratchLM(cfg)
	load_mitosis_state_dict(
	model,
	{k: v.float() if torch.is_floating_point(v) else v for k, v in ckpt["model_state_dict"].items()},
	)
	model = model.to(device).eval()

	prompt = "<\|bos\|><\|user\|>\nSolve 3x + 7 = 22 step by step.\n<\|assistant\|>\n"
	ids = torch.tensor([tok.encode(prompt, max_len=512)], dtype=torch.long, device=device)

	with torch.no_grad():
	for _ in range(160):
	logits = model(ids)["logits"][0, -1, :].float()
	next_id = int(torch.argmax(logits).item())
	if next_id == tok.EOS_ID:
	break
	ids = torch.cat([ids, torch.tensor([[next_id]], dtype=torch.long, device=device)], dim=1)

	print(tok.decode(ids[0].tolist(), skip_special_tokens=False))
	```

	---

	## 7. Training & Evaluation Telemetry

	### A. Base Pre-Training Progression (`100.0%` of 35 Datasets — `33,977 / 33,977` Steps)

	Trained on `3,627,127` complete sequences (`1,561,410,323` non-pad BPE tokens) across a 4-stage curriculum:

	\| Stage / Milestone \| Step \| Corpus Coverage \| Validation CE (`val_ce`) \| Validation Perplexity (`val_ppl`) \| Cortical Branch Acc \| Specialist Leaf Acc \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \|
	\| Stage 1 (Foundational Language) \| `2,500` \| `15.4%` \| `2.6410` \| `14.03` \| `78.4%` \| `64.1%` \|
	\| Stage 2 (Code, Math & STEM) \| `10,000` \| `48.2%` \| `2.1840` \| `8.88` \| `73.2%` \| `63.8%` \|
	\| Stage 3 (PhD Specialist Synthesis) \| `19,500` \| `75.8%` \| `1.9799` \| `7.24` \| `69.5%` \| `63.2%` \|
	\| Stage 4 (Deep `<think>` & Tool Agents) \| `33,977` \| `100.0%` \| `1.6793` \| `5.36` \| `75.5%` \| `66.3%` \|

	### B. Upper-Cortex (`L5–L7`) Post-Training (`60` Books + `50/50` Replay + `0.35 * KL` Anchor)

	\| Held-Out Evaluation Metric \| Base Checkpoint (`Step 33,977`) \| `Phobos-1` Final (Post-Trained) \| Delta \|
	\| :--- \| :---: \| :---: \| :---: \|
	\| System-2 Reasoning Perplexity (`think_ppl`) \| `9.76` (`ce=2.2778`) \| `8.53` (`ce=2.1433`) \| `-12.6%` \|
	\| System-1 Factual Perplexity (`fact_ppl`) \| `2.32` (`ce=0.8404`) \| `1.98` (`ce=0.6841`) \| `-14.7%` \|
	\| Held-Out Cortical Branch Accuracy \| `63.77%` \| `64.74%` \| `+0.97%` \|
	\| Held-Out Specialist Leaf Accuracy \| `49.73%` \| `50.22%` \| `+0.49%` \|

	### C. Head-to-Head Benchmark vs. Open-Source Models (`50M – 165M` Parameter Class)

	![MarsNext Phobos-1 Artificial Analysis Style Benchmark](assets/phobos1_benchmark_card.png)

	Evaluated via `run_phobos1_benchmark_suite.py`, `run_multi_model_comparison.py`, and `generate_benchmark_charts.py` (`multi_model_head_to_head_benchmark.json`):
	- `MC-20` (`ARC-Easy / SciQ / PIQA / CS-Math`): 20 standardized 4-choice questions scored via tokenizer-independent length-normalized log-likelihood.
	- `Gen-25` (`25-Task Generative Suite`): Python `Pass@1` (`exec()` unit-test verification, 5 tasks), SQL Query Synthesis (5 tasks), STEM & Science QA (5 tasks), Symbolic Math & Definitions (5 tasks), and System-2 `<think>` Planning (5 tasks).

	\| Model \| Org \| Total Params \| Active Params / Token \| Training Tokens \| `MC-20` (`ARC/SciQ/PIQA`) \| `Gen-25` Overall \| Python `Pass@1` (`exec()`) \| SQL Synthesis \| STEM QA \| Math Symbolic \| `<think>` Planning \| Combined Avg \|
	\| :--- \| :--- \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \|
	\| `marsnext/Phobos-1` (Final) \| MarsNext \| `96.26M` \| `52.22M` \| `1.56B` \| `35.0%` (`7/20`) \| `80.0%` (`20/25`) \| `60%` (`3/5`) \| `80%` (`4/5`) \| `80%` (`4/5`) \| `80%` (`4/5`) \| `100%` (`5/5`) \| `57.5%` \|
	\| `marsnext/Phobos-1` (Base) \| MarsNext \| `96.26M` \| `52.22M` \| `1.56B` \| `35.0%` (`7/20`) \| `68.0%` (`17/25`) \| `60%` (`3/5`) \| `80%` (`4/5`) \| `60%` (`3/5`) \| `40%` (`2/5`) \| `100%` (`5/5`) \| `51.5%` \|
	\| `HuggingFaceTB/SmolLM2-135M` \| HuggingFace \| `134.52M` \| `134.52M` (`2.6x`) \| `2,000B` \| `60.0%` (`12/20`) \| `36.0%` (`9/25`) \| `0%` (`0/5`) \| `40%` (`2/5`) \| `80%` (`4/5`) \| `20%` (`1/5`) \| `40%` (`2/5`) \| `48.0%` \|
	\| `openai-community/gpt2` \| OpenAI \| `124.44M` \| `124.44M` (`2.4x`) \| `~40B` \| `40.0%` (`8/20`) \| `20.0%` (`5/25`) \| `0%` (`0/5`) \| `20%` (`1/5`) \| `20%` (`1/5`) \| `20%` (`1/5`) \| `40%` (`2/5`) \| `30.0%` \|
	\| `EleutherAI/pythia-70m-deduped` \| EleutherAI \| `70.43M` \| `70.43M` (`1.35x`) \| `300B` \| `30.0%` (`6/20`) \| `12.0%` (`3/25`) \| `0%` (`0/5`) \| `0%` (`0/5`) \| `20%` (`1/5`) \| `20%` (`1/5`) \| `20%` (`1/5`) \| `21.0%` \|
	\| `EleutherAI/pythia-160m-deduped` \| EleutherAI \| `162.32M` \| `162.32M` (`3.1x`) \| `300B` \| `25.0%` (`5/20`) \| `12.0%` (`3/25`) \| `0%` (`0/5`) \| `0%` (`0/5`) \| `20%` (`1/5`) \| `0%` (`0/5`) \| `40%` (`2/5`) \| `18.5%` \|

	---

	## 8. Repository Files

	- `model.safetensors`: `BF16` SafeTensors weights (`96,256,569` parameters).
	- `phobos1_post_trained_bf16.pt`: Post-trained PyTorch checkpoint (60-Book Pedagogy + `50/50` Replay + `KL` Anchor).
	- `phobos1_base_100pct_step33977_bf16.pt`: Untouched `100.0%` Full-Corpus Base Pre-Trained checkpoint (`Step 33,977 / 33,977`).
	- `config.json` & `phobos_bpe_32k.json` (`tokenizer.json`): Model configuration and `32,768` Byte-Level BPE tokenizer.
	- `chat_cli.py`: Interactive terminal chat & single-prompt inference CLI.
	- `serve_openai_api.py`: Zero-dependency OpenAI-compatible HTTP server (`/v1/chat/completions`, `/v1/models`).
	- `architecture.py`, `deepinfer_modules.py`, `mitosis.py`, `bpe_tokenizer.py`: Self-contained PyTorch architecture modules.
	- `training_telemetry_33977steps.json` & `benchmark_10prompts_comparison.json`: Complete training curve, `44` mitosis events, and side-by-side evaluation logs.

	---

	## 9. Technical Limitations

	- `Phobos-1` is an ultra-compact (`96.26M` total / `52.22M` active parameters) model designed for research and resource-constrained edge inference. While it handles foundational Python functions, single-variable linear equations, basic biology/chemistry, and simple deductions reliably, multi-hop combinatorics, adversarial linguistic word-traps, and multi-step arithmetic divisions can produce calculation slips without external tool verification.
	- Maximum context window is `2,048` tokens.