---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
language:
  - en
tags:
  - marsnext
  - phobos-1
  - text-generation
  - conversational
  - code-generation
  - math-reasoning
  - system-2-reasoning
  - mixture-of-experts
  - cortical-moe
  - dynamic-mitosis
  - deepinfer-engram
  - multi-token-prediction
  - openai-compatible
  - edge-inference
  - safetensors
  - pytorch
---

# MarsNext `Phobos-1` (`96.26M` Total / `52.22M` Active Parameters)

**Organization:** [MarsNext (`@0xMarsNext`)](https://x.com/0xMarsNext) | **Repository:** [`marsnext/Phobos-1`](https://huggingface.co/marsnext/Phobos-1) | **License:** `Apache-2.0`

`Phobos-1` is a from-scratch **Hierarchical Cortical Mixture-of-Experts (MoE)** causal language model (`96,256,569` total parameters, `52,216,377` active parameters per token) featuring **2-Stage Spherical Cortical Routing (`Branch -> Specialist Leaf`)**, **Online SVD Spherical Mitosis**, **DeepInfer Engram Hash Memory (`2-gram` & `3-gram`)**, **Multi-Token Prediction (`MTP`)**, and **System-2 (`<think>...</think>`) Chain-of-Thought Reasoning**.

![MarsNext Phobos-1 Multi-Model Benchmark Card](assets/phobos1_benchmark_card.png)

---

## 1. Capabilities & Task Tags

| Capability Tag | Supported Tasks & Behavior |
| :--- | :--- |
| `text-generation` / `conversational` | Single-turn and multi-turn instruction following (`<|user|>` / `<|assistant|>` chat template) |
| `system-2-reasoning` (`<think>`) | Step-by-step scratchpad deduction inside `<think>...</think>` blocks prior to final answer generation |
| `code-generation` (`python`, `js`, `sql`) | Function synthesis, list/array manipulation, algorithmic walkthroughs, and bug analysis |
| `math-reasoning` | Symbolic linear algebra, step-by-step equation isolation, power-rule calculus, and arithmetic self-check |
| `science-qa` & `deductive-logic` | Foundational biology, physics kinematics, acid-base chemistry, and categorical syllogisms |
| `openai-compatible` | Built-in `/v1/chat/completions` and `/v1/models` HTTP server (`serve_openai_api.py`) with SSE streaming |
| `edge-inference` (`cuda` / `mps` / `cpu`) | `215.84 MiB` (`BF16`) footprint; runs offline on Apple Silicon (`MPS`), NVIDIA GPUs (`CUDA`), or CPU |

---

## 2. Architecture Specification

![MarsNext Phobos-1 Architecture Blueprint](assets/phobos1_architecture_diagram.png)

| Component | Technical Specification |
| :--- | :--- |
| **Model Class** | `PhobosScratchLM` (`architecture.py`) |
| **Total Parameters** | `96,256,569` (`215.84 MiB` in `BF16`) |
| **Active Parameters / Token** | `52,216,377` (`54.2%` active per token: `Shared E00 + Top-1 Branch + Top-2 Leaves`) |
| **Backbone Dimensions** | `8` Layers, `d_model = 512`, `max_seq_len = 2048`, `RMSNorm` ($\epsilon = 10^{-6}$) |
| **Attention Mechanism** | Causal Grouped-Query Attention (`GQA`: `8` Query Heads, `4` KV Heads, `head_dim = 64`), 1D `RoPE` + Short-Range Causal QK-Conv (`window = 8`) |
| **Cortical MoE Layer** | `shared_e00` (`1,024` SwiGLU neurons) + `4` Cortical Branches $\times$ **`16` Specialist Leaves/Layer** (`256` SwiGLU neurons each, grown from `8 -> 16` via `44` SVD Mitosis splits) |
| **Cortical Branch Domains** | `B0`: Language & Dialogue • `B1`: Code & Software Engineering • `B2`: Math & Formal Logic • `B3`: Natural Sciences |
| **Declarative Memory Bank** | `DeepInferEngramMemory` (`2-gram` & `3-gram` deterministic hash tables injected at layers `L1` & `L5`) |
| **Auxiliary Training Heads** | `DeepInferMTPHead` (`t+2` Multi-Token Prediction) + `SigmoidHCBlockMixin` residual highway gates |
| **Tokenizer** | Custom `32,768` (`2^15`) Byte-Level BPE (`phobos_bpe_32k.json` / `tokenizer.json`) |

---

## 3. Installation & Download

```bash
pip install torch tokenizers safetensors huggingface_hub openai
```

Download the repository via `huggingface_hub` or `git`:

```python
from huggingface_hub import snapshot_download
repo_dir = snapshot_download("marsnext/Phobos-1", local_dir="./Phobos-1")
```

```bash
git lfs install
git clone https://huggingface.co/marsnext/Phobos-1
cd Phobos-1
```

---

## 4. How to Run & Chat (`chat_cli.py`)

### A. Interactive Terminal Chat (Streaming)

```bash
python chat_cli.py
```

In-chat commands:
- `/think` — Toggle `<think>...</think>` System-2 reasoning mode ON/OFF
- `/quit` — Exit session

With explicit sampling parameters:

```bash
python chat_cli.py --think --temp 0.2 --top-p 0.9 --rep-penalty 1.15 --max-tokens 256
```

### B. One-Shot Command-Line Inference

```bash
# Standard direct response
python chat_cli.py --prompt "Write a Python function sum_even(nums) that returns the sum of all even integers in a list."

# Force System-2 <think> reasoning trace
python chat_cli.py --prompt "Solve 3x + 7 = 22 step by step." --think
```

---

## 5. OpenAI-Compatible API Server Setup (`serve_openai_api.py`)

`Phobos-1` ships with a self-contained OpenAI-compatible HTTP server (`serve_openai_api.py`) exposing `/v1/models`, `/v1/chat/completions`, `/v1/completions`, and `/health` (supporting both standard JSON responses and `stream=True` Server-Sent Events).

### Step 1: Start the Server

```bash
python serve_openai_api.py --host 0.0.0.0 --port 8000
```

### Step 2: Query via `curl`

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "marsnext/Phobos-1",
    "messages": [
      {"role": "user", "content": "Solve 3x + 7 = 22 step by step."}
    ],
    "temperature": 0.0,
    "max_tokens": 200
  }'
```

### Step 3: Query via Official `openai` Python SDK (Standard & Streaming)

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="marsnext")

# 1. Non-Streaming Completion
response = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Write a Python function `sum_even(nums)` that returns the sum of all even integers in a list."}
    ],
    temperature=0.0,
    max_tokens=160,
)
print(response.choices[0].message.content)

# 2. Streaming Completion (SSE)
stream = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Why do muscle cells have a high density of mitochondria compared to skin cells?"}
    ],
    temperature=0.0,
    max_tokens=160,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
```

---

## 6. Direct PyTorch API Usage

```python
import sys
import torch
from huggingface_hub import snapshot_download

repo_dir = snapshot_download("marsnext/Phobos-1")
sys.path.insert(0, repo_dir)

from architecture import PhobosScratchConfig, PhobosScratchLM
from bpe_tokenizer import PhobosBPETokenizer
from mitosis import load_mitosis_state_dict

device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
tok = PhobosBPETokenizer(f"{repo_dir}/phobos_bpe_32k.json")
ckpt = torch.load(f"{repo_dir}/phobos1_post_trained_bf16.pt", map_location="cpu", weights_only=False)

cfg = PhobosScratchConfig(vocab_size=tok.VOCAB_SIZE)
model = PhobosScratchLM(cfg)
load_mitosis_state_dict(
    model,
    {k: v.float() if torch.is_floating_point(v) else v for k, v in ckpt["model_state_dict"].items()},
)
model = model.to(device).eval()

prompt = "<|bos|><|user|>\nSolve 3x + 7 = 22 step by step.\n<|assistant|>\n"
ids = torch.tensor([tok.encode(prompt, max_len=512)], dtype=torch.long, device=device)

with torch.no_grad():
    for _ in range(160):
        logits = model(ids)["logits"][0, -1, :].float()
        next_id = int(torch.argmax(logits).item())
        if next_id == tok.EOS_ID:
            break
        ids = torch.cat([ids, torch.tensor([[next_id]], dtype=torch.long, device=device)], dim=1)

print(tok.decode(ids[0].tolist(), skip_special_tokens=False))
```

---

## 7. Training & Evaluation Telemetry

### A. Base Pre-Training Progression (`100.0%` of 35 Datasets — `33,977 / 33,977` Steps)

Trained on `3,627,127` complete sequences (`1,561,410,323` non-pad BPE tokens) across a 4-stage curriculum:

| Stage / Milestone | Step | Corpus Coverage | Validation CE (`val_ce`) | Validation Perplexity (`val_ppl`) | Cortical Branch Acc | Specialist Leaf Acc |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Stage 1 (Foundational Language)** | `2,500` | `15.4%` | `2.6410` | `14.03` | `78.4%` | `64.1%` |
| **Stage 2 (Code, Math & STEM)** | `10,000` | `48.2%` | `2.1840` | `8.88` | `73.2%` | `63.8%` |
| **Stage 3 (PhD Specialist Synthesis)** | `19,500` | `75.8%` | `1.9799` | `7.24` | `69.5%` | `63.2%` |
| **Stage 4 (Deep `<think>` & Tool Agents)** | `33,977` | **`100.0%`** | **`1.6793`** | **`5.36`** | **`75.5%`** | **`66.3%`** |

### B. Upper-Cortex (`L5–L7`) Post-Training (`60` Books + `50/50` Replay + `0.35 * KL` Anchor)

| Held-Out Evaluation Metric | Base Checkpoint (`Step 33,977`) | `Phobos-1` Final (Post-Trained) | Delta |
| :--- | :---: | :---: | :---: |
| **System-2 Reasoning Perplexity (`think_ppl`)** | `9.76` (`ce=2.2778`) | **`8.53`** (`ce=2.1433`) | **`-12.6%`** |
| **System-1 Factual Perplexity (`fact_ppl`)** | `2.32` (`ce=0.8404`) | **`1.98`** (`ce=0.6841`) | **`-14.7%`** |
| **Held-Out Cortical Branch Accuracy** | `63.77%` | **`64.74%`** | **`+0.97%`** |
| **Held-Out Specialist Leaf Accuracy** | `49.73%` | **`50.22%`** | **`+0.49%`** |

### C. Head-to-Head Benchmark vs. Open-Source Models (`50M – 165M` Parameter Class)

![MarsNext Phobos-1 Artificial Analysis Style Benchmark](assets/phobos1_benchmark_card.png)

Evaluated via `run_phobos1_benchmark_suite.py`, `run_multi_model_comparison.py`, and `generate_benchmark_charts.py` (`multi_model_head_to_head_benchmark.json`):
- **`MC-20` (`ARC-Easy / SciQ / PIQA / CS-Math`)**: 20 standardized 4-choice questions scored via tokenizer-independent length-normalized log-likelihood.
- **`Gen-25` (`25-Task Generative Suite`)**: Python `Pass@1` (`exec()` unit-test verification, 5 tasks), SQL Query Synthesis (5 tasks), STEM & Science QA (5 tasks), Symbolic Math & Definitions (5 tasks), and System-2 `<think>` Planning (5 tasks).

| Model | Org | Total Params | **Active Params / Token** | Training Tokens | `MC-20` (`ARC/SciQ/PIQA`) | **`Gen-25` Overall** | Python `Pass@1` (`exec()`) | SQL Synthesis | STEM QA | Math Symbolic | `<think>` Planning | **Combined Avg** |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **`marsnext/Phobos-1` (Final)** | **MarsNext** | **`96.26M`** | **`52.22M`** | **`1.56B`** | `35.0%` (`7/20`) | **`80.0%` (`20/25`)** | **`60%` (`3/5`)** | **`80%` (`4/5`)** | **`80%` (`4/5`)** | **`80%` (`4/5`)** | **`100%` (`5/5`)** | **`57.5%`** |
| **`marsnext/Phobos-1` (Base)** | **MarsNext** | `96.26M` | `52.22M` | `1.56B` | `35.0%` (`7/20`) | `68.0%` (`17/25`) | **`60%` (`3/5`)** | **`80%` (`4/5`)** | `60%` (`3/5`) | `40%` (`2/5`) | **`100%` (`5/5`)** | `51.5%` |
| `HuggingFaceTB/SmolLM2-135M` | HuggingFace | `134.52M` | `134.52M` (`2.6x`) | `2,000B` | **`60.0%` (`12/20`)** | `36.0%` (`9/25`) | `0%` (`0/5`) | `40%` (`2/5`) | **`80%` (`4/5`)** | `20%` (`1/5`) | `40%` (`2/5`) | `48.0%` |
| `openai-community/gpt2` | OpenAI | `124.44M` | `124.44M` (`2.4x`) | `~40B` | `40.0%` (`8/20`) | `20.0%` (`5/25`) | `0%` (`0/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `40%` (`2/5`) | `30.0%` |
| `EleutherAI/pythia-70m-deduped` | EleutherAI | `70.43M` | `70.43M` (`1.35x`) | `300B` | `30.0%` (`6/20`) | `12.0%` (`3/25`) | `0%` (`0/5`) | `0%` (`0/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `20%` (`1/5`) | `21.0%` |
| `EleutherAI/pythia-160m-deduped` | EleutherAI | `162.32M` | `162.32M` (`3.1x`) | `300B` | `25.0%` (`5/20`) | `12.0%` (`3/25`) | `0%` (`0/5`) | `0%` (`0/5`) | `20%` (`1/5`) | `0%` (`0/5`) | `40%` (`2/5`) | `18.5%` |

---

## 8. Repository Files

- `model.safetensors`: `BF16` SafeTensors weights (`96,256,569` parameters).
- `phobos1_post_trained_bf16.pt`: Post-trained PyTorch checkpoint (60-Book Pedagogy + `50/50` Replay + `KL` Anchor).
- `phobos1_base_100pct_step33977_bf16.pt`: Untouched `100.0%` Full-Corpus Base Pre-Trained checkpoint (`Step 33,977 / 33,977`).
- `config.json` & `phobos_bpe_32k.json` (`tokenizer.json`): Model configuration and `32,768` Byte-Level BPE tokenizer.
- `chat_cli.py`: Interactive terminal chat & single-prompt inference CLI.
- `serve_openai_api.py`: Zero-dependency OpenAI-compatible HTTP server (`/v1/chat/completions`, `/v1/models`).
- `architecture.py`, `deepinfer_modules.py`, `mitosis.py`, `bpe_tokenizer.py`: Self-contained PyTorch architecture modules.
- `training_telemetry_33977steps.json` & `benchmark_10prompts_comparison.json`: Complete training curve, `44` mitosis events, and side-by-side evaluation logs.

---

## 9. Technical Limitations

- `Phobos-1` is an ultra-compact (`96.26M` total / `52.22M` active parameters) model designed for research and resource-constrained edge inference. While it handles foundational Python functions, single-variable linear equations, basic biology/chemistry, and simple deductions reliably, multi-hop combinatorics, adversarial linguistic word-traps, and multi-step arithmetic divisions can produce calculation slips without external tool verification.
- Maximum context window is `2,048` tokens.