README.md

metadata

license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
language:
  - en
tags:
  - marsnext
  - phobos-1
  - text-generation
  - conversational
  - code-generation
  - math-reasoning
  - system-2-reasoning
  - mixture-of-experts
  - cortical-moe
  - dynamic-mitosis
  - deepinfer-engram
  - multi-token-prediction
  - openai-compatible
  - edge-inference
  - safetensors
  - pytorch

MarsNext Phobos-1 (96.26M Total / 52.22M Active Parameters)

Organization: MarsNext (@0xMarsNext) | Repository: marsnext/Phobos-1 | License: Apache-2.0

Phobos-1 is a from-scratch Hierarchical Cortical Mixture-of-Experts (MoE) causal language model (96,256,569 total parameters, 52,216,377 active parameters per token) featuring 2-Stage Spherical Cortical Routing (Branch -> Specialist Leaf), Online SVD Spherical Mitosis, DeepInfer Engram Hash Memory (2-gram & 3-gram), Multi-Token Prediction (MTP), and System-2 (<think>...</think>) Chain-of-Thought Reasoning.

---

1. Capabilities & Task Tags

Capability Tag	Supported Tasks & Behavior
text-generation / conversational	Single-turn and multi-turn instruction following (`<
system-2-reasoning (<think>)	Step-by-step scratchpad deduction inside <think>...</think> blocks prior to final answer generation
code-generation (python, js, sql)	Function synthesis, list/array manipulation, algorithmic walkthroughs, and bug analysis
math-reasoning	Symbolic linear algebra, step-by-step equation isolation, power-rule calculus, and arithmetic self-check
science-qa & deductive-logic	Foundational biology, physics kinematics, acid-base chemistry, and categorical syllogisms
openai-compatible	Built-in /v1/chat/completions and /v1/models HTTP server (serve_openai_api.py) with SSE streaming
edge-inference (cuda / mps / cpu)	215.84 MiB (BF16) footprint; runs offline on Apple Silicon (MPS), NVIDIA GPUs (CUDA), or CPU

---

2. Architecture Specification

Component	Technical Specification
Model Class	PhobosScratchLM (architecture.py)
Total Parameters	96,256,569 (215.84 MiB in BF16)
Active Parameters / Token	52,216,377 (54.2% active per token: Shared E00 + Top-1 Branch + Top-2 Leaves)
Backbone Dimensions	8 Layers, d_model = 512, max_seq_len = 2048, RMSNorm ($\epsilon = 10^{-6}$)
Attention Mechanism	Causal Grouped-Query Attention (GQA: 8 Query Heads, 4 KV Heads, head_dim = 64), 1D RoPE + Short-Range Causal QK-Conv (window = 8)
Cortical MoE Layer	shared_e00 (1,024 SwiGLU neurons) + 4 Cortical Branches $\times$ 16 Specialist Leaves/Layer (256 SwiGLU neurons each, grown from 8 -> 16 via 44 SVD Mitosis splits)
Cortical Branch Domains	B0: Language & Dialogue • B1: Code & Software Engineering • B2: Math & Formal Logic • B3: Natural Sciences
Declarative Memory Bank	DeepInferEngramMemory (2-gram & 3-gram deterministic hash tables injected at layers L1 & L5)
Auxiliary Training Heads	DeepInferMTPHead (t+2 Multi-Token Prediction) + SigmoidHCBlockMixin residual highway gates
Tokenizer	Custom 32,768 (2^15) Byte-Level BPE (phobos_bpe_32k.json / tokenizer.json)

---

3. Installation & Download

pip install torch tokenizers safetensors huggingface_hub openai

Download the repository via huggingface_hub or git:

from huggingface_hub import snapshot_download
repo_dir = snapshot_download("marsnext/Phobos-1", local_dir="./Phobos-1")

git lfs install
git clone https://huggingface.co/marsnext/Phobos-1
cd Phobos-1

---

4. How to Run & Chat (chat_cli.py)

A. Interactive Terminal Chat (Streaming)

python chat_cli.py

In-chat commands:

• /think — Toggle <think>...</think> System-2 reasoning mode ON/OFF
• /quit — Exit session

With explicit sampling parameters:

python chat_cli.py --think --temp 0.2 --top-p 0.9 --rep-penalty 1.15 --max-tokens 256

B. One-Shot Command-Line Inference

# Standard direct response
python chat_cli.py --prompt "Write a Python function sum_even(nums) that returns the sum of all even integers in a list."

# Force System-2 <think> reasoning trace
python chat_cli.py --prompt "Solve 3x + 7 = 22 step by step." --think

---

5. OpenAI-Compatible API Server Setup (serve_openai_api.py)

Phobos-1 ships with a self-contained OpenAI-compatible HTTP server (serve_openai_api.py) exposing /v1/models, /v1/chat/completions, /v1/completions, and /health (supporting both standard JSON responses and stream=True Server-Sent Events).

Step 1: Start the Server

python serve_openai_api.py --host 0.0.0.0 --port 8000

Step 2: Query via curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "marsnext/Phobos-1",
    "messages": [
      {"role": "user", "content": "Solve 3x + 7 = 22 step by step."}
    ],
    "temperature": 0.0,
    "max_tokens": 200
  }'

Step 3: Query via Official openai Python SDK (Standard & Streaming)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="marsnext")

# 1. Non-Streaming Completion
response = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Write a Python function `sum_even(nums)` that returns the sum of all even integers in a list."}
    ],
    temperature=0.0,
    max_tokens=160,
)
print(response.choices[0].message.content)

# 2. Streaming Completion (SSE)
stream = client.chat.completions.create(
    model="marsnext/Phobos-1",
    messages=[
        {"role": "user", "content": "Why do muscle cells have a high density of mitochondria compared to skin cells?"}
    ],
    temperature=0.0,
    max_tokens=160,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

---

6. Direct PyTorch API Usage

import sys
import torch
from huggingface_hub import snapshot_download

repo_dir = snapshot_download("marsnext/Phobos-1")
sys.path.insert(0, repo_dir)

from architecture import PhobosScratchConfig, PhobosScratchLM
from bpe_tokenizer import PhobosBPETokenizer
from mitosis import load_mitosis_state_dict

device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
tok = PhobosBPETokenizer(f"{repo_dir}/phobos_bpe_32k.json")
ckpt = torch.load(f"{repo_dir}/phobos1_post_trained_bf16.pt", map_location="cpu", weights_only=False)

cfg = PhobosScratchConfig(vocab_size=tok.VOCAB_SIZE)
model = PhobosScratchLM(cfg)
load_mitosis_state_dict(
    model,
    {k: v.float() if torch.is_floating_point(v) else v for k, v in ckpt["model_state_dict"].items()},
)
model = model.to(device).eval()

prompt = "<|bos|><|user|>\nSolve 3x + 7 = 22 step by step.\n<|assistant|>\n"
ids = torch.tensor([tok.encode(prompt, max_len=512)], dtype=torch.long, device=device)

with torch.no_grad():
    for _ in range(160):
        logits = model(ids)["logits"][0, -1, :].float()
        next_id = int(torch.argmax(logits).item())
        if next_id == tok.EOS_ID:
            break
        ids = torch.cat([ids, torch.tensor([[next_id]], dtype=torch.long, device=device)], dim=1)

print(tok.decode(ids[0].tolist(), skip_special_tokens=False))

---

7. Training & Evaluation Telemetry

A. Base Pre-Training Progression (100.0% of 35 Datasets — 33,977 / 33,977 Steps)

Trained on 3,627,127 complete sequences (1,561,410,323 non-pad BPE tokens) across a 4-stage curriculum:

Stage / Milestone	Step	Corpus Coverage	Validation CE (val_ce)	Validation Perplexity (val_ppl)	Cortical Branch Acc	Specialist Leaf Acc
Stage 1 (Foundational Language)	2,500	15.4%	2.6410	14.03	78.4%	64.1%
Stage 2 (Code, Math & STEM)	10,000	48.2%	2.1840	8.88	73.2%	63.8%
Stage 3 (PhD Specialist Synthesis)	19,500	75.8%	1.9799	7.24	69.5%	63.2%
Stage 4 (Deep <think> & Tool Agents)	33,977	100.0%	1.6793	5.36	75.5%	66.3%

B. Upper-Cortex (L5–L7) Post-Training (60 Books + 50/50 Replay + 0.35 * KL Anchor)

Held-Out Evaluation Metric	Base Checkpoint (Step 33,977)	Phobos-1 Final (Post-Trained)	Delta
System-2 Reasoning Perplexity (think_ppl)	9.76 (ce=2.2778)	8.53 (ce=2.1433)	-12.6%
System-1 Factual Perplexity (fact_ppl)	2.32 (ce=0.8404)	1.98 (ce=0.6841)	-14.7%
Held-Out Cortical Branch Accuracy	63.77%	64.74%	+0.97%
Held-Out Specialist Leaf Accuracy	49.73%	50.22%	+0.49%

C. Head-to-Head Benchmark vs. Open-Source Models (50M – 165M Parameter Class)

Evaluated via run_phobos1_benchmark_suite.py, run_multi_model_comparison.py, and generate_benchmark_charts.py (multi_model_head_to_head_benchmark.json):

• MC-20 (ARC-Easy / SciQ / PIQA / CS-Math): 20 standardized 4-choice questions scored via tokenizer-independent length-normalized log-likelihood.
• Gen-25 (25-Task Generative Suite): Python Pass@1 (exec() unit-test verification, 5 tasks), SQL Query Synthesis (5 tasks), STEM & Science QA (5 tasks), Symbolic Math & Definitions (5 tasks), and System-2 <think> Planning (5 tasks).

Model	Org	Total Params	Active Params / Token	Training Tokens	MC-20 (ARC/SciQ/PIQA)	Gen-25 Overall	Python Pass@1 (exec())	SQL Synthesis	STEM QA	Math Symbolic	<think> Planning	Combined Avg
marsnext/Phobos-1 (Final)	MarsNext	96.26M	52.22M	1.56B	35.0% (7/20)	80.0% (20/25)	60% (3/5)	80% (4/5)	80% (4/5)	80% (4/5)	100% (5/5)	57.5%
marsnext/Phobos-1 (Base)	MarsNext	96.26M	52.22M	1.56B	35.0% (7/20)	68.0% (17/25)	60% (3/5)	80% (4/5)	60% (3/5)	40% (2/5)	100% (5/5)	51.5%
HuggingFaceTB/SmolLM2-135M	HuggingFace	134.52M	134.52M (2.6x)	2,000B	60.0% (12/20)	36.0% (9/25)	0% (0/5)	40% (2/5)	80% (4/5)	20% (1/5)	40% (2/5)	48.0%
openai-community/gpt2	OpenAI	124.44M	124.44M (2.4x)	~40B	40.0% (8/20)	20.0% (5/25)	0% (0/5)	20% (1/5)	20% (1/5)	20% (1/5)	40% (2/5)	30.0%
EleutherAI/pythia-70m-deduped	EleutherAI	70.43M	70.43M (1.35x)	300B	30.0% (6/20)	12.0% (3/25)	0% (0/5)	0% (0/5)	20% (1/5)	20% (1/5)	20% (1/5)	21.0%
EleutherAI/pythia-160m-deduped	EleutherAI	162.32M	162.32M (3.1x)	300B	25.0% (5/20)	12.0% (3/25)	0% (0/5)	0% (0/5)	20% (1/5)	0% (0/5)	40% (2/5)	18.5%

---

8. Repository Files

• model.safetensors: BF16 SafeTensors weights (96,256,569 parameters).
• phobos1_post_trained_bf16.pt: Post-trained PyTorch checkpoint (60-Book Pedagogy + 50/50 Replay + KL Anchor).
• phobos1_base_100pct_step33977_bf16.pt: Untouched 100.0% Full-Corpus Base Pre-Trained checkpoint (Step 33,977 / 33,977).
• config.json & phobos_bpe_32k.json (tokenizer.json): Model configuration and 32,768 Byte-Level BPE tokenizer.
• chat_cli.py: Interactive terminal chat & single-prompt inference CLI.
• serve_openai_api.py: Zero-dependency OpenAI-compatible HTTP server (/v1/chat/completions, /v1/models).
• architecture.py, deepinfer_modules.py, mitosis.py, bpe_tokenizer.py: Self-contained PyTorch architecture modules.
• training_telemetry_33977steps.json & benchmark_10prompts_comparison.json: Complete training curve, 44 mitosis events, and side-by-side evaluation logs.

---

9. Technical Limitations

• Phobos-1 is an ultra-compact (96.26M total / 52.22M active parameters) model designed for research and resource-constrained edge inference. While it handles foundational Python functions, single-variable linear equations, basic biology/chemistry, and simple deductions reliably, multi-hop combinatorics, adversarial linguistic word-traps, and multi-step arithmetic divisions can produce calculation slips without external tool verification.
• Maximum context window is 2,048 tokens.