| --- |
| license: apache-2.0 |
| language: |
| - en |
| - multilingual |
| tags: |
| - text-generation |
| - safetensors |
| - reasoning |
| - solana |
| - from-scratch |
| - small-model |
| - multilingual |
| - custom-architecture |
| pipeline_tag: text-generation |
| inference: true |
| --- |
| |
| # Mind-v1: Deterministic Reasoning Model (35.8M Parameters) |
|
|
| **Mind-v1** is a compact 35.8-million parameter decoder-only transformer trained from scratch on algorithmic and logical reasoning problems generated by **The Mind** autonomous protocol on Solana. |
|
|
| Unlike marketing-heavy releases, Mind-v1 does not claim to replace frontier foundation models. It is a purpose-built, lightweight, 100% open-weights reasoning model designed to run with sub-100ms latency on commodity CPU hardware with zero API dependencies. |
|
|
| --- |
|
|
| ## Model Architecture |
|
|
| Mind-v1 uses a modern LLaMA-style decoder-only transformer architecture with SwiGLU activations, RMSNorm, and Rotary Position Embeddings (RoPE), configured with an independent custom Byte-Level BPE tokenizer. |
|
|
| | Parameter | Value | Note | |
| |---|---|---| |
| | **Total Parameters** | 35,756,544 (~35.8M) | Exact non-embedding + embedding weights | |
| | **Layers (Transformer Blocks)** | 6 | Decoder-only self-attention | |
| | **Hidden Size ($d_{\text{model}}$)** | 512 | Representation dimension | |
| | **Intermediate Size** | 1,376 | SwiGLU feed-forward projection | |
| | **Attention Heads** | 8 | 64 dimensions per head | |
| | **Vocabulary Size** | 16,384 | Custom Byte-Level BPE tokenizer | |
| | **Context Window** | 2,048 tokens | Trained on sequence length 128 | |
| | **Position Embeddings** | RoPE | Rotary position embeddings | |
| | **Normalization** | RMSNorm | $\epsilon = 10^{-5}$ | |
| | **Weight Format** | SafeTensors | 143 MB (`model.safetensors`, float32) | |
| |
| --- |
| |
| ## Training & Loss Convergence |
| |
| The model was trained from scratch with randomly initialized weights (no pre-trained base model weights were used). |
| |
| - **Training Dataset**: [`ShoggothMind/mind-judgements`](https://huggingface.co/datasets/ShoggothMind/mind-judgements) (`synthetic_train` split, 3,000 verified reasoning examples across 8 task types and 10 difficulty tiers). |
| - **Validation Dataset**: 500 held-out examples. |
| - **Optimizer**: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay = $0.01$, learning rate = $3\times 10^{-4}$). |
| - **Scheduler**: Cosine Annealing with 100 warmup steps. |
| - **Initial Loss**: `9.7041` (accurately matches theoretical uniform random baseline $-\ln(1/16384) = 9.704$). |
| - **Final Training Loss**: `1.0608` |
| - **Final Validation Loss**: `1.2210` |
| - **Training Time**: ~14.9 minutes on commodity hardware. |
| |
| ### Training Loss Curve |
|  |
| |
| --- |
| |
| ## Evaluation Benchmark Against Frontier Models |
| |
| We evaluated **Mind-v1** against six frontier models accessible via OpenRouter on an identical held-out evaluation slice from `test.jsonl`. |
|
|
| | Model | Provider / Type | Parameters | Exact-Match Accuracy | Avg Latency (CPU/API) | Cost per 1M Tokens | Memory Footprint | Open Weights | |
| |---|---|---|---|---|---|---|---| |
| | **Mind-v1 (Ours)** | Local / Open Weights | **35.8M** | **6.0%** (Direct output) | **96.0 ms (CPU)** | **$0.00** | **143 MB** | **Yes** | |
| | DeepSeek V3 | Cloud MoE API | 671B (37B active) | 20.0% | 5,080 ms | $0.14 / $0.28 | Cloud | No | |
| | Qwen 2.5 72B | Cloud API | 72B | 60.0% | 1,410 ms | $0.35 / $0.40 | Cloud | Yes (Weights) | |
| | Claude 3.5 Sonnet | Cloud API | Proprietary | 0.0%* | 350 ms | $3.00 / $15.00 | Cloud | No | |
| | GPT-4o | Cloud API | Proprietary | 80.0% | 1,930 ms | $2.50 / $10.00 | Cloud | No | |
| | Llama 3.3 70B | Cloud API | 70B | 60.0% | 2,300 ms | $0.30 / $0.40 | Cloud | Yes (Weights) | |
| | Mistral Large | Cloud API | Proprietary | 0.0%* | 480 ms | $2.00 / $6.00 | Cloud | No | |
|
|
| *\*Note: 0% results on some frontier models occurred due to conversational refusal formatting or answering outside strict `<answer>` tags on single-token generation constraints.* |
|
|
| ### Mind-v1 Task Breakdown |
| - **Logic & Relations**: 30.0% |
| - **Letter Counting**: 10.0% |
| - **Weekday Calculation**: 10.0% |
| - **Number Sequences**: 5.0% |
| - **Test Set Cross-Entropy Loss**: `1.8986` |
| - **Test Set Perplexity**: `6.68` |
|
|
| --- |
|
|
| ## Honest Analysis: Strengths vs Limitations |
|
|
| ### Strengths |
| 1. **Zero Cost & Offline Capability**: Runs entirely on a single CPU core in 143 MB of RAM without external API keys or cloud billing. |
| 2. **Deterministic Output Tags**: Consistently formats answers within `<answer>...</answer>` tags as learned during training. |
| 3. **Low Latency**: ~96 ms per inference on commodity x86_64 CPU. |
| 4. **Reproducible & Verifiable**: Custom 16,384 BPE tokenizer and training pipeline completely open source in the project repository. |
| |
| ### Limitations |
| 1. **Model Capacity**: At 35.8 million parameters, the model cannot perform complex multi-step mental arithmetic (e.g., multi-digit multiplication or modular exponentiation) without an external chain-of-thought scratchpad. |
| 2. **Domain Focus**: Trained strictly on synthetic math, sequence, logic, and calendar problems. It has no open-domain world knowledge, literature, or trivia understanding. |
| 3. **Context Length**: Context is configured to 2,048 tokens, trained on 128-token lengths, and is not suitable for document summarization. |
| |
| --- |
| |
| ## Quickstart & Usage |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "ShoggothMind/mind-v1" |
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
|
|
| prompt = "Question: What is 45 + 17?\\nAnswer: <answer>" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=16, |
| do_sample=False, |
| pad_token_id=tokenizer.pad_token_id, |
| eos_token_id=tokenizer.eos_token_id |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
| |
| --- |
| |
| ## Protocol Integration |
| |
| To query the live consensus swarm of The Mind on Solana via HTTP API instead of running weights locally: |
| |
| ```bash |
| python examples/query_the_mind.py --url https://shoggothmind.live --question "What is the consensus truth?" |
| ``` |
| |
| --- |
| |
| ## License |
| |
| Apache License 2.0 ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)). |
| |
