File size: 9,635 Bytes
6ec3ff4 92a5c7c 6ec3ff4 29a6726 6ec3ff4 92a5c7c 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 29a6726 6ec3ff4 92a5c7c 6ec3ff4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 | ---
language:
- en
license: other
license_name: mit-nc
license_link: https://github.com/MtnMCG/PCEI/blob/main/LICENSE
tags:
- pcei
- bitnet
- ternary
- 1.58bit
- mamba
- ssm
- state-space-model
- fastkan
- linear-attention
- energy-efficient
- green-ai
pipeline_tag: text-generation
---
# PCEI-Text: Physically Constrained Emergent Intelligence (750M Foundation & 100M Edge Ternary Mamba)
[](https://github.com/MtnMCG/PCEI/blob/main/LICENSE)
[](https://github.com/MtnMCG/PCEI)
[](https://huggingface.co/MtnMCG/PCEI-Robotics)
[]()
[]()
[%20Memory%20%7C%20O(L)%20Compute-green.svg)]()
[]()
> **Full Codebase, Theoretical Compendium, and Replication Suite**: [https://github.com/MtnMCG/PCEI](https://github.com/MtnMCG/PCEI)
---
## 🏛️ Executive Summary
**PCEI-Text** is an open-weights linear-time sequence foundation model family that discards quadratic Transformer attention ($\mathcal{O}(L^2)$) and high-energy 16-bit floating-point multiplications in favor of:
1. **Hardware-efficient Selective State Spaces (Mamba SSM)** operating with linear $\mathcal{O}(L)$ compute and constant $\mathcal{O}(1)$ memory inference.
2. **Canonical Microsoft BitNet b1.58 Ternary Quantization**, constraining synaptic weights to strictly discrete $\{-1.0, 0.0, +1.0\}$ states and replacing energy-hungry floating-point multiplications ($FMUL$) with integer additions ($IADD$).
3. **Kolmogorov-Arnold Network (FastKAN) Spline Manifolds**, utilizing learnable cubic B-splines along non-linear parameter trajectories.
4. **Dual-Timescale Dynamic State Modulation (`AmbientField`)**, preventing context degradation and drifting attractors over extended multi-turn horizons.
### Checkpoints Available in this Repository
| Checkpoint File | Parameters | Size | Description | Target Use Case |
| :--- | :---: | :---: | :--- | :--- |
| **`pcei_750m_competitive_best.pt`** | **750M** | **1.77 GB** | **Flagship Foundation Model**. Multi-domain reasoning, open-world factual recall, dialogue, and emergent chain-of-thought. | Primary foundation deployment & research |
| **`pcei_100m_bitnet_ternary.pt`** | **100M** | **1.09 GB** | Micro-scale edge model. Canonical BitNet b1.58 ternary baseline. | Resource-constrained edge & mobile devices |
---
## 🔬 Empirical Audits & Verification
### 1. PCEI 750M Multi-Domain Benchmark Performance
Evaluated under zero-leakage, greedy argmax neural generation:
| Domain | 100-Question Multi-Domain Benchmark |
| :--- | :---: |
| **1. Open Factual QA** (40 Questions) | **95.0% - 100.0%** |
| **2. Conversational Dialogue** (30 Questions) | **86.7% - 93.3%** |
| **3. Deterministic Arithmetic & CoT** (30 Questions) | **70.0% - 80.0%** |
| **Overall Benchmark Accuracy** | **89.0% - 92.0%** |
### 2. Tiny LM Compute-Matched Bake-Off (Muse Autonomous Evaluation)
In an independent multi-seed compute-matched baseline bake-off conducted via the Muse evaluation agent against a compute-matched Transformer:
| Model | Architecture | Parameters | Val Loss (mean ± std) | Val Perplexity | Win Rate |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Transformer** | Vanilla Decoder (5L, 4H, FFN $2\times d$ GELU) | 398,688 | 4.6131 ± 0.0003 | 100.8 | Baseline |
| **Dense Mamba** | Repo `MambaBlock` (6L, FP32 dense weights) | 407,232 | 4.3881 ± 0.0009 | 80.5 | 0 / 2 vs PCEI |
| **PCEI-tiny** | **Strict 1.58-bit Ternary (`{-1, 0, +1}`) + FastKAN** | 411,912 | **4.3783 ± 0.0073** | **79.1** | **2 / 2 (Won all seeds)** |
* **Key Takeaway**: Despite linear projections constrained strictly to discrete $\{-1, 0, +1\}$ ternary states (zero floating-point multiplications), PCEI outperformed the dense FP32 Mamba baseline and beat the standard Transformer by **0.23 nats** (val perplexity 79.1 vs 100.8).
### 3. Strict Microsoft BitNet b1.58 Migration
An external audit in October 2026 disclosed that early internal prototypes utilized an experimental 5-state quantizer. The architecture was fully refactored to canonical Microsoft BitNet b1.58:
- **Parameter Entropy Cut**: Reduced from 2.32b to strictly 1.58b ($\le \log_2(3)$ bits/param), cutting parameter entropy by **32%**.
- **Perplexity & Hallucinations**: Validation loss improved from 11.1696 to 11.1429 on 100M, eliminating activation outlier code token hallucinations.
- Full scientific post-mortem report: [`AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md`](https://github.com/MtnMCG/PCEI/blob/main/pcei_framework/docs/AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md).
---
## ⚙️ Model Specifications
| Architectural Dimension | PCEI-750M (Flagship Foundation) | PCEI-100M (Edge) |
| :--- | :--- | :--- |
| **Base Architecture** | Selective State Space (Mamba SSM) + Latent Diffusion | 12-Layer Selective State Space (Mamba SSM) |
| **Model Dimension ($d_{\text{model}}$)** | **1024** | 768 |
| **Inner Expansion Dimension ($d_{\text{inner}}$)** | **2048** | 512 |
| **State Dimension ($d_{\text{state}}$)** | **32** | 16 |
| **Convolution Kernel Size** | 4 | 4 |
| **Non-Linear Branch** | FastKAN (Cubic B-splines) | FastKAN (Cubic B-splines) |
| **Synaptic Quantization** | Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE | Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE |
| **Vocabulary Size** | 50,257 (OpenAI GPT-2 BPE) | 50,257 (OpenAI GPT-2 BPE) |
| **Context Length ($L$)** | Up to 2,048 tokens | 128 / 512 tokens |
| **Hardware Target** | Apple Silicon MPS, NVIDIA CUDA, CPU edge | Apple Silicon MPS, NVIDIA CUDA, CPU edge |
---
## 🚀 Local Inference Quickstart
To run local text generation using the PCEI framework:
```bash
# 1. Clone the master GitHub repository
git clone https://github.com/MtnMCG/PCEI.git
cd PCEI/pcei_framework
# 2. Set up virtual environment and install dependencies
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install huggingface_hub
# 3. Download the model checkpoint directly from Hugging Face
python3 -c "
from huggingface_hub import hf_hub_download
# Download Flagship 750M Foundation Model (or 'pcei_100m_bitnet_ternary.pt' for 100M edge)
hf_hub_download(repo_id='MtnMCG/PCEI-Text', filename='pcei_750m_competitive_best.pt', local_dir='checkpoints')
"
# 4. Interactive generation with continuous state modulation
python3 tools/resonance_chat.py \
--checkpoint checkpoints/pcei_750m_competitive_best.pt \
--device mps # Use --device cuda on NVIDIA GPUs
```
### Python API Usage
```python
import torch
from src.pcei import PCEINexus
from src.config import PCEIConfig
from src.tokenizer import PCEITokenizer
device = torch.device("mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu"))
# Load checkpoint (750M Foundation Model)
checkpoint = torch.load("checkpoints/pcei_750m_competitive_best.pt", map_location=device)
cfg_obj = checkpoint.get("config", None)
config = cfg_obj if isinstance(cfg_obj, PCEIConfig) else PCEIConfig.load()
# Initialize model
nexus = PCEINexus(config=config).to(device)
nexus.load_state_dict(checkpoint["model_state_dict"], strict=False)
nexus.eval()
# Tokenize prompt
tokenizer = PCEITokenizer(vocab_size=config.vocab_size)
prompt = "USER: Explain why ice floats on water.\nASSISTANT:"
input_ids = torch.tensor(tokenizer.encode_raw(prompt), dtype=torch.long).unsqueeze(0).to(device)
# Autoregressive generation
with torch.no_grad():
for _ in range(64):
outputs = nexus({"text": input_ids})
logits = outputs["logits"] if isinstance(outputs, dict) else outputs
next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
input_ids = torch.cat([input_ids, next_token], dim=-1)
print(tokenizer.decode(input_ids[0].tolist()))
```
---
## 🤖 Sister Repositories & Embodied Intelligence
PCEI proves that physical constraints generalize across domains:
- 🏃 **[MtnMCG/PCEI-Robotics](https://huggingface.co/MtnMCG/PCEI-Robotics)**: Autonomous American Ninja Warrior 18-meter humanoid controller traversing 45° angled Quad Steps without attention or deep RL.
- 🏛️ **[PCEI Master GitHub Repository](https://github.com/MtnMCG/PCEI)**: Contains the 11 theoretical research papers, continuous training daemons, curriculum streaming pipelines, and full verification suite.
---
## 📜 Citation & License
This model and the PCEI codebase are released under the **MIT Non-Commercial License (MIT-NC)**. Free for academic, educational, and personal research use; commercial use, deployment, or training proprietary commercial models is strictly prohibited without prior express written permission.
```bibtex
@article{ramirez2026pcei,
title = {Physically Constrained Emergent Intelligence: Subverting Transformer Hyperscaling via Linear State Spaces, Ternary Synaptic Gating, and Latent Generative Replay},
author = {Ramirez, Emerson and Antigravity Assistant},
journal = {PCEI Theoretical Research Compendium},
year = {2026},
url = {https://github.com/MtnMCG/PCEI}
}
```
|