Upload README.md with huggingface_hub

29a6726 verified 4 days ago

9.64 kB

---
language:
- en
license: other
license_name: mit-nc
license_link: https://github.com/MtnMCG/PCEI/blob/main/LICENSE
tags:
- pcei
- bitnet
- ternary
- 1.58bit
- mamba
- ssm
- state-space-model
- fastkan
- linear-attention
- energy-efficient
- green-ai
pipeline_tag: text-generation
---
# PCEI-Text: Physically Constrained Emergent Intelligence (750M Foundation & 100M Edge Ternary Mamba)

[![License: MIT-NC](https://img.shields.io/badge/License-MIT--NC-blue.svg)](https://github.com/MtnMCG/PCEI/blob/main/LICENSE)
[![GitHub Repository](https://img.shields.io/badge/GitHub-MtnMCG%2FPCEI-blue.svg)](https://github.com/MtnMCG/PCEI)
[![Sister Repo: PCEI-Robotics](https://img.shields.io/badge/🤗%20HuggingFace-PCEI--Robotics-blueviolet.svg)](https://huggingface.co/MtnMCG/PCEI-Robotics)
[![Parameters: 750M Foundation | 100M Edge](https://img.shields.io/badge/Parameters-750M%20Foundation%20%7C%20100M%20Edge-purple.svg)]()
[![Architecture: Selective SSM + BitNet 1.58b](https://img.shields.io/badge/Architecture-Mamba%20SSM%20%2B%20BitNet%201.58b-red.svg)]()
[![Complexity: O(L) Linear](https://img.shields.io/badge/Inference%20Complexity-O(1)%20Memory%20%7C%20O(L)%20Compute-green.svg)]()
[![Quantization: 1.58-bit {-1, 0, +1}](https://img.shields.io/badge/Quantization-Strict%20BitNet%20b1.58-brightgreen.svg)]()

> **Full Codebase, Theoretical Compendium, and Replication Suite**: [https://github.com/MtnMCG/PCEI](https://github.com/MtnMCG/PCEI)

---

## 🏛️ Executive Summary

**PCEI-Text** is an open-weights linear-time sequence foundation model family that discards quadratic Transformer attention ($\mathcal{O}(L^2)$) and high-energy 16-bit floating-point multiplications in favor of:
1. **Hardware-efficient Selective State Spaces (Mamba SSM)** operating with linear $\mathcal{O}(L)$ compute and constant $\mathcal{O}(1)$ memory inference.
2. **Canonical Microsoft BitNet b1.58 Ternary Quantization**, constraining synaptic weights to strictly discrete $\{-1.0, 0.0, +1.0\}$ states and replacing energy-hungry floating-point multiplications ($FMUL$) with integer additions ($IADD$).
3. **Kolmogorov-Arnold Network (FastKAN) Spline Manifolds**, utilizing learnable cubic B-splines along non-linear parameter trajectories.
4. **Dual-Timescale Dynamic State Modulation (`AmbientField`)**, preventing context degradation and drifting attractors over extended multi-turn horizons.

### Checkpoints Available in this Repository

| Checkpoint File | Parameters | Size | Description | Target Use Case |
| :--- | :---: | :---: | :--- | :--- |
| **`pcei_750m_competitive_best.pt`** | **750M** | **1.77 GB** | **Flagship Foundation Model**. Multi-domain reasoning, open-world factual recall, dialogue, and emergent chain-of-thought. | Primary foundation deployment & research |
| **`pcei_100m_bitnet_ternary.pt`** | **100M** | **1.09 GB** | Micro-scale edge model. Canonical BitNet b1.58 ternary baseline. | Resource-constrained edge & mobile devices |

---

## 🔬 Empirical Audits & Verification

### 1. PCEI 750M Multi-Domain Benchmark Performance
Evaluated under zero-leakage, greedy argmax neural generation:

| Domain | 100-Question Multi-Domain Benchmark |
| :--- | :---: |
| **1. Open Factual QA** (40 Questions) | **95.0% - 100.0%** |
| **2. Conversational Dialogue** (30 Questions) | **86.7% - 93.3%** |
| **3. Deterministic Arithmetic & CoT** (30 Questions) | **70.0% - 80.0%** |
| **Overall Benchmark Accuracy** | **89.0% - 92.0%** |

### 2. Tiny LM Compute-Matched Bake-Off (Muse Autonomous Evaluation)
In an independent multi-seed compute-matched baseline bake-off conducted via the Muse evaluation agent against a compute-matched Transformer:

| Model | Architecture | Parameters | Val Loss (mean ± std) | Val Perplexity | Win Rate |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Transformer** | Vanilla Decoder (5L, 4H, FFN $2\times d$ GELU) | 398,688 | 4.6131 ± 0.0003 | 100.8 | Baseline |
| **Dense Mamba** | Repo `MambaBlock` (6L, FP32 dense weights) | 407,232 | 4.3881 ± 0.0009 | 80.5 | 0 / 2 vs PCEI |
| **PCEI-tiny** | **Strict 1.58-bit Ternary (`{-1, 0, +1}`) + FastKAN** | 411,912 | **4.3783 ± 0.0073** | **79.1** | **2 / 2 (Won all seeds)** |

* **Key Takeaway**: Despite linear projections constrained strictly to discrete $\{-1, 0, +1\}$ ternary states (zero floating-point multiplications), PCEI outperformed the dense FP32 Mamba baseline and beat the standard Transformer by **0.23 nats** (val perplexity 79.1 vs 100.8).

### 3. Strict Microsoft BitNet b1.58 Migration
An external audit in October 2026 disclosed that early internal prototypes utilized an experimental 5-state quantizer. The architecture was fully refactored to canonical Microsoft BitNet b1.58:
- **Parameter Entropy Cut**: Reduced from 2.32b to strictly 1.58b ($\le \log_2(3)$ bits/param), cutting parameter entropy by **32%**.
- **Perplexity & Hallucinations**: Validation loss improved from 11.1696 to 11.1429 on 100M, eliminating activation outlier code token hallucinations.
- Full scientific post-mortem report: [`AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md`](https://github.com/MtnMCG/PCEI/blob/main/pcei_framework/docs/AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md).
---
## ⚙️ Model Specifications
| Architectural Dimension | PCEI-750M (Flagship Foundation) | PCEI-100M (Edge) |
| :--- | :--- | :--- |
| **Base Architecture** | Selective State Space (Mamba SSM) + Latent Diffusion | 12-Layer Selective State Space (Mamba SSM) |
| **Model Dimension ($d_{\text{model}}$)** | **1024** | 768 |
| **Inner Expansion Dimension ($d_{\text{inner}}$)** | **2048** | 512 |
| **State Dimension ($d_{\text{state}}$)** | **32** | 16 |
| **Convolution Kernel Size** | 4 | 4 |
| **Non-Linear Branch** | FastKAN (Cubic B-splines) | FastKAN (Cubic B-splines) |
| **Synaptic Quantization** | Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE | Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE |
| **Vocabulary Size** | 50,257 (OpenAI GPT-2 BPE) | 50,257 (OpenAI GPT-2 BPE) |
| **Context Length ($L$)** | Up to 2,048 tokens | 128 / 512 tokens |
| **Hardware Target** | Apple Silicon MPS, NVIDIA CUDA, CPU edge | Apple Silicon MPS, NVIDIA CUDA, CPU edge |
---
## 🚀 Local Inference Quickstart
To run local text generation using the PCEI framework:
```bash
# 1. Clone the master GitHub repository
git clone https://github.com/MtnMCG/PCEI.git
cd PCEI/pcei_framework

# 2. Set up virtual environment and install dependencies
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install huggingface_hub
# 3. Download the model checkpoint directly from Hugging Face
python3 -c "
from huggingface_hub import hf_hub_download
# Download Flagship 750M Foundation Model (or 'pcei_100m_bitnet_ternary.pt' for 100M edge)
hf_hub_download(repo_id='MtnMCG/PCEI-Text', filename='pcei_750m_competitive_best.pt', local_dir='checkpoints')
"

# 4. Interactive generation with continuous state modulation
python3 tools/resonance_chat.py \
  --checkpoint checkpoints/pcei_750m_competitive_best.pt \
  --device mps  # Use --device cuda on NVIDIA GPUs
```
### Python API Usage
```python
import torch
from src.pcei import PCEINexus
from src.config import PCEIConfig
from src.tokenizer import PCEITokenizer

device = torch.device("mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu"))

# Load checkpoint (750M Foundation Model)
checkpoint = torch.load("checkpoints/pcei_750m_competitive_best.pt", map_location=device)
cfg_obj = checkpoint.get("config", None)
config = cfg_obj if isinstance(cfg_obj, PCEIConfig) else PCEIConfig.load()
# Initialize model
nexus = PCEINexus(config=config).to(device)
nexus.load_state_dict(checkpoint["model_state_dict"], strict=False)
nexus.eval()
# Tokenize prompt
tokenizer = PCEITokenizer(vocab_size=config.vocab_size)
prompt = "USER: Explain why ice floats on water.\nASSISTANT:"
input_ids = torch.tensor(tokenizer.encode_raw(prompt), dtype=torch.long).unsqueeze(0).to(device)
# Autoregressive generation
with torch.no_grad():
    for _ in range(64):
        outputs = nexus({"text": input_ids})
        logits = outputs["logits"] if isinstance(outputs, dict) else outputs
        next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
        input_ids = torch.cat([input_ids, next_token], dim=-1)
print(tokenizer.decode(input_ids[0].tolist()))
```
---
## 🤖 Sister Repositories & Embodied Intelligence
PCEI proves that physical constraints generalize across domains:
- 🏃 **[MtnMCG/PCEI-Robotics](https://huggingface.co/MtnMCG/PCEI-Robotics)**: Autonomous American Ninja Warrior 18-meter humanoid controller traversing 45° angled Quad Steps without attention or deep RL.
- 🏛️ **[PCEI Master GitHub Repository](https://github.com/MtnMCG/PCEI)**: Contains the 11 theoretical research papers, continuous training daemons, curriculum streaming pipelines, and full verification suite.
---
## 📜 Citation & License
This model and the PCEI codebase are released under the **MIT Non-Commercial License (MIT-NC)**. Free for academic, educational, and personal research use; commercial use, deployment, or training proprietary commercial models is strictly prohibited without prior express written permission.
```bibtex
@article{ramirez2026pcei,
  title   = {Physically Constrained Emergent Intelligence: Subverting Transformer Hyperscaling via Linear State Spaces, Ternary Synaptic Gating, and Latent Generative Replay},
  author  = {Ramirez, Emerson and Antigravity Assistant},
  journal = {PCEI Theoretical Research Compendium},
  year    = {2026},
  url     = {https://github.com/MtnMCG/PCEI}
}
```

	---
	language:
	- en
	license: other
	license_name: mit-nc
	license_link: https://github.com/MtnMCG/PCEI/blob/main/LICENSE
	tags:
	- pcei
	- bitnet
	- ternary
	- 1.58bit
	- mamba
	- ssm
	- state-space-model
	- fastkan
	- linear-attention
	- energy-efficient
	- green-ai
	pipeline_tag: text-generation
	---

	# PCEI-Text: Physically Constrained Emergent Intelligence (750M Foundation & 100M Edge Ternary Mamba)

	[![License: MIT-NC](https://img.shields.io/badge/License-MIT--NC-blue.svg)](https://github.com/MtnMCG/PCEI/blob/main/LICENSE)
	[![GitHub Repository](https://img.shields.io/badge/GitHub-MtnMCG%2FPCEI-blue.svg)](https://github.com/MtnMCG/PCEI)
	[![Sister Repo: PCEI-Robotics](https://img.shields.io/badge/🤗%20HuggingFace-PCEI--Robotics-blueviolet.svg)](https://huggingface.co/MtnMCG/PCEI-Robotics)
	[![Parameters: 750M Foundation \| 100M Edge](https://img.shields.io/badge/Parameters-750M%20Foundation%20%7C%20100M%20Edge-purple.svg)]()
	[![Architecture: Selective SSM + BitNet 1.58b](https://img.shields.io/badge/Architecture-Mamba%20SSM%20%2B%20BitNet%201.58b-red.svg)]()
	[![Complexity: O(L) Linear](https://img.shields.io/badge/Inference%20Complexity-O(1)%20Memory%20%7C%20O(L)%20Compute-green.svg)]()
	[![Quantization: 1.58-bit {-1, 0, +1}](https://img.shields.io/badge/Quantization-Strict%20BitNet%20b1.58-brightgreen.svg)]()

	> Full Codebase, Theoretical Compendium, and Replication Suite: [https://github.com/MtnMCG/PCEI](https://github.com/MtnMCG/PCEI)

	---

	## 🏛️ Executive Summary

	PCEI-Text is an open-weights linear-time sequence foundation model family that discards quadratic Transformer attention ($\mathcal{O}(L^2)$) and high-energy 16-bit floating-point multiplications in favor of:
	1. Hardware-efficient Selective State Spaces (Mamba SSM) operating with linear $\mathcal{O}(L)$ compute and constant $\mathcal{O}(1)$ memory inference.
	2. Canonical Microsoft BitNet b1.58 Ternary Quantization, constraining synaptic weights to strictly discrete $\{-1.0, 0.0, +1.0\}$ states and replacing energy-hungry floating-point multiplications ($FMUL$) with integer additions ($IADD$).
	3. Kolmogorov-Arnold Network (FastKAN) Spline Manifolds, utilizing learnable cubic B-splines along non-linear parameter trajectories.
	4. Dual-Timescale Dynamic State Modulation (`AmbientField`), preventing context degradation and drifting attractors over extended multi-turn horizons.

	### Checkpoints Available in this Repository

	\| Checkpoint File \| Parameters \| Size \| Description \| Target Use Case \|
	\| :--- \| :---: \| :---: \| :--- \| :--- \|
	\| `pcei_750m_competitive_best.pt` \| 750M \| 1.77 GB \| Flagship Foundation Model. Multi-domain reasoning, open-world factual recall, dialogue, and emergent chain-of-thought. \| Primary foundation deployment & research \|
	\| `pcei_100m_bitnet_ternary.pt` \| 100M \| 1.09 GB \| Micro-scale edge model. Canonical BitNet b1.58 ternary baseline. \| Resource-constrained edge & mobile devices \|

	---

	## 🔬 Empirical Audits & Verification

	### 1. PCEI 750M Multi-Domain Benchmark Performance
	Evaluated under zero-leakage, greedy argmax neural generation:

	\| Domain \| 100-Question Multi-Domain Benchmark \|
	\| :--- \| :---: \|
	\| 1. Open Factual QA (40 Questions) \| 95.0% - 100.0% \|
	\| 2. Conversational Dialogue (30 Questions) \| 86.7% - 93.3% \|
	\| 3. Deterministic Arithmetic & CoT (30 Questions) \| 70.0% - 80.0% \|
	\| Overall Benchmark Accuracy \| 89.0% - 92.0% \|

	### 2. Tiny LM Compute-Matched Bake-Off (Muse Autonomous Evaluation)
	In an independent multi-seed compute-matched baseline bake-off conducted via the Muse evaluation agent against a compute-matched Transformer:

	\| Model \| Architecture \| Parameters \| Val Loss (mean ± std) \| Val Perplexity \| Win Rate \|
	\| :--- \| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| Transformer \| Vanilla Decoder (5L, 4H, FFN $2\times d$ GELU) \| 398,688 \| 4.6131 ± 0.0003 \| 100.8 \| Baseline \|
	\| Dense Mamba \| Repo `MambaBlock` (6L, FP32 dense weights) \| 407,232 \| 4.3881 ± 0.0009 \| 80.5 \| 0 / 2 vs PCEI \|
	\| PCEI-tiny \| Strict 1.58-bit Ternary (`{-1, 0, +1}`) + FastKAN \| 411,912 \| 4.3783 ± 0.0073 \| 79.1 \| 2 / 2 (Won all seeds) \|

	* Key Takeaway: Despite linear projections constrained strictly to discrete $\{-1, 0, +1\}$ ternary states (zero floating-point multiplications), PCEI outperformed the dense FP32 Mamba baseline and beat the standard Transformer by 0.23 nats (val perplexity 79.1 vs 100.8).

	### 3. Strict Microsoft BitNet b1.58 Migration
	An external audit in October 2026 disclosed that early internal prototypes utilized an experimental 5-state quantizer. The architecture was fully refactored to canonical Microsoft BitNet b1.58:
	- Parameter Entropy Cut: Reduced from 2.32b to strictly 1.58b ($\le \log_2(3)$ bits/param), cutting parameter entropy by 32%.
	- Perplexity & Hallucinations: Validation loss improved from 11.1696 to 11.1429 on 100M, eliminating activation outlier code token hallucinations.
	- Full scientific post-mortem report: [`AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md`](https://github.com/MtnMCG/PCEI/blob/main/pcei_framework/docs/AUDIT_REPORT_TERNARY_QUANTIZATION_POSTMORTEM.md).

	---

	## ⚙️ Model Specifications

	\| Architectural Dimension \| PCEI-750M (Flagship Foundation) \| PCEI-100M (Edge) \|
	\| :--- \| :--- \| :--- \|
	\| Base Architecture \| Selective State Space (Mamba SSM) + Latent Diffusion \| 12-Layer Selective State Space (Mamba SSM) \|
	\| Model Dimension ($d_{\text{model}}$) \| 1024 \| 768 \|
	\| Inner Expansion Dimension ($d_{\text{inner}}$) \| 2048 \| 512 \|
	\| State Dimension ($d_{\text{state}}$) \| 32 \| 16 \|
	\| Convolution Kernel Size \| 4 \| 4 \|
	\| Non-Linear Branch \| FastKAN (Cubic B-splines) \| FastKAN (Cubic B-splines) \|
	\| Synaptic Quantization \| Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE \| Strict BitNet b1.58 ($\{-1.0, 0.0, +1.0\}$) via STE \|
	\| Vocabulary Size \| 50,257 (OpenAI GPT-2 BPE) \| 50,257 (OpenAI GPT-2 BPE) \|
	\| Context Length ($L$) \| Up to 2,048 tokens \| 128 / 512 tokens \|
	\| Hardware Target \| Apple Silicon MPS, NVIDIA CUDA, CPU edge \| Apple Silicon MPS, NVIDIA CUDA, CPU edge \|

	---

	## 🚀 Local Inference Quickstart

	To run local text generation using the PCEI framework:

	```bash
	# 1. Clone the master GitHub repository
	git clone https://github.com/MtnMCG/PCEI.git
	cd PCEI/pcei_framework

	# 2. Set up virtual environment and install dependencies
	python3 -m venv venv
	source venv/bin/activate
	pip install -r requirements.txt
	pip install huggingface_hub

	# 3. Download the model checkpoint directly from Hugging Face
	python3 -c "
	from huggingface_hub import hf_hub_download
	# Download Flagship 750M Foundation Model (or 'pcei_100m_bitnet_ternary.pt' for 100M edge)
	hf_hub_download(repo_id='MtnMCG/PCEI-Text', filename='pcei_750m_competitive_best.pt', local_dir='checkpoints')
	"

	# 4. Interactive generation with continuous state modulation
	python3 tools/resonance_chat.py \
	--checkpoint checkpoints/pcei_750m_competitive_best.pt \
	--device mps # Use --device cuda on NVIDIA GPUs
	```

	### Python API Usage

	```python
	import torch
	from src.pcei import PCEINexus
	from src.config import PCEIConfig
	from src.tokenizer import PCEITokenizer

	device = torch.device("mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu"))

	# Load checkpoint (750M Foundation Model)
	checkpoint = torch.load("checkpoints/pcei_750m_competitive_best.pt", map_location=device)
	cfg_obj = checkpoint.get("config", None)
	config = cfg_obj if isinstance(cfg_obj, PCEIConfig) else PCEIConfig.load()

	# Initialize model
	nexus = PCEINexus(config=config).to(device)
	nexus.load_state_dict(checkpoint["model_state_dict"], strict=False)
	nexus.eval()

	# Tokenize prompt
	tokenizer = PCEITokenizer(vocab_size=config.vocab_size)
	prompt = "USER: Explain why ice floats on water.\nASSISTANT:"
	input_ids = torch.tensor(tokenizer.encode_raw(prompt), dtype=torch.long).unsqueeze(0).to(device)

	# Autoregressive generation
	with torch.no_grad():
	for _ in range(64):
	outputs = nexus({"text": input_ids})
	logits = outputs["logits"] if isinstance(outputs, dict) else outputs
	next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
	input_ids = torch.cat([input_ids, next_token], dim=-1)

	print(tokenizer.decode(input_ids[0].tolist()))
	```

	---

	## 🤖 Sister Repositories & Embodied Intelligence

	PCEI proves that physical constraints generalize across domains:
	- 🏃 [MtnMCG/PCEI-Robotics](https://huggingface.co/MtnMCG/PCEI-Robotics): Autonomous American Ninja Warrior 18-meter humanoid controller traversing 45° angled Quad Steps without attention or deep RL.
	- 🏛️ [PCEI Master GitHub Repository](https://github.com/MtnMCG/PCEI): Contains the 11 theoretical research papers, continuous training daemons, curriculum streaming pipelines, and full verification suite.

	---

	## 📜 Citation & License

	This model and the PCEI codebase are released under the MIT Non-Commercial License (MIT-NC). Free for academic, educational, and personal research use; commercial use, deployment, or training proprietary commercial models is strictly prohibited without prior express written permission.

	```bibtex
	@article{ramirez2026pcei,
	title = {Physically Constrained Emergent Intelligence: Subverting Transformer Hyperscaling via Linear State Spaces, Ternary Synaptic Gating, and Latent Generative Replay},
	author = {Ramirez, Emerson and Antigravity Assistant},
	journal = {PCEI Theoretical Research Compendium},
	year = {2026},
	url = {https://github.com/MtnMCG/PCEI}
	}
	```