README.md

---
library_name: transformers
pipeline_tag: text-generation
license: other
license_name: orion-t2-research
tags:
- project-prism
- orion
- custom-code
- causal-lm
---

# 🔺 Orion Flagship

**Orion Flagship** is an experimental Project Prism model using the Orion T2
architecture.

T2 separates parts of the computation into a Universal Cortex, shared sparse
Procedure Banks, a sparse Knowledge Vault, persistent Working State, and a
cheap recurrent deliberation pass.

This repository contains an experimental pretraining checkpoint and custom
Transformers integration. It is **not instruction tuned**.

## ⚠️ Access & Usage Restrictions

- **No use of the Orion T2 architecture for your own models.** The
  architecture, its code, and its design are not licensed for building,
  training, or deriving other models. You can howvever make your own models and publish privately under this organisation.
- **Private repo only.** This repository must stay private under the
  Project Prism organization. It is for Project Prism members only.
- **No outsider access (for now).** Do not share, mirror, fork, or
  redistribute the weights or code outside Project Prism.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "Project-Prism/Orion-Flagship-Mini-Preview"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    device_map="auto",
)
```

The current T2 v1 compatibility layer deliberately uses `use_cache=False`.
Generation therefore recomputes the full prefix each token so the recurrent
deliberation + Working State graph remains faithful to training.

## 🧠 Architecture

![Orion T2 Architecture](./assets/orion-t2-architecture.png)

### Transformer 2 (T2)

**Transformer 2 (T2)** is an experimental language-model architecture developed
under Project Prism.

The central idea behind T2 is to separate several roles that are normally
entangled inside the parameters and hidden activations of a conventional
Transformer.

Instead of relying on a single dense stack for knowledge storage, reusable
computation, and temporary reasoning state, T2 introduces dedicated systems
for each role:

**KNOW → DO → THINK → VERIFY**

### 🧠 Universal Cortex

The **Universal Cortex** is the main sequence-processing backbone of Orion.

Mini-T2 uses **12 Cortex stages** with a hidden dimension of **768**. Each
stage contains attention and a routed Procedure Bank feed-forward path, using
12 query heads and 4 KV heads.

The Cortex is responsible for integrating:

- token context,
- retrieved knowledge,
- routed procedures,
- persistent Working State,
- and recurrent deliberation.

Unlike a standard Transformer where every layer owns an independent FFN,
groups of Cortex stages access shared Procedure Banks.

### ⚙️ Procedure Banks

T2 separates reusable transformations into **shared sparse Procedure Banks**.

Mini-T2 contains:

- **3 shared Procedure Banks**
- **16 experts per bank**
- **Top-2 routed experts**
- **1 shared expert**

Each bank is reused across a group of Cortex stages. A sparse router selects
the most relevant procedures for each token while the shared expert provides
a common computation path.

The goal is to let the model learn reusable computational procedures without
requiring every Cortex stage to independently store its own complete set of
feed-forward transformations.

### 📚 Knowledge Vault

The **Knowledge Vault** provides a dedicated sparse memory system for
declarative information.

Mini-T2 contains **65,536 memory slots** and uses product-key retrieval.

For each retrieval:

1. The query searches the two product-key components.
2. The top **12 candidates from each side** form **144 candidate addresses**.
3. The best **4 memories** are retrieved.
4. Retrieved information is gated into the Cortex.

Knowledge interfaces are used at approximately **stages 3, 7, and 11**.

This creates a separate pathway through which the model can retrieve stored
information instead of requiring all declarative knowledge to be represented
only through the main computational weights.

### 📝 Working State

T2 also maintains a separate **persistent causal Working State**.

Working State acts as temporary computational memory. It can be read by the
Cortex and updated as the sequence is processed.

Updates use a learned gated replacement/blending mechanism rather than
unbounded additive accumulation.

In Mini-T2, Working State interacts with the Cortex around the same major
interface points used by the Knowledge Vault.

This gives T2 an explicit distinction between:

- **stored knowledge** — Knowledge Vault,
- **reusable computation** — Procedure Banks,
- **current computation** — Working State.

### 🔁 Deliberation

After the normal 12-stage Cortex pass, Mini-T2 performs an additional
**deliberation pass** by reusing the final three Cortex stages:

`9 → 10 → 11`

This provides additional computation without repeating the entire network.

The resulting representation is passed through the final RMSNorm and language
model head to produce the next-token distribution.

### 📐 Mini-T2 v1 Configuration

| Component | Configuration |
|---|---:|
| Total parameters | **~219M** |
| Approx. active-path parameters | **~142M** |
| Cortex stages | **12** |
| Hidden dimension | **768** |
| Attention | **12 Q / 4 KV heads** |
| Procedure Banks | **3 shared banks** |
| Experts per bank | **16** |
| Expert routing | **Top-2 + shared expert** |
| Knowledge Vault | **65,536 slots** |
| Product-key candidates | **12 × 12 = 144** |
| Retrieved memories | **Top-4** |
| Context length | **1,024 tokens** |
| Deliberation | **Repeat stages 9 → 10 → 11** |

### 🧪 Research Status

T2 is an **experimental research architecture**, not an established
replacement for the Transformer.

The current Mini-T2 checkpoint demonstrates that the architecture can be
pretrained stably and produce coherent language. Internal ablation experiments
also indicate that some of its architectural components materially affect
model behaviour.

However, controlled comparisons against parameter- and training-matched
standard Transformer baselines are still required before making claims about
relative efficiency or capability.