Refract-Labs/Orion-Flagship-Mini-Base

🔺 Orion Flagship

Orion Flagship is an experimental Project Prism model using the Orion T2 architecture.

T2 separates parts of the computation into a Universal Cortex, shared sparse Procedure Banks, a sparse Knowledge Vault, persistent Working State, and a cheap recurrent deliberation pass.

This repository contains an experimental pretraining checkpoint and custom Transformers integration. It is not instruction tuned.

⚠️ Access & Usage Restrictions

• No use of the Orion T2 architecture for your own models. The architecture, its code, and its design are not licensed for building, training, or deriving other models. You can howvever make your own models and publish privately under this organisation.
• Private repo only. This repository must stay private under the Project Prism organization. It is for Project Prism members only.
• No outsider access (for now). Do not share, mirror, fork, or redistribute the weights or code outside Project Prism.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "Project-Prism/Orion-Flagship-Mini-Preview"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    device_map="auto",
)

The current T2 v1 compatibility layer deliberately uses use_cache=False. Generation therefore recomputes the full prefix each token so the recurrent deliberation + Working State graph remains faithful to training.

🧠 Architecture

Transformer 2 (T2)

Transformer 2 (T2) is an experimental language-model architecture developed under Project Prism.

The central idea behind T2 is to separate several roles that are normally entangled inside the parameters and hidden activations of a conventional Transformer.

Instead of relying on a single dense stack for knowledge storage, reusable computation, and temporary reasoning state, T2 introduces dedicated systems for each role:

KNOW → DO → THINK → VERIFY

🧠 Universal Cortex

The Universal Cortex is the main sequence-processing backbone of Orion.

Mini-T2 uses 12 Cortex stages with a hidden dimension of 768. Each stage contains attention and a routed Procedure Bank feed-forward path, using 12 query heads and 4 KV heads.

The Cortex is responsible for integrating:

• token context,
• retrieved knowledge,
• routed procedures,
• persistent Working State,
• and recurrent deliberation.

Unlike a standard Transformer where every layer owns an independent FFN, groups of Cortex stages access shared Procedure Banks.

⚙️ Procedure Banks

T2 separates reusable transformations into shared sparse Procedure Banks.

Mini-T2 contains:

• 3 shared Procedure Banks
• 16 experts per bank
• Top-2 routed experts
• 1 shared expert

Each bank is reused across a group of Cortex stages. A sparse router selects the most relevant procedures for each token while the shared expert provides a common computation path.

The goal is to let the model learn reusable computational procedures without requiring every Cortex stage to independently store its own complete set of feed-forward transformations.

📚 Knowledge Vault

The Knowledge Vault provides a dedicated sparse memory system for declarative information.

Mini-T2 contains 65,536 memory slots and uses product-key retrieval.

For each retrieval:

1. The query searches the two product-key components.
2. The top 12 candidates from each side form 144 candidate addresses.
3. The best 4 memories are retrieved.
4. Retrieved information is gated into the Cortex.

Knowledge interfaces are used at approximately stages 3, 7, and 11.

This creates a separate pathway through which the model can retrieve stored information instead of requiring all declarative knowledge to be represented only through the main computational weights.

📝 Working State

T2 also maintains a separate persistent causal Working State.

Working State acts as temporary computational memory. It can be read by the Cortex and updated as the sequence is processed.

Updates use a learned gated replacement/blending mechanism rather than unbounded additive accumulation.

In Mini-T2, Working State interacts with the Cortex around the same major interface points used by the Knowledge Vault.

This gives T2 an explicit distinction between:

• stored knowledge — Knowledge Vault,
• reusable computation — Procedure Banks,
• current computation — Working State.

🔁 Deliberation

After the normal 12-stage Cortex pass, Mini-T2 performs an additional deliberation pass by reusing the final three Cortex stages:

9 → 10 → 11

This provides additional computation without repeating the entire network.

The resulting representation is passed through the final RMSNorm and language model head to produce the next-token distribution.

📐 Mini-T2 v1 Configuration

Component	Configuration
Total parameters	~219M
Approx. active-path parameters	~142M
Cortex stages	12
Hidden dimension	768
Attention	12 Q / 4 KV heads
Procedure Banks	3 shared banks
Experts per bank	16
Expert routing	Top-2 + shared expert
Knowledge Vault	65,536 slots
Product-key candidates	12 × 12 = 144
Retrieved memories	Top-4
Context length	1,024 tokens
Deliberation	Repeat stages 9 → 10 → 11

🧪 Research Status

T2 is an experimental research architecture, not an established replacement for the Transformer.

The current Mini-T2 checkpoint demonstrates that the architecture can be pretrained stably and produce coherent language. Internal ablation experiments also indicate that some of its architectural components materially affect model behaviour.

However, controlled comparisons against parameter- and training-matched standard Transformer baselines are still required before making claims about relative efficiency or capability.