You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Orion T2 Mini is an experimental research model developed by Refract AI Labs, a sub-branch of SmilyAI.
Please provide the information below to request access. Your responses will be used to review and manage access to the model.
Information submitted through this form may be used by Refract AI Labs to contact you regarding your access request or important model-related updates. Please do not include sensitive personal information.
Log in or Sign Up to review the conditions and access this model content.
🔺 Orion T2 Mini
The first major model release from Refract AI Labs.
Orion T2 Mini is a compact experimental language model based on Project Prism's Orion T2 architecture.
Developed by Refract AI Labs, a research sub-branch of SmilyAI, Orion T2 Mini explores how far a small language model can be pushed using an unconventional architecture, sparse specialist components, persistent working state, and recurrent computation.
This checkpoint is the supervised fine-tuned assistant release of Orion T2 Mini.
🚀 Release
| Orion T2 Mini | |
|---|---|
| Organisation | Refract AI Labs / SmilyAI |
| Architecture | Orion T2 |
| Model type | Decoder-only causal language model |
| Parameters | ~219M |
| Training stage | Pretraining → Response-only SFT |
| SFT examples | 700,000 |
| Primary use | Conversational text generation |
| Framework | Hugging Face Transformers |
| Custom code | Required |
| Generation cache | Currently disabled |
Orion T2 Mini is the first major public model release from Refract AI Labs and the first assistant release representing the Orion T2 generation.
🧠 Architecture
Orion T2 Mini is not a standard dense Transformer.
The architecture combines several experimental components:
Universal Cortex
A 12-stage Universal Cortex performs the model's main sequence processing.
Procedure Banks
Shared sparse Procedure Banks provide specialist computational pathways that can be reused across processing stages.
Knowledge Vault
A sparse Knowledge Vault provides an additional mechanism for storing and retrieving learned representations.
Working State
Orion maintains an explicit Working State intended to provide persistent intermediate information during processing.
Recurrent Deliberation
Later processing stages can perform recurrent computation, allowing additional internal processing without simply increasing the number of unique Transformer layers.
These mechanisms are experimental. Their usefulness should be determined through controlled evaluation rather than inferred from architectural complexity alone.
📚 Training
The released assistant checkpoint begins from the completed Orion T2 Mini base checkpoint.
It was subsequently trained using response-only supervised fine-tuning (SFT) on approximately 700,000 examples.
The SFT mixture was:
| Category | Share |
|---|---|
| General / Conversation | 60% |
| Mathematics | 20% |
| Code | 20% |
Only assistant responses contribute to the SFT training objective.
This allows Orion to learn conversational behaviour without directly optimizing against the user and system portions of each training example.
💬 Chat Format
Use the bundled Hugging Face chat template.
Orion was trained using the literal text role markers:
<|system|>
<|user|>
<|assistant|>
These markers were not added as tokenizer special tokens. They are represented as ordinary tokenized text.
Example
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "Project-Prism/Orion-Flagship-Mini-SFT"
tokenizer = AutoTokenizer.from_pretrained(
repo,
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
repo,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="auto",
)
messages = [
{
"role": "system",
"content": "You are Orion, a helpful AI assistant."
},
{
"role": "user",
"content": "Explain why the sky is blue."
},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(
prompt,
return_tensors="pt",
).to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
temperature=0.7,
top_p=0.95,
do_sample=True,
use_cache=False,
)
new_tokens = output[
0,
inputs["input_ids"].shape[1]:
]
print(
tokenizer.decode(
new_tokens,
skip_special_tokens=True,
)
)
⚡ Inference Notes
Orion T2 uses a custom Transformers compatibility layer and therefore requires:
trust_remote_code=True
The current implementation uses:
use_cache=False
As a result, generation currently recomputes the prefix for every generated token.
This can make inference substantially slower than similarly sized conventional Transformer models using an optimized KV cache.
The current implementation should therefore be considered a research implementation rather than a fully optimized inference runtime.
📊 Evaluation
Orion T2 Mini is intended to be evaluated across:
- General language understanding
- Instruction following
- Reasoning
- Mathematics
- Coding
- Long-context behaviour
- Knowledge
- Small-language-model benchmarks
Leaderboard and benchmark results should use clearly documented evaluation configurations wherever possible.
Results from different prompting formats, quantization levels, context lengths, evaluation harnesses, or few-shot settings should not automatically be treated as directly comparable.
⚠️ Experimental Status
Orion T2 Mini is an experimental research model.
The architecture intentionally differs from a conventional Transformer, but architectural novelty alone does not establish improved capability or efficiency.
Claims about the benefits of the Universal Cortex, Procedure Banks, Knowledge Vault, Working State, or recurrent deliberation should be tested through matched evaluations and ablations.
The model may:
- Hallucinate incorrect information
- Produce incorrect mathematical reasoning
- Generate broken or insecure code
- Misunderstand instructions
- Produce unexpected outputs
- Inherit biases or undesirable behaviours from its training data
Outputs should not be treated as inherently factual or reliable.
🔬 Project Prism
Orion is developed as part of Project Prism, an experimental model-development effort exploring alternative architectures and training approaches for compact language models.
Orion T2 Mini represents the first major release of the T2 generation from Refract AI Labs.
🔺 Orion T2
Small model. Experimental architecture. Built to find out what actually works.
- Downloads last month
- -
Model tree for Refract-Labs/Orion-Flagship-Mini
Base model
Project-Prism/Orion-Flagship-Mini-Base