brick-sysone-V1 / README.md
massaindustries's picture
docs: add install and terminal chat quickstart (#1)
074d7cf
|
Raw History Blame Contribute Delete
4.31 kB
---
license: apache-2.0
base_model: answerdotai/ModernBERT-large
library_name: transformers
pipeline_tag: text-classification
tags:
- text-classification
- modernbert
- llm-routing
- capability-vector
- system-one
task_categories:
- text-classification
language:
- en
---
# Brick SystemOne Large v2 — query to capability vector
Fine-tuned `answerdotai/ModernBERT-large` (8192-token context) that maps a raw
query to a six-dimensional capability vector. Each dimension is an independent
sigmoid output in `[0, 1]`; the vector is neither normalized to sum to one nor
rounded.
This checkpoint is the local router model used by
[Brick](https://github.com/regolo-ai/brick-SR1).
## Output dimensions
The output order is fixed:
1. `instruction_following`
2. `coding`
3. `math_reasoning`
4. `world_knowledge`
5. `planning_agentic`
6. `creative_synthesis`
## Training
- Base: `answerdotai/ModernBERT-large`, 28 layers, hidden size 1024, 8192 context.
- Data: 68,882 query/label pairs. Labels are cross-model judge means with
difficulty modulation; v2 adds 16k pair-transfer rows (short queries inherit
detailed labels, including 3.1k ultra-short queries under 40 characters) and
3k garbage rows mapped to near-zero vectors.
- Configuration: LR 8.93e-5, 4 epochs, gradient accumulation 8, bf16, SDPA.
v2 differences from v1: the brevity gap on short prompts drops from 0.27 to
0.13, and the garbage mean-max drops from 0.36 to 0.006 with no spikes.
## Evaluation
- Holdout (4,968 rows): pearson_macro **0.9656**, mae 0.058.
- Mapping battery (30 P1, 6 P2, 40 garbage): P1 gap 0.13, P2 direction 6/6.
## Quickstart
The output is a capability profile, not a distribution over models: each value
is an independent sigmoid in `[0, 1]`.
### Install
```bash
python3 -m venv .venv-systemone
.venv-systemone/bin/python -m pip install --upgrade pip
.venv-systemone/bin/python -m pip install --index-url https://download.pytorch.org/whl/cu128 \
torch==2.11.0
.venv-systemone/bin/python -m pip install transformers==5.20.2 safetensors==0.8.0 \
tokenizers==0.23.3 numpy==1.26.4 huggingface-hub==2.2.0
```
Use `https://download.pytorch.org/whl/cu126` for driver 560-569, or
`https://download.pytorch.org/whl/cpu` for CPU-only. The script below picks
BF16 on CUDA and FP32 on CPU, as recommended.
### Terminal chat
Type a query, read the vector. Save as `chat.py` and run
`.venv-systemone/bin/python chat.py`.
```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
LABELS = ["instruction_following", "coding", "math_reasoning",
"world_knowledge", "planning_agentic", "creative_synthesis"]
REVISION = "9263a397a366ee22921214ddeebab1f5e4e8facd"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
model = AutoModelForSequenceClassification.from_pretrained(
"regolo/brick-sysone-V1", revision=REVISION, dtype=dtype,
attn_implementation="sdpa",
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained("regolo/brick-sysone-V1", revision=REVISION)
limit = min(model.config.max_position_embeddings, 8192)
while True:
try:
query = input("query> ").strip()
except (EOFError, KeyboardInterrupt):
break
if not query:
continue
encoded = tokenizer(query, return_tensors="pt", truncation=True, max_length=limit).to(device)
with torch.inference_mode():
vector = model(**encoded).logits[0].sigmoid().float().tolist()
for label, value in zip(LABELS, vector):
print(f" {label:<20} {value:.4f}")
print(f" vector: [{', '.join(f'{v:.4f}' for v in vector)}]")
```
```
query> write a python parser that streams csv rows
instruction_following 0.5117
coding 0.9258
math_reasoning 0.0903
world_knowledge 0.1157
planning_agentic 0.0630
creative_synthesis 0.0198
vector: [0.5117, 0.9258, 0.0903, 0.1157, 0.0630, 0.0198]
```
Input is truncated at 8192 tokens; longer inputs are outside the training
distribution. The Brick router classifies only the last user message, never
system or assistant turns. Full scripts with `--json`, `--device` and one-shot
mode: `examples/systemone/` in the Brick repository.
## License
Apache-2.0, matching the base model. See `LICENSE`.