--- license: apache-2.0 base_model: answerdotai/ModernBERT-large library_name: transformers pipeline_tag: text-classification tags: - text-classification - modernbert - llm-routing - capability-vector - system-one task_categories: - text-classification language: - en --- # Brick SystemOne Large v2 — query to capability vector Fine-tuned `answerdotai/ModernBERT-large` (8192-token context) that maps a raw query to a six-dimensional capability vector. Each dimension is an independent sigmoid output in `[0, 1]`; the vector is neither normalized to sum to one nor rounded. This checkpoint is the local router model used by [Brick](https://github.com/regolo-ai/brick-SR1). ## Output dimensions The output order is fixed: 1. `instruction_following` 2. `coding` 3. `math_reasoning` 4. `world_knowledge` 5. `planning_agentic` 6. `creative_synthesis` ## Training - Base: `answerdotai/ModernBERT-large`, 28 layers, hidden size 1024, 8192 context. - Data: 68,882 query/label pairs. Labels are cross-model judge means with difficulty modulation; v2 adds 16k pair-transfer rows (short queries inherit detailed labels, including 3.1k ultra-short queries under 40 characters) and 3k garbage rows mapped to near-zero vectors. - Configuration: LR 8.93e-5, 4 epochs, gradient accumulation 8, bf16, SDPA. v2 differences from v1: the brevity gap on short prompts drops from 0.27 to 0.13, and the garbage mean-max drops from 0.36 to 0.006 with no spikes. ## Evaluation - Holdout (4,968 rows): pearson_macro **0.9656**, mae 0.058. - Mapping battery (30 P1, 6 P2, 40 garbage): P1 gap 0.13, P2 direction 6/6. ## Quickstart The output is a capability profile, not a distribution over models: each value is an independent sigmoid in `[0, 1]`. ### Install ```bash python3 -m venv .venv-systemone .venv-systemone/bin/python -m pip install --upgrade pip .venv-systemone/bin/python -m pip install --index-url https://download.pytorch.org/whl/cu128 \ torch==2.11.0 .venv-systemone/bin/python -m pip install transformers==5.20.2 safetensors==0.8.0 \ tokenizers==0.23.3 numpy==1.26.4 huggingface-hub==2.2.0 ``` Use `https://download.pytorch.org/whl/cu126` for driver 560-569, or `https://download.pytorch.org/whl/cpu` for CPU-only. The script below picks BF16 on CUDA and FP32 on CPU, as recommended. ### Terminal chat Type a query, read the vector. Save as `chat.py` and run `.venv-systemone/bin/python chat.py`. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer LABELS = ["instruction_following", "coding", "math_reasoning", "world_knowledge", "planning_agentic", "creative_synthesis"] REVISION = "9263a397a366ee22921214ddeebab1f5e4e8facd" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 model = AutoModelForSequenceClassification.from_pretrained( "regolo/brick-sysone-V1", revision=REVISION, dtype=dtype, attn_implementation="sdpa", ).to(device).eval() tokenizer = AutoTokenizer.from_pretrained("regolo/brick-sysone-V1", revision=REVISION) limit = min(model.config.max_position_embeddings, 8192) while True: try: query = input("query> ").strip() except (EOFError, KeyboardInterrupt): break if not query: continue encoded = tokenizer(query, return_tensors="pt", truncation=True, max_length=limit).to(device) with torch.inference_mode(): vector = model(**encoded).logits[0].sigmoid().float().tolist() for label, value in zip(LABELS, vector): print(f" {label:<20} {value:.4f}") print(f" vector: [{', '.join(f'{v:.4f}' for v in vector)}]") ``` ``` query> write a python parser that streams csv rows instruction_following 0.5117 coding 0.9258 math_reasoning 0.0903 world_knowledge 0.1157 planning_agentic 0.0630 creative_synthesis 0.0198 vector: [0.5117, 0.9258, 0.0903, 0.1157, 0.0630, 0.0198] ``` Input is truncated at 8192 tokens; longer inputs are outside the training distribution. The Brick router classifies only the last user message, never system or assistant turns. Full scripts with `--json`, `--device` and one-shot mode: `examples/systemone/` in the Brick repository. ## License Apache-2.0, matching the base model. See `LICENSE`.