smoll-swarm/SmollExtractor-1.7B

🐝 SMOL EXTRACTOR 1.7B

Deterministic JSON Schema Extraction Engine for local multi-agent systems

1.71B Parameters · Native JSON Format · Lossless Schema Mapping · Local-First Deployment

---

📌 Overview

SmolExtractor-1.7B is a specialized causal language model fine-tuned for deterministic, structured schema extraction from messy, unstructured text. Built as a core worker node for the Smol Swarm local multi-agent architecture, the model forces the extraction of raw, valid JSON conforming exactly to a user-provided schema.

Unlike general-purpose models, SmolExtractor is explicitly trained to eliminate conversational artifacts (e.g., "Sure! Here is your JSON"), ignore output formatting padding, and generate typed properties (strings, integers, floats, booleans) without wrapping the response in markdown backticks.

🎯 Target Domains and Supported Schemas

Target Domain	Typical Input	Target Output Format
Invoices & Receipts	Billing text, taxes, transaction IDs, statuses	{"invoice_id": "string", "vendor": "string", "amount": "number", "tax": "number", "status": "string"}
System & Security Logs	System dumps, warnings, stack traces	{"level": "string", "service": "string", "status_code": "number", "error_message": "string", "retry_count": "number"}
E-Commerce Logistics	Purchase records, customer notes, item quantities	{"order_number": "number", "buyer": "string", "quantity": "number", "item": "string", "is_expedited": "boolean"}
Ticketing Systems	Customer complaints, prioritize, emails	{"ticket_id": "string", "requester_email": "string", "priority": "string", "subject": "string", "is_urgent": "boolean"}
Vitals & Clinical Data	Clinical summaries, doctor observations	{"name": "string", "age": "number", "bp": "string", "pulse_bpm": "number", "condition": "string"}

---

⚡ Quick Start

Installation

pip install transformers torch accelerate

Python Inference Script

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "smoll-swarm/SmolExtractor-1.7B"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto"
)

SYSTEM_PROMPT = """You are SmolExtractor, a deterministic data extraction engine.
Extract data from the provided text to strictly match the requested JSON schema.
Output ONLY a raw, valid JSON object matching the exact keys and types. Do not output markdown, code blocks, or explanations."""

raw_text = "URGENT: Invoice INV-9821 issued by Stripe Inc to Alice Smith on 2025-04-18. Total due: $1820.50, Tax: $85.00. Status: PAID."

schema = {
    "invoice_id": "string",
    "vendor": "string",
    "customer": "string",
    "amount": "number",
    "tax": "number",
    "is_urgent": "boolean",
    "status": "string"
}

user_content = f"Text: \"{raw_text}\"\nSchema: {json.dumps(schema)}"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_content}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=False,  # Enforce deterministic greedy decoding
        pad_token_id=tokenizer.eos_token_id
    )

output_ids = outputs[0][inputs.input_ids.shape[1]:]
extracted_json = tokenizer.decode(output_ids, skip_special_tokens=True).strip()

result = json.loads(extracted_json)
print(result)
# Output:
# {'invoice_id': 'INV-9821', 'vendor': 'Stripe Inc', 'customer': 'Alice Smith', 'amount': 1820.5, 'tax': 85.0, 'is_urgent': True, 'status': 'PAID'}

---

📊 Input / Output Protocol

Raw ChatML Prompt Structure

<|im_start|>system
You are SmolExtractor, a deterministic data extraction engine.
Extract data from the provided text to strictly match the requested JSON schema.
Output ONLY a raw, valid JSON object matching the exact keys and types. Do not output markdown, code blocks, or explanations.<|im_end|>
<|im_start|>user
Text: "Order #84210 placed by Tony Stark. Items: 4x 'Mechanical Keyboard'. Expedited shipping: true. Destination City: 'Austin'."
Schema: {"order_number": "number", "buyer": "string", "quantity": "number", "item": "string", "is_expedited": "boolean", "city": "string"}<|im_end|>
<|im_start|>assistant

Deterministic Response

{"order_number": 84210, "buyer": "Tony Stark", "quantity": 4, "item": "Mechanical Keyboard", "is_expedited": true, "city": "Austin"}<|im_end|>

---

🏗️ Swarm Integration

In production, SmolExtractor-1.7B executes requests routed by SmolRouter-135M in a pipeline:

                        Incoming User Request
                                 │
                                 ▼
                       ┌──────────────────┐
                       │ SmolRouter-135M  │  ~2ms latency classification
                       └─────────┬────────┘
                                 │
             ┌───────────────────┴───────────────────┐
             ▼ route == 'extractor'                  ▼ other route
 ┌───────────────────────┐               ┌───────────────────────┐
 │ SmolExtractor-1.7B    │               │  Other Swarm Agent    │
 │ (JSON Parser Engine)  │               │ (Summarizer/Coder/etc)│
 └───────────┬───────────┘               └───────────────────────┘
             │
             ▼
    Strict Typed JSON

---

🛠️ Training Configurations

The model was trained using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA), and adapters were fused back to baseline weights prior to serialization.

• Dataset Size: 1,500 balanced data extraction conversations across 5 high-priority verticals.
• Masking Objective: Loss calculated strictly on target tokens; system prompt, input text, and target schemas are masked from the loss calculation using a -100 label bias.
• LoRA Configuration: $r=16, \alpha=32$, dropout $= 0.05$.
• Target Projections: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (all linear modules of the self-attention block and MLP layers).
• Optimization Strategy: AdamW ($\beta_1=0.9, \beta_2=0.999$, weight decay $= 0.01$).
• Learning Rate: $2 \times 10^{-4}$ with a cosine schedule and 20 warmup steps.
• Precision Parameters: 16-bit Automatic Mixed Precision (AMP) with torch gradient scaling.
• Convergence Loss: Final training loss settled at ~0.00 across 3 epochs.

---

⚙️ Model Specifications

Parameter	Metric
Tuned Parameters	1,729,464,320
Trainable LoRA Weight %	1.04% (18,087,936 parameters)
Context Length Window	2,048 tokens
Output Type	Strict JSON (Greedy Generation)
License Agreement	Apache 2.0

---

⚠️ Limitations & Usage Guidelines

1. Greedy Sampling Recommendation: To ensure strict alignment with formatting rules, generation parameters must disable sampling: do_sample=False or temperature=0.0.
2. Schema Integrity: The model relies on valid datatypes (string, number, boolean, array) in the schema parameter. Highly complex, deeply nested, custom objects may occasionally experience schema misalignment.
3. Language Context: The fine-tuning target dataset consisted exclusively of English prompts. Performance on other languages is uncalibrated.

---

📚 Citation

@misc{smolextractor2025,
  title   = {SmolExtractor-1.7B: A Deterministic Schema-Constrained Extraction Model},
  author  = {SmolSwarm Contributors},
  year    = {2026},
  url     = {https://huggingface.co/smoll-swarm/SmollExtractor-1.7B}
}

---

Built by QuillBytes

Small models. Big swarm. Zero cloud bills.