Ferr0 commited on
Commit
0d2863f
·
verified ·
1 Parent(s): 204625e

Structured Output Playground — constrained decoding demo (Qwen2.5-0.5B + llama.cpp)

Browse files
Files changed (5) hide show
  1. README.md +49 -7
  2. app.py +177 -0
  3. examples.py +26 -0
  4. requirements.txt +6 -0
  5. schemas.py +67 -0
README.md CHANGED
@@ -1,13 +1,55 @@
1
  ---
2
  title: Structured Output Playground
3
- emoji: 📈
4
- colorFrom: red
5
- colorTo: purple
6
  sdk: gradio
7
- sdk_version: 6.19.0
8
- python_version: '3.13'
9
  app_file: app.py
10
- pinned: false
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Structured Output Playground
3
+ emoji: 🔒
4
+ colorFrom: indigo
5
+ colorTo: green
6
  sdk: gradio
7
+ sdk_version: "6.19.0"
 
8
  app_file: app.py
9
+ pinned: true
10
+ license: mit
11
+ short_description: Lock any LLM's output to a JSON schema
12
  ---
13
 
14
+ # 🔒 Structured Output Playground
15
+
16
+ **Lock any LLM's output to a JSON schema.** Paste free text, pick (or write) a schema,
17
+ and a *tiny* local model extracts structured data that is **guaranteed** to be valid —
18
+ because the decoder is constrained to the schema at generation time, not asked nicely afterwards.
19
+
20
+ > The point of this demo isn't the model's size. It's that **structure is a property of the
21
+ > decoder, not the model.** Even a 0.5B model on a free CPU returns perfectly typed JSON, every time.
22
+
23
+ ## The toggle is the demo
24
+
25
+ There's a **Constraints ON / OFF** switch.
26
+
27
+ - **ON** — the JSON Schema is compiled to a grammar; the model can only emit tokens that keep the
28
+ output valid. You always get well-formed JSON with the right fields and types.
29
+ - **OFF** — the same model just *tries* to follow the schema in plain prompting. Watch it add a
30
+ markdown fence, drop a field, or invent a key. That gap is exactly what constrained decoding closes.
31
+
32
+ ## How it works
33
+
34
+ - **Model** — [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF) (GGUF, 4-bit). Small on purpose: it's fast on CPU, and small models are exactly where constrained decoding earns its keep.
35
+ - **Inference** — [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python) on CPU.
36
+ - **Constraint** — the JSON Schema is turned into a GBNF grammar by llama.cpp's
37
+ `response_format={"type": "json_object", "schema": ...}`, so only schema-valid token sequences
38
+ are allowed.
39
+ - **Validation** — every output is checked with `jsonschema` so you can *see* valid vs. broken.
40
+
41
+ Three presets (job posting, contact card, product) plus a **Custom** mode where you paste your own
42
+ JSON Schema. All example texts are fictional.
43
+
44
+ ## Running cost
45
+
46
+ A free **CPU Basic** Space — no GPU, no external API, no secrets. It loads a ~400 MB quantized model
47
+ on startup and runs entirely on the box. First call is slow (cold CPU); later calls take a few seconds.
48
+
49
+ ## About
50
+
51
+ Built by **[Ferr0](https://huggingface.co/Ferr0)** — infra-minded AI: local LLM inference,
52
+ structured generation & tool-calling, offline RAG, defensive AI security.
53
+ More at **[pixelium.win](https://pixelium.win)** · **[GitHub](https://github.com/ferr079)**.
54
+
55
+ License: MIT.
app.py ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Structured Output Playground — lock any LLM's output to a JSON schema.
2
+
3
+ A small local model (Qwen2.5-Instruct, GGUF / llama.cpp) extracts structured data
4
+ from free text. With *constrained decoding* on, the output is guaranteed to be valid
5
+ JSON matching the schema — even though the model is tiny. Flip constraints off to see
6
+ the same model drift into broken or extra output: that contrast *is* the demo.
7
+
8
+ Runs on a free CPU Space, no GPU, no external API, no secrets.
9
+ """
10
+
11
+ import json
12
+ import os
13
+ import time
14
+
15
+ import gradio as gr
16
+ import jsonschema
17
+ from llama_cpp import Llama
18
+
19
+ from examples import CONTACT_TEXT, EXAMPLES
20
+ from schemas import CUSTOM_LABEL, PRESETS, preset_schema
21
+
22
+ # --- model config (overridable via Space variables, no secret needed) ----------
23
+ MODEL_REPO = os.environ.get("MODEL_REPO", "Qwen/Qwen2.5-0.5B-Instruct-GGUF")
24
+ MODEL_FILE = os.environ.get("MODEL_FILE", "*q4_k_m.gguf")
25
+ N_CTX = int(os.environ.get("N_CTX", "4096"))
26
+
27
+ SYSTEM_PROMPT = (
28
+ "You are a precise information-extraction engine. "
29
+ "You read the user's text and output a single JSON object and nothing else."
30
+ )
31
+
32
+ DEFAULT_CUSTOM_SCHEMA = json.dumps(
33
+ {
34
+ "type": "object",
35
+ "properties": {
36
+ "summary": {"type": "string"},
37
+ "topics": {"type": "array", "items": {"type": "string"}},
38
+ "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
39
+ },
40
+ "required": ["summary", "sentiment"],
41
+ },
42
+ indent=2,
43
+ )
44
+
45
+ print(f"[init] loading {MODEL_REPO} ({MODEL_FILE}) …")
46
+ _t0 = time.perf_counter()
47
+ llm = Llama.from_pretrained(
48
+ repo_id=MODEL_REPO,
49
+ filename=MODEL_FILE,
50
+ n_ctx=N_CTX,
51
+ n_threads=os.cpu_count(),
52
+ verbose=False,
53
+ )
54
+ print(f"[init] model ready in {time.perf_counter() - _t0:.1f}s")
55
+
56
+
57
+ def resolve_schema(preset: str, custom_schema: str) -> dict:
58
+ """Return the JSON Schema dict for the selected preset (or the custom editor)."""
59
+ if preset == CUSTOM_LABEL:
60
+ return json.loads(custom_schema)
61
+ return preset_schema(preset)
62
+
63
+
64
+ def build_prompt(text: str, schema: dict) -> str:
65
+ return (
66
+ "Extract the relevant fields from the text below and return a JSON object "
67
+ "that conforms to this JSON Schema:\n\n"
68
+ f"{json.dumps(schema, indent=2)}\n\n"
69
+ "Text:\n"
70
+ f'"""{text}"""\n\n'
71
+ "Return only the JSON object."
72
+ )
73
+
74
+
75
+ def extract(text, preset, custom_schema, constraints_on):
76
+ text = (text or "").strip()
77
+ if not text:
78
+ return "", "Paste some text first."
79
+
80
+ try:
81
+ schema = resolve_schema(preset, custom_schema)
82
+ except json.JSONDecodeError as e:
83
+ return "", f"❌ Your custom schema is not valid JSON: {e}"
84
+
85
+ messages = [
86
+ {"role": "system", "content": SYSTEM_PROMPT},
87
+ {"role": "user", "content": build_prompt(text, schema)},
88
+ ]
89
+
90
+ kwargs = dict(messages=messages, temperature=0.0, max_tokens=512)
91
+ if constraints_on:
92
+ # llama.cpp turns the JSON Schema into a grammar -> output is forced to match.
93
+ kwargs["response_format"] = {"type": "json_object", "schema": schema}
94
+
95
+ t0 = time.perf_counter()
96
+ resp = llm.create_chat_completion(**kwargs)
97
+ dt = time.perf_counter() - t0
98
+ raw = resp["choices"][0]["message"]["content"] or ""
99
+
100
+ # Validate whatever came back against the schema.
101
+ try:
102
+ parsed = json.loads(raw)
103
+ jsonschema.validate(parsed, schema)
104
+ pretty = json.dumps(parsed, indent=2, ensure_ascii=False)
105
+ badge = (
106
+ f"✅ **Valid** against the schema · {dt:.1f}s"
107
+ + ("" if constraints_on else " — and the model got it right *without* constraints this time.")
108
+ )
109
+ return pretty, badge
110
+ except json.JSONDecodeError as e:
111
+ if raw.lstrip().startswith("```"):
112
+ why = "the model wrapped its answer in a markdown code fence (```), so a JSON parser chokes on the very first character"
113
+ else:
114
+ why = f"`{e}`"
115
+ mode = (
116
+ "But constraints were ON — please report this."
117
+ if constraints_on
118
+ else "That's exactly what constrained decoding prevents: with constraints on you get no fence, no prose — always parseable."
119
+ )
120
+ return raw, f"❌ **Not valid JSON** · {dt:.1f}s — {why}. {mode}"
121
+ except jsonschema.ValidationError as e:
122
+ mode = "But constraints were ON — please report this." if constraints_on else "Turn constraints ON to force schema compliance."
123
+ return raw, f"⚠️ **Valid JSON but breaks the schema** · {dt:.1f}s\n\n`{e.message}` — {mode}"
124
+
125
+
126
+ def on_preset_change(preset):
127
+ return gr.update(visible=(preset == CUSTOM_LABEL))
128
+
129
+
130
+ INTRO = """
131
+ # 🔒 Structured Output Playground
132
+ **Lock any LLM's output to a JSON schema.** A *tiny* local model (Qwen2.5-0.5B, GGUF/llama.cpp)
133
+ extracts structured data from free text. With **constraints ON**, the decoder is restricted to
134
+ tokens that keep the output valid against the schema — so the JSON is *always* well-formed and
135
+ typed, no matter how small the model is. Toggle constraints **OFF** to watch the same model drift.
136
+
137
+ Free CPU Space · no GPU · no external API · no secrets · built by
138
+ [Ferr0](https://huggingface.co/Ferr0) · [pixelium.win](https://pixelium.win) · [GitHub](https://github.com/ferr079)
139
+ """
140
+
141
+ with gr.Blocks(title="Structured Output Playground") as demo:
142
+ gr.Markdown(INTRO)
143
+
144
+ with gr.Row():
145
+ with gr.Column(scale=1):
146
+ preset = gr.Dropdown(
147
+ choices=list(PRESETS.keys()) + [CUSTOM_LABEL],
148
+ value="Contact card",
149
+ label="Schema preset",
150
+ )
151
+ custom = gr.Code(
152
+ value=DEFAULT_CUSTOM_SCHEMA,
153
+ language="json",
154
+ label="Custom JSON Schema",
155
+ visible=False,
156
+ )
157
+ text = gr.Textbox(
158
+ value=CONTACT_TEXT,
159
+ lines=8,
160
+ label="Source text",
161
+ placeholder="Paste any text to extract from…",
162
+ )
163
+ constraints = gr.Checkbox(value=True, label="Constraints ON (force schema)")
164
+ go = gr.Button("Extract", variant="primary")
165
+
166
+ with gr.Column(scale=1):
167
+ out = gr.Code(label="Extracted JSON", language="json")
168
+ badge = gr.Markdown()
169
+
170
+ gr.Examples(examples=EXAMPLES, inputs=[text, preset, constraints])
171
+
172
+ preset.change(on_preset_change, inputs=preset, outputs=custom)
173
+ go.click(extract, inputs=[text, preset, custom, constraints], outputs=[out, badge])
174
+
175
+
176
+ if __name__ == "__main__":
177
+ demo.launch()
examples.py ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fictional demo texts. Nothing here is real — generic, portfolio-safe."""
2
+
3
+ JOB_TEXT = (
4
+ "We're hiring a Senior Backend Engineer at NimbusForge, a fully-remote team. "
5
+ "You'll design Python services backed by PostgreSQL and ship them on Kubernetes. "
6
+ "Compensation is 75,000–95,000 EUR depending on experience."
7
+ )
8
+
9
+ CONTACT_TEXT = (
10
+ "Hi! I'm Dana Okoro, Head of Partnerships at BlueHarbor. "
11
+ "You can reach me at dana.okoro@blueharbor.example or on +33 6 12 34 56 78."
12
+ )
13
+
14
+ PRODUCT_TEXT = (
15
+ "The TrailLite X2 is a 1.1 kg ultralight tent in our camping range. "
16
+ "It's priced at 249 USD and currently in stock. Highlights: two-person capacity, "
17
+ "double-wall design, 3-season rating and fully taped seams."
18
+ )
19
+
20
+ # [source_text, preset_label, constraints_on]
21
+ EXAMPLES = [
22
+ [CONTACT_TEXT, "Contact card", True],
23
+ [PRODUCT_TEXT, "Product", True],
24
+ [JOB_TEXT, "Job posting", True],
25
+ [PRODUCT_TEXT, "Product", False], # same input, constraints OFF — watch it break
26
+ ]
requirements.txt ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ # gradio is provided by the Space SDK (sdk_version in README.md) — do not pin it here.
2
+ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
3
+ llama-cpp-python==0.3.30
4
+ jsonschema==4.26.0
5
+ pydantic==2.13.4
6
+ huggingface_hub
schemas.py ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Demo schemas for the Structured Output Playground.
2
+
3
+ Each preset is a Pydantic model. We use Pydantic only to *generate* a clean
4
+ JSON Schema (`model_json_schema`); the actual generation is constrained by
5
+ that schema at decode time, and validation is done with `jsonschema`.
6
+ """
7
+
8
+ from enum import Enum
9
+ from typing import List, Optional
10
+
11
+ from pydantic import BaseModel, Field
12
+
13
+
14
+ class Seniority(str, Enum):
15
+ junior = "junior"
16
+ mid = "mid"
17
+ senior = "senior"
18
+ lead = "lead"
19
+ unknown = "unknown"
20
+
21
+
22
+ class SalaryRange(BaseModel):
23
+ min: Optional[int] = None
24
+ max: Optional[int] = None
25
+ currency: Optional[str] = None
26
+
27
+
28
+ class JobPosting(BaseModel):
29
+ title: str
30
+ company: Optional[str] = None
31
+ location: Optional[str] = None
32
+ remote: bool = False
33
+ seniority: Seniority = Seniority.unknown
34
+ skills: List[str] = Field(default_factory=list)
35
+ salary: Optional[SalaryRange] = None
36
+
37
+
38
+ class ContactCard(BaseModel):
39
+ name: str
40
+ email: Optional[str] = None
41
+ phone: Optional[str] = None
42
+ company: Optional[str] = None
43
+ role: Optional[str] = None
44
+
45
+
46
+ class Product(BaseModel):
47
+ name: str
48
+ category: Optional[str] = None
49
+ price: Optional[float] = None
50
+ currency: Optional[str] = None
51
+ in_stock: bool = True
52
+ features: List[str] = Field(default_factory=list)
53
+
54
+
55
+ # preset label -> Pydantic model
56
+ PRESETS = {
57
+ "Job posting": JobPosting,
58
+ "Contact card": ContactCard,
59
+ "Product": Product,
60
+ }
61
+
62
+ CUSTOM_LABEL = "Custom (edit the schema)"
63
+
64
+
65
+ def preset_schema(label: str) -> dict:
66
+ """Return the JSON Schema dict for a preset label."""
67
+ return PRESETS[label].model_json_schema()