--- title: Parsed, Plausible, Wrong β€” Pizza Order Parser emoji: πŸ• colorFrom: indigo colorTo: gray sdk: gradio sdk_version: 6.32.0 app_file: app.py python_version: "3.12" short_description: Pizza-order semantic parsers from arXiv:2611.17169 startup_duration_timeout: 30m --- # Parsed, Plausible, Wrong β€” pizza order parser Live demo of the semantic parsers released with *Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing* ([arXiv:2611.17169](https://arxiv.org/abs/2611.17169), IDM Lab, ITMO University). Type a spoken-style pizza order. The model answers a structured **WORDS** ticket (`pizza 2 medium ham no olives`), decoded live to the PIZZA benchmark's native s-expression, plus a readable ticket. Orders that aren't from the menu β€” off-menu items, chatter, gibberish β€” are answered `?` whole, never half-parsed: a loud refusal instead of a silent wrong pizza. Spoken self-corrections ("a pizza with ham, actually make that bacon") are applied. Two models ship with the paper, both selectable: | model | base | exact match | silent errors | |---|---|---|---| | [umarsheikh303/pizza-words-gguf](https://huggingface.co/umarsheikh303/pizza-words-gguf) | Qwen3.5-0.8B | 88.2% | 3.8% | | [umarsheikh303/pizza-lfm2-words-gguf](https://huggingface.co/umarsheikh303/pizza-lfm2-words-gguf) | LFM2.5-350M | 79.3% | 10.2% | Both run in llama.cpp (GGUF Q8_0) under the exact prompt, greedy decoding, stop string and token cap of their export manifests β€” the same contract the GGUFs were server-verified under (99.0% / 93% agreement with the training runs over 200 orders). The WORDS decoder and the benchmark's unordered exact-match matcher are ports of the paper's own reference implementation ([UmarGit/coffee-llm](https://github.com/UmarGit/coffee-llm), `src/bench/tasks/pizza/{formats,ladder}.py`). **The finding:** writing the same target tree as natural words instead of the dataset's native s-expression keeps accuracy (87.5% vs 85.8% exact match, mean of 5 seeds) while cutting silent errors β€” parses that look fine and are wrong β€” from 10.3% to 4.5%, with 3.8Γ— fewer tokens and 3.8Γ— lower latency. Trained on the [PIZZA benchmark](https://github.com/amazon-science/pizza-semantic-parsing-dataset) (Arkoudas et al. 2022, CC-BY-NC-4.0 β€” non-commercial research use only). ## Files - `app.py` β€” Gradio app: prompt contract from the export manifests, greedy llama.cpp inference, WORDSβ†’EXR decode, readable ticket - `pizza_words.py` β€” port of the paper's WORDS decoder + benchmark matcher - `data/pizza-words-q8_0.json`, `data/pizza-lfm2-words-q8_0.json` β€” the authors' export manifests (prompt, stop, cap), shipped verbatim - `data/catalog.json` β€” the PIZZA catalog (entity words), from the authors' site data, CC-BY-NC-4.0