| --- |
| title: Lilly |
| emoji: 🌉 |
| colorFrom: blue |
| colorTo: gray |
| sdk: gradio |
| sdk_version: 5.50.0 |
| python_version: "3.12" |
| app_file: lilly_space.py |
| pinned: false |
| license: other |
| models: |
| - Safak11/lilly |
| --- |
| |
| # Lilly |
|
|
| Bosnian into English — typed, spoken, or photographed. |
|
|
| [Try the app](https://huggingface.co/spaces/Safak11/Lilly-api) · |
| [Download the weights](https://huggingface.co/Safak11/lilly) · |
| [Read the paper](https://github.com/ssaaffaakk/Lilly/blob/main/docs/WHITE-PAPER.md) |
|
|
|  |
|
|
| ## Results |
|
|
| Measured through the shipped path, on held-out data unless noted: |
|
|
| | Ability | Result | Evaluation | |
| |---|---|---| |
| | Translate, bs → en | **43.25 BLEU / 68.10 chrF2**, 0 leaked tags | 1,012 FLORES-200 devtest sentences | |
| | Reply, en → bs | **32.22 BLEU / 61.55 chrF2**; **99.2%** Bosnian forms | 2,009 FLORES-200 pairs, served int8 path | |
| | Listen | **11.9% word error** | 200 held-out FLEURS clips | |
| | Read | **67.0% found / 65 invented** | 40 Commons photographs | |
|
|
| The listener was shipped by the owner's decision after failing a pre-registered |
| variety gate; the reader's larger held-out test-v2 score is **57.8% / 450** |
| on 132 photographs with text. Full methods, intervals, and failed runs are in |
| the [model card](https://huggingface.co/Safak11/lilly) and |
| [white paper](https://github.com/ssaaffaakk/Lilly/blob/main/docs/WHITE-PAPER.md). |
|
|
| Type a sentence and it translates. Record yourself and it listens first. Point a |
| camera at a sign and it reads that. The English comes back written, and out loud |
| if you want it. Swap the arrow and it runs the other way: say or photograph |
| English, get Bosnian back — spoken too. |
|
|
| The live camera needs this Space in its own tab |
| ([safak11-lilly-api.hf.space](https://safak11-lilly-api.hf.space)). The |
| huggingface.co page puts Lilly inside another page, and a phone will not give |
| that page a camera. Open the tab, then press the camera button; if the live |
| view still cannot start, the same panel takes a still with the phone camera. |
|
|
| Everything runs here. No API is called and nothing you send leaves this machine. |
| The weights are fetched when the Space starts, so the first visit after it has |
| been asleep can take a few minutes; the serving process then loads the reader |
| and the translators before it listens, so the first photograph is not the one |
| that constructs them. The large listener on two CPU cores takes up to a minute |
| for a ten-second clip. |
|
|
| ## What it is |
|
|
| The models under one roof, published at |
| [Safak11/lilly](https://huggingface.co/Safak11/lilly), plus one fetched from |
| where it lives: |
|
|
| | | | |
| |---|---| |
| | translator | OPUS-MT, fine-tuned, quantised to int8 | |
| | listen | Whisper large-v3, fine-tuned on Bosnian speech; shipped 8 Sep 2026 by the owner's decision after it was refused at its pre-registered gate (the gated whisper-small stays the baseline) | |
| | read | PaddleOCR PP-OCRv6, off the shelf, chosen by a pre-registered rule over the fine-tuned EasyOCR one | |
| | speak | Kokoro-82M, as published, for English | |
| | speak-bs | Piper's `sr_RS-serbski_institut-medium` for the Bosnian reply — a stock regional voice from upstream; Piper has no Bosnian checkpoint. Pulled from `rhasspy/piper-voices` at start, not from the bundle | |
|
|
| ## How well it works |
|
|
| Measured, not estimated. Every number has a test set and most have a p-value; |
| the model card carries the full tables and the method. |
|
|
| | | the first builds (unrecorded) | first recorded | today | |
| |---|---|---|---| |
| | Translation, BLEU on FLORES devtest, as the user sees it | **< 30** | 37.72, language tag leaked into 308 of 1,012 outputs | **43.25**, leaked into **0** (re-measured 8 Sep on a T4 after the ordinal splitter fix) | |
| | Speech, word error on 200 held-out clips | **> 55%** | 38.5% | **11.9%**, whisper-large-v3, shipped 8 Sep by the owner's decision, refused at its gate (regional forms 1.1% → 6.1%); the gated whisper-small reads 34.9% | |
| | Photographs, words found per photograph, 40 real Commons photographs | **< 30%** | 36.0% | **67.0%** | |
| | Photographs, words invented that are on no sign | **> 280** | 224 | **65** | |
|
|
| The first column is the owner's notes from before any measurement was committed. |
| The second is the first number that was written down. The repository README |
| ("Where it started") and the model card carry the full tables and the method. |
|
|
| Three things the numbers do not say, which the model card says at length. |
|
|
| The fine-tuning does **not** measurably improve understanding of Bosnian-specific |
| terms — a benchmark built to test exactly that claim returns 91.7% against 92.2% |
| at p = 0.360. The base model is already trained across South Slavic languages and |
| arrives there on its own. |
|
|
| The photograph numbers are on real photographs from Wikimedia Commons — the 40, |
| and a second held-out set of 280 (57.8% on its 132 with text) — transcribed by |
| eye, never trained on. An earlier version of this card quoted 75% from synthetic |
| text; the first real photograph read 36%, and that number was never real. Nobody |
| has yet measured a photograph taken on a phone in Bosnia. |
|
|
| And the 11.9% listener cleared two of the three pre-registered gate rows by a |
| wide margin and failed the third twice — by one word on 200 clips, then by |
| 1.1% → 6.1% regional-form substitution (p = 0.018) on all 925, the same two words |
| each time. By the rule written before either run it gets no further look. The |
| owner then shipped it anyway, on 8 September, for what it gets right (14.1% |
| against 39.5% of words wrong on 925 clips) and accepting what it gets wrong; |
| that decision and its reason are written down beside the gate's result, the |
| failed row is on the model card, and the gated whisper-small stays the |
| baseline every listener is measured against. |
|
|
| ## The correction box |
|
|
| If a translation is wrong, the box under it takes a better one. On this Space |
| those corrections are **not kept** — a free Space has no persistent storage and |
| they are lost when it restarts. They are kept when Lilly runs on a machine with |
| a volume. |
|
|
