README.md

metadata

title: Lilly
emoji: 🌉
colorFrom: blue
colorTo: gray
sdk: gradio
sdk_version: 5.50.0
python_version: '3.12'
app_file: lilly_space.py
pinned: false
license: other
models:
  - Safak11/lilly

Lilly

Bosnian into English — typed, spoken, or photographed.

Try the app · Download the weights · Read the paper

Results

Measured through the shipped path, on held-out data unless noted:

Ability	Result	Evaluation
Translate, bs → en	43.25 BLEU / 68.10 chrF2, 0 leaked tags	1,012 FLORES-200 devtest sentences
Reply, en → bs	32.22 BLEU / 61.55 chrF2; 99.2% Bosnian forms	2,009 FLORES-200 pairs, served int8 path
Listen	11.9% word error	200 held-out FLEURS clips
Read	67.0% found / 65 invented	40 Commons photographs

The listener was shipped by the owner's decision after failing a pre-registered variety gate; the reader's larger held-out test-v2 score is 57.8% / 450 on 132 photographs with text. Full methods, intervals, and failed runs are in the model card and white paper.

Type a sentence and it translates. Record yourself and it listens first. Point a camera at a sign and it reads that. The English comes back written, and out loud if you want it. Swap the arrow and it runs the other way: say or photograph English, get Bosnian back — spoken too.

The live camera needs this Space in its own tab (safak11-lilly-api.hf.space). The huggingface.co page puts Lilly inside another page, and a phone will not give that page a camera. Open the tab, then press the camera button; if the live view still cannot start, the same panel takes a still with the phone camera.

Everything runs here. No API is called and nothing you send leaves this machine. The weights are fetched when the Space starts, so the first visit after it has been asleep can take a few minutes; the serving process then loads the reader and the translators before it listens, so the first photograph is not the one that constructs them. The large listener on two CPU cores takes up to a minute for a ten-second clip.

What it is

The models under one roof, published at Safak11/lilly, plus one fetched from where it lives:

translator	OPUS-MT, fine-tuned, quantised to int8
listen	Whisper large-v3, fine-tuned on Bosnian speech; shipped 8 Sep 2026 by the owner's decision after it was refused at its pre-registered gate (the gated whisper-small stays the baseline)
read	PaddleOCR PP-OCRv6, off the shelf, chosen by a pre-registered rule over the fine-tuned EasyOCR one
speak	Kokoro-82M, as published, for English
speak-bs	Piper's sr_RS-serbski_institut-medium for the Bosnian reply — a stock regional voice from upstream; Piper has no Bosnian checkpoint. Pulled from rhasspy/piper-voices at start, not from the bundle

How well it works

Measured, not estimated. Every number has a test set and most have a p-value; the model card carries the full tables and the method.

	the first builds (unrecorded)	first recorded	today
Translation, BLEU on FLORES devtest, as the user sees it	< 30	37.72, language tag leaked into 308 of 1,012 outputs	43.25, leaked into 0 (re-measured 8 Sep on a T4 after the ordinal splitter fix)
Speech, word error on 200 held-out clips	> 55%	38.5%	11.9%, whisper-large-v3, shipped 8 Sep by the owner's decision, refused at its gate (regional forms 1.1% → 6.1%); the gated whisper-small reads 34.9%
Photographs, words found per photograph, 40 real Commons photographs	< 30%	36.0%	67.0%
Photographs, words invented that are on no sign	> 280	224	65

The first column is the owner's notes from before any measurement was committed. The second is the first number that was written down. The repository README ("Where it started") and the model card carry the full tables and the method.

Three things the numbers do not say, which the model card says at length.

The fine-tuning does not measurably improve understanding of Bosnian-specific terms — a benchmark built to test exactly that claim returns 91.7% against 92.2% at p = 0.360. The base model is already trained across South Slavic languages and arrives there on its own.

The photograph numbers are on real photographs from Wikimedia Commons — the 40, and a second held-out set of 280 (57.8% on its 132 with text) — transcribed by eye, never trained on. An earlier version of this card quoted 75% from synthetic text; the first real photograph read 36%, and that number was never real. Nobody has yet measured a photograph taken on a phone in Bosnia.

And the 11.9% listener cleared two of the three pre-registered gate rows by a wide margin and failed the third twice — by one word on 200 clips, then by 1.1% → 6.1% regional-form substitution (p = 0.018) on all 925, the same two words each time. By the rule written before either run it gets no further look. The owner then shipped it anyway, on 8 September, for what it gets right (14.1% against 39.5% of words wrong on 925 clips) and accepting what it gets wrong; that decision and its reason are written down beside the gate's result, the failed row is on the model card, and the gated whisper-small stays the baseline every listener is measured against.

The correction box

If a translation is wrong, the box under it takes a better one. On this Space those corrections are not kept — a free Space has no persistent storage and they are lost when it restarts. They are kept when Lilly runs on a machine with a volume.