Spaces:
Paused
Notatka nazywa pliki tak, jak uzytkownik je wgral, i mowi czemu ktorys odlozyla
Browse filesDwie wady wyszly na przebiegu korpusu "co to jest oaza i czym sie zajmuje".
1. Model przepisuje kazda nazwe pliku, ktora cytuje, i po drodze ja po cichu
"poprawia". Plik nazwany formacja_do2ONZ_szanse_i_zagrozenia.pdf wrocil w
Sources jako ...zagroΕΌenia.pdf, z kreska, ktorej autor nie postawil.
Sources i linia odlozonych to jedyne rozliczenie, jakie uzytkownik dostaje
za swoj upload, wiec nazwa, ktorej nie da sie dopasowac do wyslanego pliku,
to realna wada, nie kosmetyka.
Naprawione w kodzie, nie promptem: instrukcje "nie zmieniaj pisowni" model
nadpisuje dokladnie wtedy, gdy pisownia wyglada mu na bledna. Porownanie
znosi wielkosc liter, znaki diakrytyczne i separatory, wiec lapie tez
"wrzesien 2024.pdf" zamiast "wrzesien-2024.pdf" i dziala na nazwach ze
spacjami. Na przebiegu: 2 naprawione nazwy.
2. Linia odlozonych prosila o jeden wspolny powod, i to opcjonalnie, wiec
model pomijal go calkiem. Teraz powod jest wymagany przy kazdym pliku i
ma nazywac, czym ten plik JEST. Przed: sama lista nazw. Po: "allogol.png
(ikona aplikacji), filech3_0 (plik danych), logonnnl.svg (grafika
wektorowa)". To wprost odpowiada na skarge, ze LUNA "ignoruje wazne pliki":
decyzja jest teraz widoczna i mozna ja podwazyc.
Ustalone przy okazji, nie jest to blad: wrzesien-2024.pdf dostaje od MAP
ocene "low", bo ten plik ani razu nie nazywa ruchu - nie ma w nim slowa
"oaza" ani "Swiatlo-Zycie". MAP ocenia kazdy plik osobno, wiec nie widzi, ze
pozostale ustalaja kontekst. Pliki z ocena "low" i tak docieraja do REDUCE z
pelnymi notatkami, wiec to REDUCE swiadomie je odklada, zgodnie z zalozeniem.
Testy: 30 -> 36.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- synthesis.py +78 -2
- test_luna.py +44 -1
|
@@ -4,6 +4,7 @@
|
|
| 4 |
# REDUCE - fuse all kept notes into one structured Note (strong model, streamed).
|
| 5 |
import json
|
| 6 |
import re
|
|
|
|
| 7 |
|
| 8 |
from config import (
|
| 9 |
DEBUG, MAP_MAX_TOKENS, MAP_MODEL, MAP_TEMPERATURE, REDUCE_BUDGET,
|
|
@@ -414,8 +415,14 @@ def _reduce_system(language: str) -> str:
|
|
| 414 |
"group related facts even when they came from different files.\n"
|
| 415 |
"4. '## Sources' - a bullet per file you ACTUALLY used, formatted "
|
| 416 |
"`- **<filename>** - <what it contributed>`. Then, if any files were set aside, one "
|
| 417 |
-
"final line: `_Not used: <
|
| 418 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 419 |
"Be thorough but do not repeat yourself. Optimize for a reader skimming the Note."
|
| 420 |
)
|
| 421 |
|
|
@@ -483,6 +490,71 @@ def _reduce_user(prompt: str, kept: list[dict], skipped: list[dict], fmt: str =
|
|
| 483 |
)
|
| 484 |
|
| 485 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 486 |
def reduce_stream(prompt: str, kept: list[dict], skipped: list[dict], fmt: str = ""):
|
| 487 |
"""Stream the final Note from the strong model. Yields the accumulated Markdown.
|
| 488 |
The model mirrors the request's language itself (more reliable than langdetect,
|
|
@@ -524,12 +596,16 @@ def reduce_stream(prompt: str, kept: list[dict], skipped: list[dict], fmt: str =
|
|
| 524 |
yield acc
|
| 525 |
if not acc.strip():
|
| 526 |
raise RuntimeError("empty REDUCE response")
|
|
|
|
|
|
|
|
|
|
| 527 |
except Exception as e:
|
| 528 |
print(f"[LUNA] REDUCE error: {e}")
|
| 529 |
# If the stream already produced a real Note, keep it. Replacing a nearly
|
| 530 |
# finished Note with a raw stitch of the notes throws away the better answer.
|
| 531 |
if len(acc.strip()) > 400:
|
| 532 |
print(f"[LUNA] keeping the partial Note ({len(acc)} chars)")
|
|
|
|
| 533 |
yield acc.rstrip() + "\n\n_(The Note was cut short: the synthesis stream " \
|
| 534 |
"ended early. Everything above is complete.)_"
|
| 535 |
return
|
|
|
|
| 4 |
# REDUCE - fuse all kept notes into one structured Note (strong model, streamed).
|
| 5 |
import json
|
| 6 |
import re
|
| 7 |
+
import unicodedata
|
| 8 |
|
| 9 |
from config import (
|
| 10 |
DEBUG, MAP_MAX_TOKENS, MAP_MODEL, MAP_TEMPERATURE, REDUCE_BUDGET,
|
|
|
|
| 415 |
"group related facts even when they came from different files.\n"
|
| 416 |
"4. '## Sources' - a bullet per file you ACTUALLY used, formatted "
|
| 417 |
"`- **<filename>** - <what it contributed>`. Then, if any files were set aside, one "
|
| 418 |
+
"final line: `_Not used: <name> (<short reason>), <name> (<short reason>)_`. Give a "
|
| 419 |
+
"reason for EVERY set-aside file. Each reason must name what THAT file actually is - "
|
| 420 |
+
"its subject, or that it is a photo, a logo, a spreadsheet, a data dump - and then "
|
| 421 |
+
"why that did not answer the request. Never write the same reason twice: if two "
|
| 422 |
+
"reasons come out identical you have described the request, not the file. "
|
| 423 |
+
"Files listed under 'Files set aside' above already carry a reason; use it. "
|
| 424 |
+
"A reader who uploaded a file must be able to tell a deliberate decision from a "
|
| 425 |
+
"file that was missed. If every file was used, omit the Not-used line.\n\n"
|
| 426 |
"Be thorough but do not repeat yourself. Optimize for a reader skimming the Note."
|
| 427 |
)
|
| 428 |
|
|
|
|
| 490 |
)
|
| 491 |
|
| 492 |
|
| 493 |
+
def _fold(s: str) -> str:
|
| 494 |
+
"""Case, diacritics and separators removed, so two spellings of one filename meet."""
|
| 495 |
+
s = unicodedata.normalize("NFKD", s or "")
|
| 496 |
+
s = "".join(c for c in s if not unicodedata.combining(c))
|
| 497 |
+
return re.sub(r"[^a-z0-9]+", "", s.lower())
|
| 498 |
+
|
| 499 |
+
|
| 500 |
+
def _fix_filenames(note: str, names: list[str]) -> str:
|
| 501 |
+
"""Put the real filenames back.
|
| 502 |
+
|
| 503 |
+
The model retypes every filename it cites, and a strong model will quietly
|
| 504 |
+
"correct" one on the way past: a file actually called
|
| 505 |
+
`formacja_do2ONZ_szanse_i_zagrozenia.pdf` came back as `...zagroΕΌenia.pdf`,
|
| 506 |
+
with the diacritic the author had left out. The Note's Sources and Not-used
|
| 507 |
+
lines are the only accounting a user gets for what happened to their upload,
|
| 508 |
+
so a name they cannot match against the file they sent is a real fault, not a
|
| 509 |
+
cosmetic one. Repaired here rather than in the prompt, because an instruction
|
| 510 |
+
not to touch a spelling is exactly the kind a model overrides when the spelling
|
| 511 |
+
looks wrong to it.
|
| 512 |
+
|
| 513 |
+
Matching folds case, diacritics and separators, so it also catches
|
| 514 |
+
`wrzesien 2024.pdf` for `wrzesien-2024.pdf`, and works on names with spaces.
|
| 515 |
+
"""
|
| 516 |
+
if not note or not names:
|
| 517 |
+
return note
|
| 518 |
+
missing = [n for n in names if n and n not in note]
|
| 519 |
+
if not missing:
|
| 520 |
+
return note
|
| 521 |
+
# fold the note, remembering where each surviving character came from
|
| 522 |
+
keep, pos = [], []
|
| 523 |
+
flat = unicodedata.normalize("NFKD", note)
|
| 524 |
+
src = []
|
| 525 |
+
i = 0
|
| 526 |
+
for ch in note:
|
| 527 |
+
for c in unicodedata.normalize("NFKD", ch):
|
| 528 |
+
src.append(i)
|
| 529 |
+
i += 1
|
| 530 |
+
for j, c in enumerate(flat):
|
| 531 |
+
if unicodedata.combining(c):
|
| 532 |
+
continue
|
| 533 |
+
lo = c.lower()
|
| 534 |
+
if re.match(r"[a-z0-9]", lo):
|
| 535 |
+
keep.append(lo)
|
| 536 |
+
pos.append(src[j])
|
| 537 |
+
folded = "".join(keep)
|
| 538 |
+
edits = []
|
| 539 |
+
for real in missing:
|
| 540 |
+
target = _fold(real)
|
| 541 |
+
if not target:
|
| 542 |
+
continue
|
| 543 |
+
start = folded.find(target)
|
| 544 |
+
while start != -1:
|
| 545 |
+
a = pos[start]
|
| 546 |
+
b = pos[start + len(target) - 1]
|
| 547 |
+
edits.append((a, b + 1, real))
|
| 548 |
+
start = folded.find(target, start + len(target))
|
| 549 |
+
if not edits:
|
| 550 |
+
return note
|
| 551 |
+
out = note
|
| 552 |
+
for a, b, real in sorted(edits, key=lambda e: -e[0]):
|
| 553 |
+
out = out[:a] + real + out[b:]
|
| 554 |
+
print(f"[LUNA] repaired {len(edits)} filename(s) in the Note")
|
| 555 |
+
return out
|
| 556 |
+
|
| 557 |
+
|
| 558 |
def reduce_stream(prompt: str, kept: list[dict], skipped: list[dict], fmt: str = ""):
|
| 559 |
"""Stream the final Note from the strong model. Yields the accumulated Markdown.
|
| 560 |
The model mirrors the request's language itself (more reliable than langdetect,
|
|
|
|
| 596 |
yield acc
|
| 597 |
if not acc.strip():
|
| 598 |
raise RuntimeError("empty REDUCE response")
|
| 599 |
+
fixed = _fix_filenames(acc, [k["name"] for k in kept] + [x["name"] for x in skipped])
|
| 600 |
+
if fixed != acc:
|
| 601 |
+
yield fixed
|
| 602 |
except Exception as e:
|
| 603 |
print(f"[LUNA] REDUCE error: {e}")
|
| 604 |
# If the stream already produced a real Note, keep it. Replacing a nearly
|
| 605 |
# finished Note with a raw stitch of the notes throws away the better answer.
|
| 606 |
if len(acc.strip()) > 400:
|
| 607 |
print(f"[LUNA] keeping the partial Note ({len(acc)} chars)")
|
| 608 |
+
acc = _fix_filenames(acc, [k["name"] for k in kept] + [x["name"] for x in skipped])
|
| 609 |
yield acc.rstrip() + "\n\n_(The Note was cut short: the synthesis stream " \
|
| 610 |
"ended early. Everything above is complete.)_"
|
| 611 |
return
|
|
@@ -12,7 +12,8 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
| 12 |
|
| 13 |
from extract import _looks_binary, _word_density, category_of, extract # noqa: E402
|
| 14 |
from pipeline import _resolve, _validate # noqa: E402
|
| 15 |
-
from synthesis import
|
|
|
|
| 16 |
|
| 17 |
|
| 18 |
# ββ language ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -258,3 +259,45 @@ def test_malformed_svg_still_gives_up_its_words():
|
|
| 258 |
assert "Plan spotkania" in r["payload"]
|
| 259 |
assert "800 x 600" in r["payload"]
|
| 260 |
assert "iVBORw" not in r["payload"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
from extract import _looks_binary, _word_density, category_of, extract # noqa: E402
|
| 14 |
from pipeline import _resolve, _validate # noqa: E402
|
| 15 |
+
from synthesis import (_fix_filenames, _fold, _parse_json, # noqa: E402
|
| 16 |
+
_reduce_user, detect_language) # noqa: E402
|
| 17 |
|
| 18 |
|
| 19 |
# ββ language ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
|
|
| 259 |
assert "Plan spotkania" in r["payload"]
|
| 260 |
assert "800 x 600" in r["payload"]
|
| 261 |
assert "iVBORw" not in r["payload"]
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
# --- the Note has to name files the way the user sent them -------------------
|
| 265 |
+
|
| 266 |
+
def test_fold_ignores_case_diacritics_and_separators():
|
| 267 |
+
assert _fold("wrzesien-2024.pdf") == _fold("Wrzesien 2024.PDF")
|
| 268 |
+
assert _fold("zagrozenia.pdf") == _fold("zagroΕΌenia.pdf")
|
| 269 |
+
assert _fold("OND_ERΕ»_O Εwiecie.pdf") == _fold("ond_erz_o swiecie.pdf")
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
def test_model_correcting_a_filename_is_undone():
|
| 273 |
+
"""Observed: a file called ..._zagrozenia.pdf came back with the diacritic added."""
|
| 274 |
+
real = ["formacja_do2ONZ_szanse_i_zagrozenia.pdf"]
|
| 275 |
+
note = "## Sources\n- **formacja_do2ONZ_szanse_i_zagroΕΌenia.pdf** - formacja"
|
| 276 |
+
assert "zagrozenia.pdf" in _fix_filenames(note, real)
|
| 277 |
+
assert "zagroΕΌenia.pdf" not in _fix_filenames(note, real)
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
def test_filenames_with_spaces_and_missing_diacritics_are_repaired():
|
| 281 |
+
real = ["OND_ERΕ»_O Εwiecie wirtualnym.pdf", "wrzesien-2024.pdf"]
|
| 282 |
+
note = "_Not used: OND_ERZ_O swiecie wirtualnym.pdf, wrzesien 2024.pdf_"
|
| 283 |
+
out = _fix_filenames(note, real)
|
| 284 |
+
assert "OND_ERΕ»_O Εwiecie wirtualnym.pdf" in out
|
| 285 |
+
assert "wrzesien-2024.pdf" in out
|
| 286 |
+
|
| 287 |
+
|
| 288 |
+
def test_correct_note_is_left_exactly_as_it_is():
|
| 289 |
+
real = ["a.pdf", "b.docx"]
|
| 290 |
+
note = "## Sources\n- **a.pdf** - x\n\n_Not used: b.docx (zdjecie)_"
|
| 291 |
+
assert _fix_filenames(note, real) == note
|
| 292 |
+
|
| 293 |
+
|
| 294 |
+
def test_repair_does_not_invent_a_mention():
|
| 295 |
+
real = ["nieobecny.pdf"]
|
| 296 |
+
note = "## Sources\n- **a.pdf** - x"
|
| 297 |
+
assert _fix_filenames(note, real) == note
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def test_every_mention_of_a_name_is_repaired():
|
| 301 |
+
real = ["wrzesien-2024.pdf"]
|
| 302 |
+
note = "wrzesien 2024.pdf w tekscie, oraz _Not used: wrzesien 2024.pdf_"
|
| 303 |
+
assert _fix_filenames(note, real).count("wrzesien-2024.pdf") == 2
|