feat: live runs use the curated journal (v2); README results for memory and the student
Browse files- README.md +59 -52
- app.py +5 -3
- lantern/journal.py +22 -8
README.md
CHANGED
|
@@ -46,60 +46,65 @@ Lantern is three things:
|
|
| 46 |
|
| 47 |
All runs use the same 8 levels ร 5 seeds (40 dungeons per condition), temperature 0.7.
|
| 48 |
|
| 49 |
-
### Does
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
|
| 56 |
-
|
|
| 57 |
-
|
|
| 58 |
-
|
|
| 59 |
-
|
|
| 60 |
-
|
|
| 61 |
-
|
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
- **
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
The trouble was what the lessons said:
|
| 77 |
|
| 78 |
- **A lesson that sounded wise made Beam slower.** After one loss Beam wrote "execute and verify
|
| 79 |
one movement at a time... instead of batched moves". From then on it planned one or two moves
|
| 80 |
-
per call.
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
with it.
|
| 84 |
- **The journal filled up with one idea.** By the last run, all 12 lessons were rewordings of
|
| 85 |
-
"explore the nearest unexplored area first" and "don't re-test known walls". Only exact
|
| 86 |
-
|
| 87 |
-
- **Some lessons described actions Beam doesn't have.** The final lesson told Beam to "repeat
|
| 88 |
-
search several times" to find secret doors, but Corridors only allows moving. Beam brought in
|
| 89 |
-
general NetHack knowledge that doesn't apply.
|
| 90 |
|
| 91 |
-
Reflection without
|
| 92 |
-
|
| 93 |
|
| 94 |
-
- **Lessons are credited by outcome.** Each
|
| 95 |
-
|
| 96 |
- **Beam curates instead of appending.** After each run it sees the whole journal with those
|
| 97 |
records and rewrites it: merging duplicates, dropping lessons linked to losses, keeping at most
|
| 98 |
8, and preferring lessons that make it faster, since calls are limited.
|
| 99 |
- **Two checks in code.** A lesson present for 3 losses and no wins is removed, and so is any
|
| 100 |
-
lesson that
|
| 101 |
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
### How much can a 1.7B model learn from Beam?
|
| 105 |
|
|
@@ -150,24 +155,26 @@ agent work right now. Lantern gives Beam two kinds of memory:
|
|
| 150 |
|
| 151 |
- **Working memory, within a run.** Beam can call `write_notes` at any time to record where it has
|
| 152 |
been, dead ends, what items do, and its current plan. The notes come back to it every turn.
|
| 153 |
-
- **Long-term memory, across runs.**
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
|
|
|
| 157 |
|
| 158 |
```
|
| 159 |
-
play a run ->
|
| 160 |
-
^
|
| 161 |
-
+---- the next run on that level reads ----+
|
| 162 |
```
|
| 163 |
|
| 164 |
This is self-improvement through memory: what changes between runs is what Beam knows when it
|
| 165 |
starts, not the model. It's the same idea as Reflexion (Shinn et al., 2023), where an agent writes
|
| 166 |
reflections on its attempts and retries with them in context.
|
| 167 |
|
| 168 |
-
It can fail.
|
| 169 |
-
|
| 170 |
-
measured, on and off, rather than assumed to work.
|
|
|
|
| 171 |
|
| 172 |
The student model is a different kind of learning: its weights change, but it learns from Beam,
|
| 173 |
not from itself. Training the student on its own wins and repeating (expert iteration) would close
|
|
|
|
| 46 |
|
| 47 |
All runs use the same 8 levels ร 5 seeds (40 dungeons per condition), temperature 0.7.
|
| 48 |
|
| 49 |
+
### Does memory help Beam?
|
| 50 |
+
|
| 51 |
+
Three conditions on the same 40 dungeons: no journal (**off**), the first journal, which appends
|
| 52 |
+
lessons (**v1**), and the curated journal described below (**v2**). Steps are averaged over won
|
| 53 |
+
runs only.
|
| 54 |
+
|
| 55 |
+
| Level | Wins (off / v1 / v2) | Steps when won | Tokens per run |
|
| 56 |
+
|---|---|---|---|
|
| 57 |
+
| Small Room | 5 / 5 / 5 | 4 / 3 / 3 | 0.9k / 1k / 1k |
|
| 58 |
+
| Room with Monsters | 3 / 4 / 4 | 11 / 9 / 10 | 34k / 22k / 16k |
|
| 59 |
+
| Key Room | 5 / 5 / 5 | 12 / 12 / 18 | 14k / 15k / 21k |
|
| 60 |
+
| Memento | 3 / 3 / 3 | 92 / 76 / 89 | 99k / 74k / 87k |
|
| 61 |
+
| River Crossing | 3 / 3 / 3 | 265 / 111 / 91 | 426k / 343k / 293k |
|
| 62 |
+
| Hide and Seek | 0 / 1 / 1 | โ / 55 / 15 | 49k / 69k / 50k |
|
| 63 |
+
| Maze Walk | 2 / 1 / 1 | 108 / 32 / 58 | 349k / 355k / 331k |
|
| 64 |
+
| Corridors | 2 / 2 / 3 | 52 / 94 / 124 | 364k / 524k / 429k |
|
| 65 |
+
| **Total** | **23 / 24 / 25** of 40 each | | |
|
| 66 |
+
|
| 67 |
+
- **Neither journal changed how often Beam wins.** On seeds 2 to 5, the only runs where a journal
|
| 68 |
+
can help (seed 1 starts empty in every condition), Beam won 18, 18 and 19 of 32. At this
|
| 69 |
+
sample size that's noise.
|
| 70 |
+
- **The curated journal removed the harm and cut cost.** v2 used the fewest tokens of the three
|
| 71 |
+
(153k per run on average, against 167k with no journal and 175k with v1) and brought Corridors
|
| 72 |
+
back to normal speed (see below).
|
| 73 |
+
- **Wins on puzzle levels got shorter with either journal.** River Crossing wins took 265 steps
|
| 74 |
+
with no journal, 111 with v1 and 91 with v2. Key Room went the other way with v2 (12 to 18
|
| 75 |
+
steps), so treat these per-level differences as leads, not findings.
|
| 76 |
+
- **Where Beam struggles:** exploring large layouts (Corridors, Maze Walk) and surviving a hunter
|
| 77 |
+
(Hide and Seek, at most 1 win in 5 in any condition).
|
| 78 |
+
|
| 79 |
+
### What went wrong with the first journal, and the fix
|
| 80 |
+
|
| 81 |
+
On Corridors, Beam read and used its v1 lessons heavily, citing them in 68 to 109 turns per run.
|
| 82 |
The trouble was what the lessons said:
|
| 83 |
|
| 84 |
- **A lesson that sounded wise made Beam slower.** After one loss Beam wrote "execute and verify
|
| 85 |
one movement at a time... instead of batched moves". From then on it planned one or two moves
|
| 86 |
+
per call: 2.0 moves per call with v1 against 2.7 with no journal, and 456 calls against 322.
|
| 87 |
+
Corridors is mostly lost by running out of calls, so caution turned straight into losses. One
|
| 88 |
+
run that took 20 calls with no journal took 96 with v1.
|
|
|
|
| 89 |
- **The journal filled up with one idea.** By the last run, all 12 lessons were rewordings of
|
| 90 |
+
"explore the nearest unexplored area first" and "don't re-test known walls". Only exact repeats
|
| 91 |
+
were merged, so near-duplicates crowded out anything new.
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
+
Reflection without quality control can make an agent worse. The curated journal (v2) changes how
|
| 94 |
+
lessons are kept:
|
| 95 |
|
| 96 |
+
- **Lessons are credited by outcome.** Each records how many runs it was present for that were
|
| 97 |
+
won or lost.
|
| 98 |
- **Beam curates instead of appending.** After each run it sees the whole journal with those
|
| 99 |
records and rewrites it: merging duplicates, dropping lessons linked to losses, keeping at most
|
| 100 |
8, and preferring lessons that make it faster, since calls are limited.
|
| 101 |
- **Two checks in code.** A lesson present for 3 losses and no wins is removed, and so is any
|
| 102 |
+
lesson that relies on an action the level doesn't have.
|
| 103 |
|
| 104 |
+
It worked where v1 failed. The first lesson v2 wrote on Corridors was "batch multiple moves in one
|
| 105 |
+
call to save the 120-call budget", the opposite of v1's. Moves per call went back to 2.7, Corridors
|
| 106 |
+
won 3 of 5, and that level's journal settled on a few distinct lessons instead of one idea twelve
|
| 107 |
+
times over.
|
| 108 |
|
| 109 |
### How much can a 1.7B model learn from Beam?
|
| 110 |
|
|
|
|
| 155 |
|
| 156 |
- **Working memory, within a run.** Beam can call `write_notes` at any time to record where it has
|
| 157 |
been, dead ends, what items do, and its current plan. The notes come back to it every turn.
|
| 158 |
+
- **Long-term memory, across runs.** Every lesson in the journal carries a record of how many runs
|
| 159 |
+
it was present for that were won or lost. When a run ends, Beam sees the journal with those
|
| 160 |
+
records, plus its notes and recent actions, and rewrites it for a *new* layout of the same
|
| 161 |
+
level: up to 8 lessons, duplicates merged, losers dropped. The next run reads it before it
|
| 162 |
+
starts. (The first version just appended lessons; see Results for why that wasn't enough.)
|
| 163 |
|
| 164 |
```
|
| 165 |
+
play a run -> credit each lesson with the outcome -> Beam rewrites the journal
|
| 166 |
+
^ |
|
| 167 |
+
+------------- the next run on that level reads ------------+
|
| 168 |
```
|
| 169 |
|
| 170 |
This is self-improvement through memory: what changes between runs is what Beam knows when it
|
| 171 |
starts, not the model. It's the same idea as Reflexion (Shinn et al., 2023), where an agent writes
|
| 172 |
reflections on its attempts and retries with them in context.
|
| 173 |
|
| 174 |
+
It can fail. A wrong lesson from an unlucky run can mislead later ones, and generic lessons ("move
|
| 175 |
+
toward the goal") don't help at all. The first journal showed both problems, which is why the
|
| 176 |
+
journal is measured, on and off, rather than assumed to work. Lessons from runs a viewer coached
|
| 177 |
+
never enter the shared journal, so a visitor can't plant instructions in it.
|
| 178 |
|
| 179 |
The student model is a different kind of learning: its weights change, but it learns from Beam,
|
| 180 |
not from itself. Training the student on its own wins and repeating (expert iteration) would close
|
app.py
CHANGED
|
@@ -103,14 +103,16 @@ class Run:
|
|
| 103 |
try:
|
| 104 |
env = DungeonEnv(TASKS[self.task_id], seed=self.seed)
|
| 105 |
lessons = journal.lessons(self.task_id)
|
| 106 |
-
|
|
|
|
|
|
|
| 107 |
for event in run_episode(env, ChatClient(), lessons, config, self.stop, self.hints):
|
| 108 |
if self.stop.is_set():
|
| 109 |
break
|
| 110 |
# A coached run's lessons could carry a viewer's text into the journal that
|
| 111 |
# every later run reads in its system prompt, so only uncoached runs write to it.
|
| 112 |
-
if event["type"] == "end" and event
|
| 113 |
-
event["journal"] = journal.
|
| 114 |
store.push_journal()
|
| 115 |
if not self._emit(event):
|
| 116 |
break
|
|
|
|
| 103 |
try:
|
| 104 |
env = DungeonEnv(TASKS[self.task_id], seed=self.seed)
|
| 105 |
lessons = journal.lessons(self.task_id)
|
| 106 |
+
# Journal v2: Beam curates the journal from each lesson's win/loss record.
|
| 107 |
+
config = EpisodeConfig(use_lessons=self.use_lessons, max_llm_calls=MAX_CALLS_PER_RUN,
|
| 108 |
+
curate=True, lesson_records=journal.records(self.task_id))
|
| 109 |
for event in run_episode(env, ChatClient(), lessons, config, self.stop, self.hints):
|
| 110 |
if self.stop.is_set():
|
| 111 |
break
|
| 112 |
# A coached run's lessons could carry a viewer's text into the journal that
|
| 113 |
# every later run reads in its system prompt, so only uncoached runs write to it.
|
| 114 |
+
if event["type"] == "end" and event.get("journal_records") is not None and not self.coached:
|
| 115 |
+
event["journal"] = journal.set_records(self.task_id, event["journal_records"])
|
| 116 |
store.push_journal()
|
| 117 |
if not self._emit(event):
|
| 118 |
break
|
lantern/journal.py
CHANGED
|
@@ -27,25 +27,39 @@ def merge_lessons(old: list[str], new: list[str], limit: int) -> list[str]:
|
|
| 27 |
|
| 28 |
|
| 29 |
class Journal:
|
| 30 |
-
"""Per-task
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
def __init__(self, path: Path, limit: int = JOURNAL_LIMIT):
|
| 33 |
self.path = Path(path)
|
| 34 |
self.limit = limit
|
| 35 |
-
self.
|
| 36 |
-
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
def lessons(self, task_id: str) -> list[str]:
|
| 40 |
-
return
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
def add(self, task_id: str, new: list[str]) -> list[str]:
|
| 43 |
-
"""Merge new lessons
|
|
|
|
| 44 |
merged = merge_lessons(self.lessons(task_id), new, self.limit)
|
| 45 |
-
self.
|
|
|
|
|
|
|
| 46 |
self.path.parent.mkdir(parents=True, exist_ok=True)
|
| 47 |
self.path.write_text(json.dumps(self._data, indent=2), "utf-8")
|
| 48 |
-
return merged
|
| 49 |
|
| 50 |
|
| 51 |
# โโ Journal v2: lessons are credited by outcome and curated, not just appended โโโโโโโโโโ
|
|
|
|
| 27 |
|
| 28 |
|
| 29 |
class Journal:
|
| 30 |
+
"""Per-task lessons with their win/loss records, stored as one JSON file.
|
| 31 |
+
|
| 32 |
+
Older journals stored plain lesson strings; those load as records with no history.
|
| 33 |
+
"""
|
| 34 |
|
| 35 |
def __init__(self, path: Path, limit: int = JOURNAL_LIMIT):
|
| 36 |
self.path = Path(path)
|
| 37 |
self.limit = limit
|
| 38 |
+
raw = json.loads(self.path.read_text("utf-8")) if self.path.exists() else {}
|
| 39 |
+
self._data: dict[str, list[dict]] = {
|
| 40 |
+
task: [new_record(x) if isinstance(x, str) else x for x in items] for task, items in raw.items()}
|
| 41 |
+
|
| 42 |
+
def records(self, task_id: str) -> list[dict]:
|
| 43 |
+
return [dict(r) for r in self._data.get(task_id, [])]
|
| 44 |
|
| 45 |
def lessons(self, task_id: str) -> list[str]:
|
| 46 |
+
return [r["text"] for r in self._data.get(task_id, [])]
|
| 47 |
+
|
| 48 |
+
def set_records(self, task_id: str, records: list[dict]) -> list[str]:
|
| 49 |
+
"""Replace a task's journal (journal v2 after curation); persist; return the lessons."""
|
| 50 |
+
self._data[task_id] = [dict(r) for r in records]
|
| 51 |
+
self._save()
|
| 52 |
+
return self.lessons(task_id)
|
| 53 |
|
| 54 |
def add(self, task_id: str, new: list[str]) -> list[str]:
|
| 55 |
+
"""Merge new lessons in the v1 way (append, newest wins), keeping existing records."""
|
| 56 |
+
known = {r["text"]: r for r in self._data.get(task_id, [])}
|
| 57 |
merged = merge_lessons(self.lessons(task_id), new, self.limit)
|
| 58 |
+
return self.set_records(task_id, [known.get(t, new_record(t)) for t in merged])
|
| 59 |
+
|
| 60 |
+
def _save(self) -> None:
|
| 61 |
self.path.parent.mkdir(parents=True, exist_ok=True)
|
| 62 |
self.path.write_text(json.dumps(self._data, indent=2), "utf-8")
|
|
|
|
| 63 |
|
| 64 |
|
| 65 |
# โโ Journal v2: lessons are credited by outcome and curated, not just appended โโโโโโโโโโ
|