visproj commited on
Commit
fb80aa2
ยท
verified ยท
1 Parent(s): c89eccb

feat: live runs use the curated journal (v2); README results for memory and the student

Browse files
Files changed (3) hide show
  1. README.md +59 -52
  2. app.py +5 -3
  3. lantern/journal.py +22 -8
README.md CHANGED
@@ -46,60 +46,65 @@ Lantern is three things:
46
 
47
  All runs use the same 8 levels ร— 5 seeds (40 dungeons per condition), temperature 0.7.
48
 
49
- ### Does the journal help Beam?
50
-
51
- | Level | Wins, journal off | Wins, journal on | Steps when won (off โ†’ on) | Tokens per run (off โ†’ on) |
52
- |---|---|---|---|---|
53
- | Small Room | 5/5 | 5/5 | 3.8 โ†’ 2.6 | 0.9k โ†’ 1.1k |
54
- | Room with Monsters | 3/5 | 4/5 | 10.7 โ†’ 9.0 | 34k โ†’ 22k |
55
- | Key Room | 5/5 | 5/5 | 12.0 โ†’ 12.4 | 14k โ†’ 15k |
56
- | Memento | 3/5 | 3/5 | 92 โ†’ 76 | 99k โ†’ 74k |
57
- | River Crossing | 3/5 | 3/5 | 265 โ†’ 111 | 426k โ†’ 343k |
58
- | Hide and Seek | 0/5 | 1/5 | โ€“ โ†’ 55 | 49k โ†’ 69k |
59
- | Maze Walk | 2/5 | 1/5 | 108 โ†’ 32 | 349k โ†’ 355k |
60
- | Corridors | 2/5 | 2/5 | 52 โ†’ 94 | 364k โ†’ 524k |
61
- | **Total** | **23/40** | **24/40** | | |
62
-
63
- - **The journal didn't change how often Beam wins** (23 vs 24 out of 40). On the first seed of
64
- each level, where both conditions start with an empty journal, the counts already differ by
65
- one (5 vs 6 of 8), so a one-win gap is within noise at this sample size.
66
- - **It may make wins cheaper on puzzle and memory levels.** When Beam won River Crossing with
67
- lessons, it took 111 steps on average instead of 265; Memento and Room with Monsters also got
68
- shorter and used fewer tokens. Corridors went the other way. With 5 seeds per side, treat this
69
- as a lead worth testing with more seeds, not a finding.
70
- - **Where Beam struggles:** exploring large layouts (Corridors, Maze Walk) and surviving a
71
- hunter (Hide and Seek, 1 win in 10).
72
-
73
- ### Why the journal hurt on Corridors
74
-
75
- Beam read and used its lessons heavily on Corridors, citing them in 68 to 109 turns per run.
 
 
 
 
 
 
76
  The trouble was what the lessons said:
77
 
78
  - **A lesson that sounded wise made Beam slower.** After one loss Beam wrote "execute and verify
79
  one movement at a time... instead of batched moves". From then on it planned one or two moves
80
- per call. Across the five runs it averaged 2.0 moves per call with the journal against 2.7
81
- without, and used 456 calls against 322. Corridors is mostly lost by running out of calls, so
82
- caution turned straight into losses: one run that took 20 calls without the journal took 96
83
- with it.
84
  - **The journal filled up with one idea.** By the last run, all 12 lessons were rewordings of
85
- "explore the nearest unexplored area first" and "don't re-test known walls". Only exact
86
- repeats were being merged, so near-duplicates crowded out anything new.
87
- - **Some lessons described actions Beam doesn't have.** The final lesson told Beam to "repeat
88
- search several times" to find secret doors, but Corridors only allows moving. Beam brought in
89
- general NetHack knowledge that doesn't apply.
90
 
91
- Reflection without any quality control can make an agent worse. So I built a second version of
92
- the journal:
93
 
94
- - **Lessons are credited by outcome.** Each lesson records how many runs it was present for that
95
- were won or lost.
96
  - **Beam curates instead of appending.** After each run it sees the whole journal with those
97
  records and rewrites it: merging duplicates, dropping lessons linked to losses, keeping at most
98
  8, and preferring lessons that make it faster, since calls are limited.
99
  - **Two checks in code.** A lesson present for 3 losses and no wins is removed, and so is any
100
- lesson that mentions an action the level doesn't allow.
101
 
102
- [TODO: journal v2 results, same 8 levels ร— 5 seeds, against journal off and journal v1]
 
 
 
103
 
104
  ### How much can a 1.7B model learn from Beam?
105
 
@@ -150,24 +155,26 @@ agent work right now. Lantern gives Beam two kinds of memory:
150
 
151
  - **Working memory, within a run.** Beam can call `write_notes` at any time to record where it has
152
  been, dead ends, what items do, and its current plan. The notes come back to it every turn.
153
- - **Long-term memory, across runs.** When a run ends, Beam reflects on its notes and recent actions
154
- and writes one to three lessons meant for a *new* layout of the same level. They go into a
155
- journal that the next run reads before it starts. The journal keeps the 12 most recent lessons
156
- per level, and a lesson Beam writes again moves to the front instead of being duplicated.
 
157
 
158
  ```
159
- play a run -> see the outcome -> Beam reflects and writes lessons
160
- ^ |
161
- +---- the next run on that level reads ----+
162
  ```
163
 
164
  This is self-improvement through memory: what changes between runs is what Beam knows when it
165
  starts, not the model. It's the same idea as Reflexion (Shinn et al., 2023), where an agent writes
166
  reflections on its attempts and retries with them in context.
167
 
168
- It can fail. Nothing checks the lessons, so a wrong lesson from an unlucky run can mislead later
169
- ones, and generic lessons ("move toward the goal") don't help at all. That's why the journal is
170
- measured, on and off, rather than assumed to work.
 
171
 
172
  The student model is a different kind of learning: its weights change, but it learns from Beam,
173
  not from itself. Training the student on its own wins and repeating (expert iteration) would close
 
46
 
47
  All runs use the same 8 levels ร— 5 seeds (40 dungeons per condition), temperature 0.7.
48
 
49
+ ### Does memory help Beam?
50
+
51
+ Three conditions on the same 40 dungeons: no journal (**off**), the first journal, which appends
52
+ lessons (**v1**), and the curated journal described below (**v2**). Steps are averaged over won
53
+ runs only.
54
+
55
+ | Level | Wins (off / v1 / v2) | Steps when won | Tokens per run |
56
+ |---|---|---|---|
57
+ | Small Room | 5 / 5 / 5 | 4 / 3 / 3 | 0.9k / 1k / 1k |
58
+ | Room with Monsters | 3 / 4 / 4 | 11 / 9 / 10 | 34k / 22k / 16k |
59
+ | Key Room | 5 / 5 / 5 | 12 / 12 / 18 | 14k / 15k / 21k |
60
+ | Memento | 3 / 3 / 3 | 92 / 76 / 89 | 99k / 74k / 87k |
61
+ | River Crossing | 3 / 3 / 3 | 265 / 111 / 91 | 426k / 343k / 293k |
62
+ | Hide and Seek | 0 / 1 / 1 | โ€“ / 55 / 15 | 49k / 69k / 50k |
63
+ | Maze Walk | 2 / 1 / 1 | 108 / 32 / 58 | 349k / 355k / 331k |
64
+ | Corridors | 2 / 2 / 3 | 52 / 94 / 124 | 364k / 524k / 429k |
65
+ | **Total** | **23 / 24 / 25** of 40 each | | |
66
+
67
+ - **Neither journal changed how often Beam wins.** On seeds 2 to 5, the only runs where a journal
68
+ can help (seed 1 starts empty in every condition), Beam won 18, 18 and 19 of 32. At this
69
+ sample size that's noise.
70
+ - **The curated journal removed the harm and cut cost.** v2 used the fewest tokens of the three
71
+ (153k per run on average, against 167k with no journal and 175k with v1) and brought Corridors
72
+ back to normal speed (see below).
73
+ - **Wins on puzzle levels got shorter with either journal.** River Crossing wins took 265 steps
74
+ with no journal, 111 with v1 and 91 with v2. Key Room went the other way with v2 (12 to 18
75
+ steps), so treat these per-level differences as leads, not findings.
76
+ - **Where Beam struggles:** exploring large layouts (Corridors, Maze Walk) and surviving a hunter
77
+ (Hide and Seek, at most 1 win in 5 in any condition).
78
+
79
+ ### What went wrong with the first journal, and the fix
80
+
81
+ On Corridors, Beam read and used its v1 lessons heavily, citing them in 68 to 109 turns per run.
82
  The trouble was what the lessons said:
83
 
84
  - **A lesson that sounded wise made Beam slower.** After one loss Beam wrote "execute and verify
85
  one movement at a time... instead of batched moves". From then on it planned one or two moves
86
+ per call: 2.0 moves per call with v1 against 2.7 with no journal, and 456 calls against 322.
87
+ Corridors is mostly lost by running out of calls, so caution turned straight into losses. One
88
+ run that took 20 calls with no journal took 96 with v1.
 
89
  - **The journal filled up with one idea.** By the last run, all 12 lessons were rewordings of
90
+ "explore the nearest unexplored area first" and "don't re-test known walls". Only exact repeats
91
+ were merged, so near-duplicates crowded out anything new.
 
 
 
92
 
93
+ Reflection without quality control can make an agent worse. The curated journal (v2) changes how
94
+ lessons are kept:
95
 
96
+ - **Lessons are credited by outcome.** Each records how many runs it was present for that were
97
+ won or lost.
98
  - **Beam curates instead of appending.** After each run it sees the whole journal with those
99
  records and rewrites it: merging duplicates, dropping lessons linked to losses, keeping at most
100
  8, and preferring lessons that make it faster, since calls are limited.
101
  - **Two checks in code.** A lesson present for 3 losses and no wins is removed, and so is any
102
+ lesson that relies on an action the level doesn't have.
103
 
104
+ It worked where v1 failed. The first lesson v2 wrote on Corridors was "batch multiple moves in one
105
+ call to save the 120-call budget", the opposite of v1's. Moves per call went back to 2.7, Corridors
106
+ won 3 of 5, and that level's journal settled on a few distinct lessons instead of one idea twelve
107
+ times over.
108
 
109
  ### How much can a 1.7B model learn from Beam?
110
 
 
155
 
156
  - **Working memory, within a run.** Beam can call `write_notes` at any time to record where it has
157
  been, dead ends, what items do, and its current plan. The notes come back to it every turn.
158
+ - **Long-term memory, across runs.** Every lesson in the journal carries a record of how many runs
159
+ it was present for that were won or lost. When a run ends, Beam sees the journal with those
160
+ records, plus its notes and recent actions, and rewrites it for a *new* layout of the same
161
+ level: up to 8 lessons, duplicates merged, losers dropped. The next run reads it before it
162
+ starts. (The first version just appended lessons; see Results for why that wasn't enough.)
163
 
164
  ```
165
+ play a run -> credit each lesson with the outcome -> Beam rewrites the journal
166
+ ^ |
167
+ +------------- the next run on that level reads ------------+
168
  ```
169
 
170
  This is self-improvement through memory: what changes between runs is what Beam knows when it
171
  starts, not the model. It's the same idea as Reflexion (Shinn et al., 2023), where an agent writes
172
  reflections on its attempts and retries with them in context.
173
 
174
+ It can fail. A wrong lesson from an unlucky run can mislead later ones, and generic lessons ("move
175
+ toward the goal") don't help at all. The first journal showed both problems, which is why the
176
+ journal is measured, on and off, rather than assumed to work. Lessons from runs a viewer coached
177
+ never enter the shared journal, so a visitor can't plant instructions in it.
178
 
179
  The student model is a different kind of learning: its weights change, but it learns from Beam,
180
  not from itself. Training the student on its own wins and repeating (expert iteration) would close
app.py CHANGED
@@ -103,14 +103,16 @@ class Run:
103
  try:
104
  env = DungeonEnv(TASKS[self.task_id], seed=self.seed)
105
  lessons = journal.lessons(self.task_id)
106
- config = EpisodeConfig(use_lessons=self.use_lessons, max_llm_calls=MAX_CALLS_PER_RUN)
 
 
107
  for event in run_episode(env, ChatClient(), lessons, config, self.stop, self.hints):
108
  if self.stop.is_set():
109
  break
110
  # A coached run's lessons could carry a viewer's text into the journal that
111
  # every later run reads in its system prompt, so only uncoached runs write to it.
112
- if event["type"] == "end" and event["new_lessons"] and not self.coached:
113
- event["journal"] = journal.add(self.task_id, event["new_lessons"])
114
  store.push_journal()
115
  if not self._emit(event):
116
  break
 
103
  try:
104
  env = DungeonEnv(TASKS[self.task_id], seed=self.seed)
105
  lessons = journal.lessons(self.task_id)
106
+ # Journal v2: Beam curates the journal from each lesson's win/loss record.
107
+ config = EpisodeConfig(use_lessons=self.use_lessons, max_llm_calls=MAX_CALLS_PER_RUN,
108
+ curate=True, lesson_records=journal.records(self.task_id))
109
  for event in run_episode(env, ChatClient(), lessons, config, self.stop, self.hints):
110
  if self.stop.is_set():
111
  break
112
  # A coached run's lessons could carry a viewer's text into the journal that
113
  # every later run reads in its system prompt, so only uncoached runs write to it.
114
+ if event["type"] == "end" and event.get("journal_records") is not None and not self.coached:
115
+ event["journal"] = journal.set_records(self.task_id, event["journal_records"])
116
  store.push_journal()
117
  if not self._emit(event):
118
  break
lantern/journal.py CHANGED
@@ -27,25 +27,39 @@ def merge_lessons(old: list[str], new: list[str], limit: int) -> list[str]:
27
 
28
 
29
  class Journal:
30
- """Per-task lesson lists stored as one JSON file."""
 
 
 
31
 
32
  def __init__(self, path: Path, limit: int = JOURNAL_LIMIT):
33
  self.path = Path(path)
34
  self.limit = limit
35
- self._data: dict[str, list[str]] = (
36
- json.loads(self.path.read_text("utf-8")) if self.path.exists() else {}
37
- )
 
 
 
38
 
39
  def lessons(self, task_id: str) -> list[str]:
40
- return list(self._data.get(task_id, []))
 
 
 
 
 
 
41
 
42
  def add(self, task_id: str, new: list[str]) -> list[str]:
43
- """Merge new lessons for a task, persist, and return the updated list."""
 
44
  merged = merge_lessons(self.lessons(task_id), new, self.limit)
45
- self._data[task_id] = merged
 
 
46
  self.path.parent.mkdir(parents=True, exist_ok=True)
47
  self.path.write_text(json.dumps(self._data, indent=2), "utf-8")
48
- return merged
49
 
50
 
51
  # โ”€โ”€ Journal v2: lessons are credited by outcome and curated, not just appended โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
 
27
 
28
 
29
  class Journal:
30
+ """Per-task lessons with their win/loss records, stored as one JSON file.
31
+
32
+ Older journals stored plain lesson strings; those load as records with no history.
33
+ """
34
 
35
  def __init__(self, path: Path, limit: int = JOURNAL_LIMIT):
36
  self.path = Path(path)
37
  self.limit = limit
38
+ raw = json.loads(self.path.read_text("utf-8")) if self.path.exists() else {}
39
+ self._data: dict[str, list[dict]] = {
40
+ task: [new_record(x) if isinstance(x, str) else x for x in items] for task, items in raw.items()}
41
+
42
+ def records(self, task_id: str) -> list[dict]:
43
+ return [dict(r) for r in self._data.get(task_id, [])]
44
 
45
  def lessons(self, task_id: str) -> list[str]:
46
+ return [r["text"] for r in self._data.get(task_id, [])]
47
+
48
+ def set_records(self, task_id: str, records: list[dict]) -> list[str]:
49
+ """Replace a task's journal (journal v2 after curation); persist; return the lessons."""
50
+ self._data[task_id] = [dict(r) for r in records]
51
+ self._save()
52
+ return self.lessons(task_id)
53
 
54
  def add(self, task_id: str, new: list[str]) -> list[str]:
55
+ """Merge new lessons in the v1 way (append, newest wins), keeping existing records."""
56
+ known = {r["text"]: r for r in self._data.get(task_id, [])}
57
  merged = merge_lessons(self.lessons(task_id), new, self.limit)
58
+ return self.set_records(task_id, [known.get(t, new_record(t)) for t in merged])
59
+
60
+ def _save(self) -> None:
61
  self.path.parent.mkdir(parents=True, exist_ok=True)
62
  self.path.write_text(json.dumps(self._data, indent=2), "utf-8")
 
63
 
64
 
65
  # โ”€โ”€ Journal v2: lessons are credited by outcome and curated, not just appended โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€