Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Abstract
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
Community
What if your agent could remember your screen - without a model ever writing the memory?
We introduce activity frames: a deterministic, zero-model compiler that turns locally-captured screen activity into memory an agent can trust - and recurring tasks into scripts it can replay.
- A day of raw capture becomes a context block 86x smaller, compiled in 68 ms, byte-identical every run
- An agent reading it answers questions about the day at 98.4%, vs 66-80% for an LLM summary of the same capture
- Recurring routines replay at zero model tokens on a guard-matched hit - model fully out of the loop
- First measurements of two agent-cost parameters read from passive, pre-delegation human activity rather than agent rollouts: routine overhead R = 60-343x (a modeled upper bound) and delegable recurrence ~8%
Everything is local-first and open source (MIT): schema, compiler, MCP server, and the full evaluation harness.
Happy to answer anything - especially skeptical questions about the modeled numerator or the single-user corpus.
What if your agent could remember your screen - without a model ever writing the memory?
We introduce activity frames: a deterministic, zero-model compiler that turns locally-captured screen activity into memory an agent can trust - and recurring tasks into scripts it can replay.
- A day of raw capture becomes a context block 86x smaller, compiled in 68 ms, byte-identical every run
- An agent reading it answers questions about the day at 98.4%, vs 66-80% for an LLM summary of the same capture
- Recurring routines replay at zero model tokens on a guard-matched hit - model fully out of the loop
- First measurements of two agent-cost parameters read from passive, pre-delegation human activity rather than agent rollouts: routine overhead R = 60-343x (a modeled upper bound) and delegable recurrence ~8%
Everything is local-first and open source (MIT): schema, compiler, MCP server, and the full evaluation harness.
code: https://github.com/nossa-y/activity-frames
project: https://usenocta.app
Happy to answer anything - especially skeptical questions about the modeled numerator or the single-user corpus.
Get this paper in your agent:
hf papers read 2608.05784 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper