Buckets:
| 1/ | |
| There are 505 "agent traces" datasets on the Hugging Face Hub (17.6 GB). | |
| I parsed them to answer: what models, which harnesses, what tasks, how long are sessions? | |
| Short answer: most of it isn't people using agents. But the part that is, is very interesting 🧵 | |
| 2/ | |
| By harness: | |
| • 172 paper-reproduction bot runs | |
| • 164 Claude Code | |
| • 66 pi | |
| • 56 Codex | |
| • ~47 other (Hermes, Factory…) | |
| 48 of the Claude Code datasets are exact copies of another one. 372/505 have no license. | |
| 3/ | |
| After deduping, ~12.3k unique sessions in my sample: | |
| • 10.1k synthetic: one generated prompt, one run, done | |
| • 335 benchmark/eval sweeps | |
| • bot runs (sampled) | |
| • ~1.9k real people working with an agent, from ~50 authors | |
| Real usage is ~15% of sessions. | |
| [image: 1_sessions_by_kind.png] | |
| 4/ | |
| How to tell them apart: real sessions have a conversation. | |
| Synthetic: 93% have exactly 1 user message, 0% interrupts, almost never hit compaction. | |
| Real: 22% single message, 53% have 2–5, 25% have 6+. 24% contain a user interrupt. | |
| [image: 2_messages_per_session.png] | |
| 5/ | |
| Models in real sessions: mostly closed. | |
| pi: gpt-5.3-codex, gpt-5.4, claude-opus-4.6, fable-5, plus local GGUFs (Gemma-4, Qwen3-Coder, GLM-4.7-Flash) | |
| Claude Code: fable-5, opus-4.8, also Kimi-K2.6 & MiniMax-M3 via proxies | |
| Codex: ~half gpt-5.5 | |
| 6/ | |
| Synthetic dumps are the opposite: almost all open-weight models (DeepSeek-v4, Qwen3.x, GLM, Gemma-4, MiniMax) served through OpenRouter or HF Inference. | |
| Open models mostly show up in generated data, not in people's day-to-day work. | |
| 7/ | |
| Session size (median / p90), real users: | |
| pi: 4/15 user msgs, 23/174 tool calls, 6/74 active min | |
| Claude Code: 2/13 msgs, 30/143 calls, 13/113 min | |
| Codex: 3/20 msgs, 43/292 calls, 19/89 min | |
| Long tail: 8% of sessions make >200 tool calls. | |
| 8/ | |
| Context is big: median ~0.5–1M input+cached tokens per session, p90 15–30M. | |
| 5–15% of real sessions hit compaction. Median logged cost on pi: $0.53/session, p90 $7.85. | |
| Tool error rate: 4–6% of calls. | |
| 9/ | |
| What tools do agents use? Boring and consistent: | |
| shell + read + edit + write ≈ 90% of calls in every harness. | |
| Subagents: <1% of sessions. Web search: ~3% of Codex calls. Agents mostly live in the terminal. | |
| 10/ | |
| What do people ask? (LLM-labelled sample, first message) | |
| pi: understand/explain a repo 20%, extend existing code 15% | |
| Claude Code: build something new 32% | |
| Codex: build new 20%, extend 16%, fix 15% | |
| Domains: dev tooling, ML/AI and web lead. Personal admin ≈ 0. | |
| 11/ | |
| The most interesting part: follow-ups. | |
| Of messages after the first one: | |
| • 29% new request | |
| • 27% correction ("no, undo that") | |
| • 24% approval ("great, please do") | |
| • 16% answering the agent's question | |
| Most common real message: "continue". | |
| [image: 3_followups.png] | |
| 12/ | |
| The paper-reproduction bots are their own genre: autonomous loops that reproduce one ICML paper each (code, experiments, logbook, poster). | |
| Median run: 37 min, 106 tool calls. Longest: ~10 days, 1,091 tool calls, ~1 human message. | |
| 13/ | |
| Takeaways if you want to train agents on this: | |
| • Filter hard: one-shot generated sessions ≠ real usage | |
| • Real signal is in the steering: corrections & interrupts show where the agent went wrong | |
| • Synthetic dumps that run the same prompt across many models are good for comparing models | |
| 14/ | |
| Method: parsed pi / Claude Code / Codex / Trackio formats, deduped copies by content, labelled ~3.2k prompts with GLM-5.3-Flash (capped per dataset so no single dump dominates). | |
| Caveat: the real set is small, and a few heavy users shape it. | |
Xet Storage Details
- Size:
- 3.54 kB
- Xet hash:
- 4321b716cd801f74afa429001934e6d8c3bb2f2751c783854baca81bf1020ebd
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.