MemLoop SFT v5 (callbacks), run04 epoch 4
A full fine-tune of Qwen/Qwen3-4B-Instruct-2507 that plays the three agents of a long-term memory system with one set of weights:
- memory agent: writes each conversation session into four memory banks (core, episodic, semantic, procedural);
- retrieval agent: searches the memory (up to 4 searches), then ranks the evidence it hands over;
- answer agent: answers the question from the Core memory and the ranked evidence, with written reasoning.
Every agent can issue a callback: at the end of its tool call it names an agent to redo its step (itself, or the agent before it) and writes a short feedback. Each agent has a budget of one callback, at most three per trajectory.
This checkpoint is epoch 4 (step 840) of a 10-epoch run. Later epochs did not improve, and epoch 4 had the best overall scores.
Training
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3-4B-Instruct-2507 |
| Data | zengrh3/memloop-sft-v5-callback, 40,320 rows built from the LongMemEval SFT split (no dev or test data) |
| Method | Full-parameter SFT (LLaMA-Factory, template qwen3_nothink); in rows that issue a callback, only the callback fields are trained |
| Schedule | 10 epochs planned, this is epoch 4; effective batch 192, learning rate 5e-7, cosine schedule with 10% warmup, max grad norm 0.1, cutoff 20,000 tokens, bf16 |
Evaluation
Judge: GPT-4o-mini (default rubric). The F1 and BLEU-1 columns are token-level. LongMemEval uses all 400 test questions. LoCoMo uses the 856 questions of conversations conv-41, 42, 48, 49 and 50; the other five conversations were used for training or model selection by one of the compared methods.
| Mode | LongMemEval accuracy / F1 / BLEU-1 | LoCoMo (856) accuracy / F1 / BLEU-1 |
|---|---|---|
| callbacks executed | 73.25 / 51.90 / 44.26 | 77.10 / 49.69 / 44.01 |
| callback fields ignored | 74.00 / 48.95 / 41.56 | 75.00 / 47.20 / 41.89 |
The two modes use the same memory. Two runs on identical inputs can differ by about ยฑ1.5 points because generation and the judge are not fully deterministic.
Prompt format
The model was trained on exact prompts; use the same system prompts and message layout as the training data
(protocol memory_banks_tool_call_v5_callback, see the dataset card). Every output is exactly one tool call, for example:
<tool_call>
{"name":"answer","arguments":{"reasoning":"...","answer":"17 days","revisit_agent":"none","feedback":""}}
</tool_call>
Use the chat template without thinking (enable_thinking=False).
Note on config.json
The config was saved by transformers 5.8, which stores the RoPE base under rope_parameters
({"rope_theta": 5000000, "rope_type": "default"}) and has no top-level rope_theta. Older inference engines (for example
vLLM 0.8.5) read only the top-level key, fall back to 10,000 and produce garbled output on long inputs. With such an engine,
add "rope_theta": 5000000 at the top level of config.json.
- Downloads last month
- -
Model tree for zengrh3/memloop-sft-v5-callback-run04-epoch4
Base model
Qwen/Qwen3-4B-Instruct-2507