MemLoop SFT v5 (callbacks), run04 epoch 4

A full fine-tune of Qwen/Qwen3-4B-Instruct-2507 that plays the three agents of a long-term memory system with one set of weights:

  • memory agent: writes each conversation session into four memory banks (core, episodic, semantic, procedural);
  • retrieval agent: searches the memory (up to 4 searches), then ranks the evidence it hands over;
  • answer agent: answers the question from the Core memory and the ranked evidence, with written reasoning.

Every agent can issue a callback: at the end of its tool call it names an agent to redo its step (itself, or the agent before it) and writes a short feedback. Each agent has a budget of one callback, at most three per trajectory.

This checkpoint is epoch 4 (step 840) of a 10-epoch run. Later epochs did not improve, and epoch 4 had the best overall scores.

Training

Item Value
Base model Qwen/Qwen3-4B-Instruct-2507
Data zengrh3/memloop-sft-v5-callback, 40,320 rows built from the LongMemEval SFT split (no dev or test data)
Method Full-parameter SFT (LLaMA-Factory, template qwen3_nothink); in rows that issue a callback, only the callback fields are trained
Schedule 10 epochs planned, this is epoch 4; effective batch 192, learning rate 5e-7, cosine schedule with 10% warmup, max grad norm 0.1, cutoff 20,000 tokens, bf16

Evaluation

Judge: GPT-4o-mini (default rubric). The F1 and BLEU-1 columns are token-level. LongMemEval uses all 400 test questions. LoCoMo uses the 856 questions of conversations conv-41, 42, 48, 49 and 50; the other five conversations were used for training or model selection by one of the compared methods.

Mode LongMemEval accuracy / F1 / BLEU-1 LoCoMo (856) accuracy / F1 / BLEU-1
callbacks executed 73.25 / 51.90 / 44.26 77.10 / 49.69 / 44.01
callback fields ignored 74.00 / 48.95 / 41.56 75.00 / 47.20 / 41.89

The two modes use the same memory. Two runs on identical inputs can differ by about ยฑ1.5 points because generation and the judge are not fully deterministic.

Prompt format

The model was trained on exact prompts; use the same system prompts and message layout as the training data (protocol memory_banks_tool_call_v5_callback, see the dataset card). Every output is exactly one tool call, for example:

<tool_call>
{"name":"answer","arguments":{"reasoning":"...","answer":"17 days","revisit_agent":"none","feedback":""}}
</tool_call>

Use the chat template without thinking (enable_thinking=False).

Note on config.json

The config was saved by transformers 5.8, which stores the RoPE base under rope_parameters ({"rope_theta": 5000000, "rope_type": "default"}) and has no top-level rope_theta. Older inference engines (for example vLLM 0.8.5) read only the top-level key, fall back to 10,000 and produce garbled output on long inputs. With such an engine, add "rope_theta": 5000000 at the top level of config.json.

Downloads last month
-
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for zengrh3/memloop-sft-v5-callback-run04-epoch4

Finetuned
(2,763)
this model

Dataset used to train zengrh3/memloop-sft-v5-callback-run04-epoch4