| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: peft |
| base_model: mistralai/Ministral-3-3B-Instruct-2512 |
| datasets: |
| - mohammed933/ConvRevise |
| pipeline_tag: text-generation |
| tags: |
| - conversational-search |
| - conversational-retrieval |
| - query-rewriting |
| - memory |
| - lora |
| --- |
| |
| <p align="center"> |
| <img alt="MemRevise" src="https://huggingface.co/mohammed933/MemRevise-models/resolve/main/assets/logo.svg" width="520"> |
| </p> |
|
|
| <h3 align="center">MemRevise: A Revisable Memory for Conversational Search</h3> |
|
|
| <p align="center"> |
| <img alt="Paper" src="https://img.shields.io/badge/Paper-coming%20soon-B31B1B"> |
| <a href="https://github.com/DataScienceUIBK/MemRevise"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?logo=github"></a> |
| <a href="https://huggingface.co/datasets/mohammed933/ConvRevise"><img alt="Benchmark" src="https://img.shields.io/badge/%F0%9F%A4%97%20Benchmark-ConvRevise-FFD21E"></a> |
| <img alt="Base model" src="https://img.shields.io/badge/Base-Ministral--3--3B-FF7000"> |
| <img alt="Licence" src="https://img.shields.io/badge/Licence-Apache%202.0%20%7C%20research%20use-lightgrey"> |
| </p> |
|
|
| **TL;DR.** In conversational search, users correct misunderstandings, revise requests and return to earlier topics. MemRevise keeps a memory of the conversation that a 3B writer revises one turn at a time, so the search query always targets the topic the user means now. This repository holds the writer's checkpoints. |
|
|
| ## Overview |
|
|
| <p align="center"> |
| <img alt="MemRevise overview" src="https://huggingface.co/mohammed933/MemRevise-models/resolve/main/assets/overview.png" width="100%"> |
| </p> |
|
|
| **(a)** The 3B writer is distilled from teacher-written memories. **(b)** At every turn it adds one node to the memory graph: the topic, the facts, and the operations the turn performs; topic statuses follow from the operations. **(c)** At the final turn the node's query is combined with a template query and evidence from the memory, and searched with a frozen retriever. |
|
|
| ## Highlights |
|
|
| - **0.701 mean MRR@10** on the three ConvRevise test sets (TopiOCQA, CORAL, OR-QuAC), against 0.589 for the best competitor. |
| - **3B and one model call per turn**: each turn is read once and folded into the memory. |
| - **Revisable**: a dropped topic keeps its facts and can be restored when the user returns to it. |
|
|
| ## The operations |
|
|
| Each node records how its turn changes the conversation: |
|
|
| | Operation | Meaning | Status of the topic | |
| |---|---|---| |
| | `ADD` | introduces a new topic | becomes current; the previous one becomes earlier | |
| | `REFINE` | continues the current topic: the turn asks more about the topic already being discussed | stays current | |
| | `REVISE` | changes an established value, e.g. *the War of 1812* → *the one in 1863* | becomes current | |
| | `ABANDON` | drops a topic, e.g. after *"no, not that one"* | dropped, its facts kept | |
| | `REACTIVATE` | restores an earlier topic when the user returns to it | current again | |
|
|
| The writer's prompt also defines `RETRACT`, for a premise the user withdraws. |
|
|
| ## Example |
|
|
| A ConvRevise test conversation: the user asks about the red fox, digresses to polo and ancient Macedonia, then asks *"Sorry for the detour — how did the one I asked about first evolve?"*. The memory the released writer builds: |
|
|
| | Turn | User | Topic | Operation | |
| |:-:|---|---|---| |
| | 1 | how long does a red fox live in the wild | red fox | `ADD` | |
| | 2 | what is red fox? | red fox | `REFINE` | |
| | 3 | where did it originate? | red fox | `REFINE` | |
| | 4 | How did polo likely begin, and where did its current form originate? | polo | `ADD` | |
| | 5 | What are some modern variants of polo? | polo | `REFINE` | |
| | 6 | Switching topics a bit, did ancient Macedonia contribute any notable inventions or early technologies? | ancient Macedonia | `ADD` | |
| | 7 | Were there any early glass discoveries linked to Macedonia? | ancient Macedonia | `REFINE` | |
| | **8** | **Sorry for the detour — how did the one I asked about first evolve?** | **red fox** | **`REACTIVATE`** | |
|
|
| The node it writes for the last turn resolves *"the one I asked about first"* to the red fox, links it to turns 1–3 and writes the query; the search then ranks the gold passage (the red fox's *Evolution* section) second among 434,179 passages: |
|
|
| ```json |
| { |
| "turn_no": 8, |
| "asked": ["red fox"], |
| "facts": [], |
| "depends_on": [1, 2, 3], |
| "contradicts": [], |
| "types": {}, |
| "ops": [{"op": "REACTIVATE", "entity": "red fox", "detail": ""}], |
| "query": "how did the red fox evolve?" |
| } |
| ``` |
|
|
| ## Files |
|
|
| | Folder | Content | Size | |
| |---|---|--:| |
| | `writer/` | **the released writer**: LoRA adapter (seed 1), tokenizer and chat template | 55 MB | |
| | `base/` | base model: Ministral-3-3B-Instruct-2512, text weights | 6.9 GB | |
| | `writer_seeds/seed_{1,2,3}/` | the three seeds reported in the paper (`seed_1` = `writer/`) | 55 MB each | |
|
|
| ## Quick start |
|
|
| ```bash |
| git clone https://github.com/DataScienceUIBK/MemRevise && cd MemRevise |
| bash scripts/download.sh # this repository's base/ and writer/, and the ConvRevise collections |
| # R and W: the retriever's and the writer's Python (two environments, see the GitHub README) |
| $R -m memrevise.encode_corpus --corpus orquac |
| $W -m memrevise.write_memory --set dev_orquac_value_revisions # memory and query of each conversation (84 questions) |
| $R -m memrevise.search --set dev_orquac_value_revisions |
| $R -m memrevise.evaluate --sets dev_orquac_value_revisions |
| ``` |
|
|
| To load the model alone: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| d = snapshot_download("mohammed933/MemRevise-models", allow_patterns=["base/*", "writer/*"]) |
| tok = AutoTokenizer.from_pretrained(f"{d}/writer") |
| model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(f"{d}/base", torch_dtype="bfloat16"), f"{d}/writer") |
| ``` |
|
|
| ## Results |
|
|
| MRR@10 on the ConvRevise test sets with the frozen retriever: |
|
|
| | | TopiOCQA | CORAL | OR-QuAC | Mean | |
| |---|:-:|:-:|:-:|:-:| |
| | MemRevise (mean of three seeds, as in the paper) | **0.686** | **0.634** | **0.783** | **0.701** | |
| | MemRevise, released writer (seed 1) | 0.689 | 0.633 | 0.788 | 0.703 | |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Data | 22,163 examples from the TopiOCQA and CORAL training conversations of ConvRevise, built from teacher memories | |
| | Adapter | LoRA: rank 16, alpha 32, dropout 0.05, on the q, k, v and o projections | |
| | Schedule | 3 epochs, learning rate 5e-5, cosine with 5% warm-up, effective batch 12 | |
| | Checkpoint | lowest held-out loss on the fields written before the query | |
|
|
| ## Licence |
|
|
| The adapters and the base model follow [Ministral-3-3B-Instruct-2512](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512) (Apache 2.0). The writer was trained on ConvRevise (CC BY-NC-SA 4.0), so use it for research only. |
|
|
| ## Citation |
|
|
| ```bibtex |
| The BibTeX entry will be added when the paper is published. |
| ``` |
|
|
