File size: 4,481 Bytes
1aa6595 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
library_name: peft
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-1.5B-Instruct
license: mit
tags:
- co-lmlm
- annotation
- lora
---
# CoLMLM-Question-Generator
The **question generator** used to build the training corpora for
[**Co-LMLM: Continuous-Query Limited Memory Language Models**](https://arxiv.org/abs/2607.07707).
Co-LMLM is trained on text in which each factual span carries the question it answers. Producing
those questions with a frontier LLM is far too expensive to run over a pretraining-scale corpus, so
this model distills that step: given a document whose fact spans are already marked and numbered,
and the id of one of them, it emits the question that span answers plus a paraphrased answer.
It is the second stage of a two-stage annotation pipeline. The first stage,
[CoLMLM-Fact-Span-Annotator](https://huggingface.co/lil-lab/CoLMLM-Fact-Span-Annotator), marks the
spans this model is asked about.
This repository contains a **LoRA adapter**, not a standalone model — the base weights are loaded
from `Qwen/Qwen2.5-1.5B-Instruct` at inference time.
## Model details
| | |
| ------------------- | ------------------------------------------------------------------------------------------------------- |
| Base model | [`Qwen/Qwen2.5-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |
| Adaptation | LoRA, r=64, α=64, dropout 0.05, on all linear projections (`q,k,v,o,gate,up,down`) |
| Also trained | embeddings for the 8 added annotation tokens (`<FACT>`, `</FACT>`, `<FACT_ID>`, `<QUESTION>`, `</QUESTION>`, `<ANSWER>`, `</ANSWER>`, `<DOC_SEP>`) |
| Precision | bfloat16 |
| Sequence length | 8192 tokens |
## Prompt format
The user message is the numbered document, then `<DOC_SEP>`, then the question for one fact id.
The chat template is Qwen's default (no system prompt is supplied, so Qwen's default system block
is used — matching training).
```
<document with <FACT>N<FACT_ID>span</FACT> tags><DOC_SEP>
What are the question and paraphrased answer for <FACT>N<FACT_ID>?
```
The model responds with `<QUESTION>...</QUESTION><ANSWER>...</ANSWER>`.
## Usage
For more details and the full annotation pipeline, see the code repository:
👉 **[github.com/lil-lab/Co-LMLM](https://github.com/lil-lab/Co-LMLM)**
Standalone:
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
adapter_id = "lil-lab/CoLMLM-Question-Generator"
base_id = "Qwen/Qwen2.5-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, adapter_id).eval()
context = ("Marie Curie was born in <FACT>1<FACT_ID>Warsaw</FACT> in "
"<FACT>2<FACT_ID>1867</FACT> and won <FACT>3<FACT_ID>two</FACT> Nobel Prizes.")
fact_id = 1
user = (f"{context}<DOC_SEP>\n\n"
f"What are the question and paraphrased answer for <FACT>{fact_id}<FACT_ID>?\n")
prompt = tokenizer.apply_chat_template([{"role": "user", "content": user}],
add_generation_prompt=True, tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))
# <QUESTION>Where was Marie Curie born?</QUESTION><ANSWER>Warsaw</ANSWER><|im_end|>
```
This model is part of the [**Co-LMLM** collection](https://huggingface.co/collections/lil-lab/co-lmlm-6a4e8216d55eae83af348f57).
## Citation
```bibtex
@misc{feldman2026colmlmcontinuousquerylimitedmemory,
title={Co-LMLM: Continuous-Query Limited Memory Language Models},
author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
year={2026},
eprint={2607.07707},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.07707},
}
```
|