Long-Context Preference Optimization
How can a model generate useful preference pairs when the evidence is scattered through a long document? This walkthrough follows chunk selection and shows what the LongMab ablations measured.
Where do the bandit arms come from?
Split the source document into chunks. LongMab initializes chunk utility with a probe based on embedding similarity, then uses an upper confidence bound to balance promising chunks against less-tested ones. The language model generates candidate answers from selected chunks, and feedback from those answers updates the bandit scores.
How much did the bandit contribute?
The same Llama-3.1-8B-Instruct backbone appears in the full and ablated runs. These are averages over five long-context question answering datasets in Table 2.
| Setting | Substring exact match | Token F1 |
|---|---|---|
| Direct preference optimization | 35.90 | 45.09 |
| LongMab without probe initialization | 37.01 | 45.48 |
| LongMab without upper confidence bound | 35.78 | 44.19 |
| Full LongMab | 40.95 | 47.43 |
Where does this method fit in a long-context pipeline?
The bandit runs when collecting candidate responses for preference training. Chosen and rejected responses then become DPO examples. In comparisons with inference-time retrieval or truncation, separate the training cost, the chunk budget during data construction and the evaluation-time context.
Paper and citation
Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization
Shaohua Duan, Pengcheng Huang, Xinze Li, and colleagues. ACL 2026.