kiyam/lost-in-decoding-pag

Lost in Decoding? — PAG Checkpoint and Evaluation Artifacts

This repository hosts the model checkpoint, document identifiers, and evaluation artifacts used in:

Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2026, pp. 2994–3005. Paper · arXiv · Code

This work is an inference-time reproducibility and robustness study of Planning Ahead in Generative Retrieval (PAG). It does not introduce a newly trained checkpoint.

The model checkpoint and document identifiers are unmodified artifacts originally released by the PAG authors, Hansi Zeng, Chen Luo, and Hamed Zamani. They are mirrored here to support reproducibility of the exact system evaluated in our study.

The perturbed-query evaluation sets and associated metadata were created for the Lost in Decoding study.

Repository contents and provenance

Artifact	Original creator	Description
pytorch_model.bin, config.json, and tokenizer files	Zeng, Luo, and Zamani (2024)	Original PAG checkpoint, lexical_ripor_direct_lng_knp_seq2seq_1, mirrored without modification.
identifiers/rq_docids.json	Zeng, Luo, and Zamani (2024)	Sequential residual-quantization document identifiers, originally released as aq_smtid/docid_to_tokenids.json.
identifiers/set_docids.json	Zeng, Luo, and Zamani (2024)	Set-based lexical document identifiers, originally released as top_bow/docid_to_tokenids.json.
lost_in_decoding_artifacts/msmarco-dev/	Lost in Decoding authors	Query variations for the MS MARCO Passage Ranking Dev set.
lost_in_decoding_artifacts/trec-dl-2019/	Lost in Decoding authors	Query variations for the TREC Deep Learning 2019 Passage Ranking topics.
lost_in_decoding_artifacts/trec-dl-2020/	Lost in Decoding authors	Query variations for the TREC Deep Learning 2020 Passage Ranking topics.
lost_in_decoding_artifacts/metadata/	Lost in Decoding authors	Perturbation types, generation settings, and random seeds.
data_download/	Lost in Decoding authors	Scripts for downloading the original MS MARCO collection, official queries, and qrels from their upstream providers.

The query-variation files are derived evaluation artifacts created from the official MS MARCO Dev queries and official TREC Deep Learning 2019/2020 topics — see LICENSE.md.

We do not redistribute the full MS MARCO passage collection or the original MS MARCO/TREC-DL queries and qrels in this repository (their redistribution rights are not clearly granted — see Source data below). Use data_download/ to fetch them directly from Microsoft/NIST:

git clone https://huggingface.co/kiyam/lost-in-decoding-pag
cd lost-in-decoding-pag
bash data_download/prepare_evaluation_data.sh

Multilingual (mMARCO) queries used for RQ3 are not re-hosted here either; they are third-party data. See the Source data section below.

Model

• Architecture: T5ForLexicalSemanticGeneration, based on T5-base.
• Weight file: pytorch_model.bin
• Hosted at: https://huggingface.co/kiyam/lost-in-decoding-pag/blob/main/pytorch_model.bin
• Serialization format: PyTorch checkpoint (original transformers 4.17-era pytorch_model.bin).
• SafeTensors conversion: none.
• Retrieval procedure: PAG first performs simultaneous lexical scoring over set-based document identifiers. The resulting document-level planning scores guide trie-constrained autoregressive decoding of sequential document identifiers.
• Training collection: MS MARCO Passage Ranking.
• Important: The custom T5ForLexicalSemanticGeneration class is not part of the standard Transformers library — it is defined in the Lost-in-Decoding repository.

Files

• Hugging Face model repository: https://huggingface.co/kiyam/lost-in-decoding-pag
• Weight file: pytorch_model.bin (direct download)

Use the repo ID (kiyam/lost-in-decoding-pag), not the /blob/main/... page URL, for from_pretrained() and programmatic downloads.

Usage

from transformers import AutoConfig, AutoTokenizer
from t5_pretrainer.modeling.t5_generative_retriever import (
    T5ForLexicalSemanticGeneration,
)

repo_id = "kiyam/lost-in-decoding-pag"

config = AutoConfig.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

model = T5ForLexicalSemanticGeneration.from_pretrained(
    repo_id,
    config=config,
)

T5ForLexicalSemanticGeneration comes from the Lost-in-Decoding repository (t5_pretrainer/modeling/t5_generative_retriever.py) — install/clone that repository so the class is importable. from_pretrained resolves repo_id against the Hub and downloads config.json + pytorch_model.bin automatically; it does not accept the /blob/main/... file page URL.

See the Lost-in-Decoding repository for the full two-stage constrained-decoding pipeline (lexical planning → sequential constrained beam search) required to actually run retrieval with this checkpoint.

Document identifiers (identifiers/)

Both files map docid (str) -> list[int] (token ID sequences forming each document's identifier). See identifiers/README.md for details.

Evaluation artifacts (lost_in_decoding_artifacts/)

Query variations (MS MARCO Dev, TREC-DL 2019, TREC-DL 2020) used in the RQ2 robustness evaluation (Table 6 in the paper, corrected — see the main repository README for the correction notice). See lost_in_decoding_artifacts/README.md for format and generation details.

Source data

• MS MARCO passage ranking dataset (queries, qrels, collection): microsoft/MSMARCO — non-commercial research use only, supplied without granting a license or other intellectual-property rights (Terms and Conditions). Not redistributed here; fetch via data_download/download_msmarco.sh.
• TREC Deep Learning 2019 / 2020 query sets and qrels: NIST TREC-DL. Not redistributed here; fetch via data_download/download_trec_dl_2019.sh / download_trec_dl_2020.sh.
• mMARCO (multilingual queries, RQ3): unicamp-dl/mmarco — not redistributed here; download via cross_lingual/scripts/download_mmarco.sh in the code repository.

Citation

If you use the perturbed-query artifacts or the RQ2/RQ3 evaluation results, please cite:

@inproceedings{mekonnen2026lost,
  title     = {Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval},
  author    = {Mekonnen, Kidist Amde and Li, Yongkang and Tang, Yubao and Lupart, Simon and de Rijke, Maarten},
  booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages     = {2994--3005},
  year      = {2026},
  doi       = {10.1145/3805712.3808567}
}

If you use the PAG checkpoint or document identifiers, please also cite the original PAG paper:

@inproceedings{zeng2024planning,
  title     = {Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding},
  author    = {Zeng, Hansi and Luo, Chen and Zamani, Hamed},
  booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages     = {469--480},
  year      = {2024},
  doi       = {10.1145/3626772.3657746}
}

• PAG paper: arXiv:2404.14600
• PAG upstream repository: github.com/HansiZeng/PAG

License

See LICENSE.md. This repository mixes materials with different provenance and licensing: artifacts created for the Lost in Decoding study are released under Apache License 2.0, while the PAG checkpoint and document-identifier files are third-party artifacts mirrored here without modification. The upstream PAG repository does not publish an explicit license file, so no specific license terms are asserted for those files beyond attribution to their original authors.