Chrononauts 3B, cutoff 2017-01, stage: init

Chrononauts is a suite of search agents frozen at 7 dates (2013-05, 2015-01, 2017-01, 2019-01, 2021-01, 2023-01, 2024-07). For each date, the pretraining data (TiC-LM), all post-training data, and the retrieval corpus stop at that date, so the model has never seen what happened later.

This repository is init for cutoff 2017-01: the context-extended base model (TiC-LM 3B, 2k -> 8k context with continued pretraining, far-retrieval SFT and a leak-judged instruction re-SFT, all on data from before the cutoff). It is not trained as an agent.

Sibling repos: mattwang123/chrononauts-3b-2017-01-init, mattwang123/chrononauts-3b-2017-01-sft, mattwang123/chrononauts-3b-2017-01-agent.

Use with the time-locked corpus

The model is meant to search the corpus frozen at its own cutoff: English Wikipedia rebuilt per cutoff from the full revision history (last revision of each page on or before the cutoff) plus CC-News batches crawled before the cutoff, embedded with BAAI/bge-base-en-v1.5 and searched with a CPU faiss index (OPQ64,IVF32768,PQ64, exact rerank of 400 candidates; top-5, at most 2 passages per article, 4,000 characters per turn). News titles carry "(news; dated no later than )".

Agent format (one episode): the prompt states the current date ("Assume today is YYYY-MM-DD.") and the question. The model alternates <think> ... </think> and <search> query </search>; results come back in <information> ... </information>. The first thinking turn states the year the question asks about; the final turn states the latest year the evidence covers and one evidence label ([SUPPORTED], [PREMISES_ONLY], [INSUFFICIENT], [FORECASTING] = not knowable yet), then <answer> ... </answer>. Evaluation used at most 8 turns, T=0.8, top-p 0.95.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("mattwang123/chrononauts-3b-2017-01-init")
model = AutoModelForCausalLM.from_pretrained("mattwang123/chrononauts-3b-2017-01-init", trust_remote_code=True, torch_dtype="bfloat16")

trust_remote_code=True is needed: TiC-LM (OpenLM) uses LayerNorm with QK-norm (modeling_open_lm_hf.py). Use up to 8,192 tokens of context (RoPE base 48,000).

Leakage control

All post-training text passed three filters: built only from snapshots before the cutoff; no year after the cutoff; an LLM leak judge (with a planted-fact check, at least 36/40 planted post-cutoff facts caught). Training and evaluation sets do not overlap.

Citation

Paper in preparation (Chrononauts, Johns Hopkins University and Apple).

Downloads last month
297
Safetensors
Model size
3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support