COLM 2026 data release: Capability Provenance in Language Models

This bucket is the entry point for the data behind Capability Provenance in Language Models: A Case Study in Social Reasoning (COLM 2026, arXiv:2606.19625). It holds the paper inputs that previously lived only in git or on our cluster, and it maps every part of the paper to the public Hugging Face bucket or dataset that holds its data. Everything listed here is public under the HCAI-Lab-GT organization.

The study is a single-ecosystem deep dive: OLMo3-7B Base (allenai/Olmo-3-1025-7B) and its Dolma3 6T training mix, a 5,678,621-document working set stratified over 576 WebOrganizer topic x format bins, TrackStar attribution via Bergson (d=16 Rademacher projection, fp32), and NGDiff rank-8 LoRA unlearning. Code lives in the public capabilibara repository.

What this bucket contains

register_robustness/ reproduces the pre-registered base-query test end to end: queries/ are the regenerated base-model prompts, query_gradients/ are the Bergson query indices built against the base preconditioner, probe_scores/ and probe_scores_final3/ hold per-query and mean bin scores against the public base document index (the run streamed the index shard by shard and kept only bin-level accumulations), zscored/ holds the bin z-scores, and olmo_base_query_block_result.csv applies the decision rule (mean register loading of at least +0.3 with at least five of seven probes positive).

Where the rest of the paper's data lives

Corpus and working set

• dolma3-6t-unique is the de-duplicated 6T corpus, and dolma3-6t-bloom-index is the Bloom-filter state used to de-duplicate it.
• soc91-labels holds the WebOrganizer labels that define the 576 bins.
• dolma3-6t-sample-10000-docs holds the working-set manifest, sampling contract, bin summary and sampled text, and dolma3-6t-sample-10000-docs-trackstar-shards is the same text resharded into the 316 index shards, so every document ID in a score file resolves to its text.
• Bucket dolma3-6t-sampling-provenance holds the stratified and representative pool manifests behind Figs 9-10, and dolma3-6t-corpus-manifest (dataset) is the unified corpus manifest behind the representative draw.

Gradients, curvature and document-level scores

• Bucket trackstar-gradient-index-base is the 316-shard OLMo3-7B Base document gradient index.
• Bucket trackstar-preconditioners (olmo-3-1025-7b/) and dolma3-6t-preconditioner-100k hold the mixed preconditioner and its 100k-document sample.
• Bucket trackstar-query-gradients-base holds the core-4 query gradients (base/20260326T163642Z_1102443), plus the ARC and BBH builds.
• Buckets trackstar-scores-base-olmes-4bench (SocialIQA, both MMLU sets) and trackstar-scores-base-gsm8k-arc (ARC-Challenge, full 5.68M coverage) hold the document-level scores that feed the bin aggregates. trackstar-perquery-bin-matrices-olmo3-7b-base and trackstar-analysis-perbin-ci hold the per-query bin matrices and the bootstrap CIs in the correctness extremes table.
• Datasets base-query-data and olmes-eval-olmo3-7b-base hold the core-4 query JSONLs (prompt, answer, correctness) and the base OLMES predictions behind the correctness cohorts.

Held-out probe suite

• Bucket colm2026-olmo3-social-eval holds the held-out OLMES evaluations, the instruct-query gradients and top-k results for every held-out probe, the held-out bin bundle, and the faithful multiseed unlearning runs.
• Bucket trackstar-scores-instruct-base-bbh holds the BBH instruct-query profiles, and shared-influential-documents-artifacts holds the cross-probe supportive consensus and document recurrence analyses.
• Buckets tracstar-query-builds-archive-20260524 and trackstar-scores-base-tom-n10b hold the ToMBench and SimpleToM query builds and scores used for the attribution-versus-unlearning comparison.

Unlearning

• Bucket arc-rescore-20260611 holds the ARC 3-variant evaluations, forget texts, and the CSVs behind the paired-significance, selectivity and 2x2 factorial tables.
• Buckets arc-rescore-faithful-unlearning-adapters (72 expA adapters, 24 topics x 3 seeds, plus expC) and unlearn-binlevel-arc-ngdiff-adapters (bin-level pilot) hold the trained adapters, and the unlearning-checkpoints model holds the exp1, exp3, expA and expC checkpoints.
• Buckets unlearn-multiseed-arc-selectivity-checkpoints, arc-rescore-unlearning-outputs and arc-rescore-unlearning-results hold the ARC coverage-robustness runs.
• Bucket tom-n10b-artifacts holds the held-out ToM unlearning grid and figure4_ci/gamma_ci.json (Fig 35).

What is not released, and why

• NegotiationToM text. The NegotiationToM authors distribute their benchmark in a password-protected archive to keep it out of scraped training data. I kept that protection: NegotiationToM metrics, top-k results, bin tables, figures and query gradients are public, but the files that carry its prompts and answers (predictions, recorded inputs, requests, query JSONLs and tokenized query sets) are not. Anyone with the authors' archive can regenerate them with the released code.
• Runs on rented GPUs. A few single-topic unlearning runs were trained on rented GPUs, and their raw outputs were not preserved.
• Full document-level scores for the held-out and register-robustness probes. I stored top-k documents and bin-level scores for these probes, not full 5.68M-document score matrices. They can be recomputed on CPU from the public query gradients and the public document index.
• Per-document lexical features. The per-document lexical feature tables behind the bin characterization were lost; the per-bin profiles in lexical_profiles/ are what the paper reports.
• Logs and intermediate state. Job logs, pickled stream state and abandoned intermediate builds are left out.

Query model for the core benchmarks

The core-4 query gradients in trackstar-query-gradients-base/base/20260326T163642Z_1102443 were built with OLMo3-7B Base (allenai/Olmo-3-1025-7B, see each index_config.json). The held-out probes and the BBH profiles use OLMo3-7B Instruct query gradients.

License and citation

The derived data in this bucket is released under the Open Data Commons Attribution License (ODC-BY 1.0), the license Dolma3 uses. Corpus text in the linked datasets keeps its upstream Dolma terms, and benchmark items keep their original licenses. Please cite the paper via arXiv:2606.19625; the BibTeX is in the repository's CITATION.cff.