PFN-DiD / README.md
lisuo's picture
Add PFN-DiD Arena inference release
b7ab973 verified
|
Raw History Blame Contribute Delete
6.3 kB
metadata
license: other
license_name: tabpfn-3-5-license-v1.0
license_link: >-
  https://huggingface.co/Prior-Labs/tabpfn_3_5/blob/06bf2ba35c80a92a3b9abb436b99cf49e7a0365e/LICENSE
library_name: pytorch
base_model: Prior-Labs/tabpfn_3_5
base_model_relation: adapter
language:
  - en
tags:
  - causal-inference
  - difference-in-differences
  - tabular-data
  - in-context-learning
  - statsarena
  - arena-methods

PFN-DiD

PFN-DiD estimates the population average treatment effect on the treated (ATT) and its uncertainty from an observed two-period panel. It combines a frozen TabPFN-3.5 backbone with separate propensity-score and outcome-regression adapters, five-fold cross-fitting, and a normalized doubly robust DiD score.

[ \operatorname{ATT}=\mathbb{E}[Y_{\mathrm{post}}(1)-Y_{\mathrm{post}}(0)\mid D=1]. ]

This repository provides an arena-methods inference bundle, prepared as a contribution to StatsArena. It does not claim official registration or endorsement by StatsArena. The supported benchmark setting is one numeric table with 500 independent units, including tables without observed covariates.

Checkpoint and files

File Contents
Arena method ZIP Complete inference bundle, observed-only example, and license notices
Adapter checkpoint One selected checkpoint: 155,712 trained PS/OR adapter parameters
Predictor Reusable Python interface and command-line entry point
Configuration Model settings and expected weight hashes
Method metadata Integration metadata for the contributed method

Only the selected adapter parameters are included. The frozen backbone is an external dependency downloaded from Prior Labs. There are no baseline implementations, data generators, training datasets, optimizer states, or alternative checkpoints in this release.

Quick Start

Use Python 3.10 and a compatible CPU or CUDA PyTorch installation. The code was validated on CPU and MUSA; CUDA uses the standard PyTorch device path but was not separately validated for this release. Install the research dependencies:

python -m pip install "huggingface_hub>=0.34.4,<1"
hf download lisuo/PFN-DiD arena-methods/requirements.txt --local-dir pfn_did_hf
python -m pip install -r pfn_did_hf/arena-methods/requirements.txt

Save this as quickstart.py and run python quickstart.py. The official backbone is downloaded once and verified by SHA-256. Its license applies to use of the model and outputs. After downloading the assets, inference can run offline.

import json
from pathlib import Path
import sys
from huggingface_hub import HfApi, hf_hub_download, snapshot_download

# Resolve the release once, then pin that commit for subsequent runs.
revision = HfApi().model_info("lisuo/PFN-DiD", token=False).sha
release = Path(snapshot_download(
    repo_id="lisuo/PFN-DiD", revision=revision,
    allow_patterns=["arena-methods/**", "NOTICE", "TABPFN-3.5-LICENSE"],
    local_dir="pfn_did_hf", token=False,
))
bundle = release / "arena-methods"
backbone = hf_hub_download(
    repo_id="Prior-Labs/tabpfn_3_5",
    revision="06bf2ba35c80a92a3b9abb436b99cf49e7a0365e",
    filename="tabpfn-v3.5-20260909.safetensors", token=False,
)
sys.path.insert(0, str(bundle.resolve()))
from predict import Predictor, read_observed

predictor = Predictor(backbone, device="cpu", threads=4)
result = predictor(read_observed(bundle / "example/observed_n500.npz"),
                   fold_seed=39007)
print(json.dumps(result, indent=2))

On the included synthetic observed-only example, CPU inference gives ATT approximately 1.40108, standard error 0.08694, and the 95% interval [1.23068, 1.57149]. Small floating-point differences across devices are expected. This example is an interface check, not an estimate of benchmark coverage.

For Arena integration, initialize Predictor once and reuse it across tables. Pass only X, D, and delta_y; the evaluator may provide a fixed fold_seed. The reported evaluation below used fold_seed=39007. Ground-truth effects, generator parameters, and evaluator metadata are rejected by the public input interface. This is a standalone integration interface, not a claim of completed integration with a particular Arena API.

See the input specification, model card, and file manifest.

Evaluation

The single released checkpoint was selected by the minimum validation nuisance loss across two training seeds, without using test results. Training used 10,000 tables and validation used 200 tables. Each table contained 500 units. Evaluation used 1,000 ID tables and 1,000 source-family-held-out OOD tables.

Split Bias Error variance RMSE 95% coverage Mean interval length
ID -0.000696 0.017696 0.132960 95.5% 0.462268
OOD -0.005695 0.029410 0.171503 93.8% 0.531544

All 2,000 estimates succeeded. Error variance is measured across test tables with different mechanisms; it is not within-mechanism sampling variance. Each mechanism contributes one table. Source-OOD means that its three families were excluded from adapter training and validation; upstream TabPFN pretraining exposure is uncontrolled. Near-nominal aggregate coverage does not establish conditional or family-specific validity.

License and attribution

This adaptation is distributed for non-commercial, non-production research under the terms applicable to TabPFN-3.5 derivatives. See the supplied TABPFN-3.5 License and NOTICE. It is not an unrestricted commercial-use release. Hosted/API services and commercial or production use require the permissions specified by Prior Labs. PFN-DiD is not an official Prior Labs product.

The DiD score is motivated by Sant'Anna and Zhao, Doubly Robust Difference-in-Differences Estimators. That reference does not by itself establish finite-sample interval validity for the fitted neural nuisances in this release.