PFN-DiD / README.md
lisuo's picture
Add PFN-DiD Arena inference release
b7ab973 verified
|
Raw History Blame Contribute Delete
6.3 kB
---
license: other
license_name: tabpfn-3-5-license-v1.0
license_link: https://huggingface.co/Prior-Labs/tabpfn_3_5/blob/06bf2ba35c80a92a3b9abb436b99cf49e7a0365e/LICENSE
library_name: pytorch
base_model: Prior-Labs/tabpfn_3_5
base_model_relation: adapter
language:
- en
tags:
- causal-inference
- difference-in-differences
- tabular-data
- in-context-learning
- statsarena
- arena-methods
---
# PFN-DiD
PFN-DiD estimates the population average treatment effect on the treated
(ATT) and its uncertainty from an observed two-period panel. It combines a
frozen TabPFN-3.5 backbone with separate propensity-score and outcome-regression
adapters, five-fold cross-fitting, and a normalized doubly robust DiD score.
\[
\operatorname{ATT}=\mathbb{E}[Y_{\mathrm{post}}(1)-Y_{\mathrm{post}}(0)\mid D=1].
\]
This repository provides an **arena-methods inference bundle**, prepared as a
contribution to StatsArena. It does not claim official registration or endorsement
by StatsArena. The supported benchmark setting is one numeric table with 500
independent units, including tables without observed covariates.
## Checkpoint and files
| File | Contents |
|---|---|
| [Arena method ZIP](PFN-DiD-arena-methods.zip) | Complete inference bundle, observed-only example, and license notices |
| [Adapter checkpoint](arena-methods/checkpoints/adapter.safetensors) | One selected checkpoint: 155,712 trained PS/OR adapter parameters |
| [Predictor](arena-methods/predict.py) | Reusable Python interface and command-line entry point |
| [Configuration](arena-methods/config.json) | Model settings and expected weight hashes |
| [Method metadata](arena-methods/methods.json) | Integration metadata for the contributed method |
Only the selected adapter parameters are included. The frozen backbone is an
external dependency downloaded from [Prior Labs](https://huggingface.co/Prior-Labs/tabpfn_3_5).
There are no baseline implementations, data generators, training datasets,
optimizer states, or alternative checkpoints in this release.
## Quick Start
Use Python 3.10 and a compatible CPU or CUDA PyTorch installation. The code was
validated on CPU and MUSA; CUDA uses the standard PyTorch device path but was
not separately validated for this release. Install the research dependencies:
```sh
python -m pip install "huggingface_hub>=0.34.4,<1"
hf download lisuo/PFN-DiD arena-methods/requirements.txt --local-dir pfn_did_hf
python -m pip install -r pfn_did_hf/arena-methods/requirements.txt
```
Save this as `quickstart.py` and run `python quickstart.py`. The official backbone
is downloaded once and verified by SHA-256. Its license applies to use of the
model and outputs. After downloading the assets, inference can run offline.
```python
import json
from pathlib import Path
import sys
from huggingface_hub import HfApi, hf_hub_download, snapshot_download
# Resolve the release once, then pin that commit for subsequent runs.
revision = HfApi().model_info("lisuo/PFN-DiD", token=False).sha
release = Path(snapshot_download(
repo_id="lisuo/PFN-DiD", revision=revision,
allow_patterns=["arena-methods/**", "NOTICE", "TABPFN-3.5-LICENSE"],
local_dir="pfn_did_hf", token=False,
))
bundle = release / "arena-methods"
backbone = hf_hub_download(
repo_id="Prior-Labs/tabpfn_3_5",
revision="06bf2ba35c80a92a3b9abb436b99cf49e7a0365e",
filename="tabpfn-v3.5-20260909.safetensors", token=False,
)
sys.path.insert(0, str(bundle.resolve()))
from predict import Predictor, read_observed
predictor = Predictor(backbone, device="cpu", threads=4)
result = predictor(read_observed(bundle / "example/observed_n500.npz"),
fold_seed=39007)
print(json.dumps(result, indent=2))
```
On the included synthetic observed-only example, CPU inference gives ATT
approximately **1.40108**, standard error **0.08694**, and the 95% interval
**[1.23068, 1.57149]**. Small floating-point differences across devices are expected.
This example is an interface check, not an estimate of benchmark coverage.
For Arena integration, initialize `Predictor` once and reuse it across tables.
Pass only `X`, `D`, and `delta_y`; the evaluator may provide a fixed `fold_seed`.
The reported evaluation below used `fold_seed=39007`. Ground-truth effects,
generator parameters, and evaluator metadata are rejected by the public input
interface. This is a standalone integration interface, not a claim of completed
integration with a particular Arena API.
See the [input specification](arena-methods/README.md),
[model card](arena-methods/MODEL_CARD.md), and [file manifest](arena-methods/manifest.json).
## Evaluation
The single released checkpoint was selected by the minimum validation nuisance
loss across two training seeds, without using test results. Training used 10,000
tables and validation used 200 tables. Each table contained 500 units. Evaluation
used 1,000 ID tables and 1,000 source-family-held-out OOD tables.
| Split | Bias | Error variance | RMSE | 95% coverage | Mean interval length |
|---|---:|---:|---:|---:|---:|
| ID | -0.000696 | 0.017696 | 0.132960 | 95.5% | 0.462268 |
| OOD | -0.005695 | 0.029410 | 0.171503 | 93.8% | 0.531544 |
All 2,000 estimates succeeded. Error variance is measured across test tables
with different mechanisms; it is not within-mechanism sampling variance. Each
mechanism contributes one table. Source-OOD means that its three families were
excluded from adapter training and validation; upstream TabPFN pretraining
exposure is uncontrolled. Near-nominal aggregate coverage does not establish
conditional or family-specific validity.
## License and attribution
This adaptation is distributed for non-commercial, non-production research
under the terms applicable to TabPFN-3.5 derivatives. See the supplied
[TABPFN-3.5 License](TABPFN-3.5-LICENSE) and [NOTICE](NOTICE).
It is not an unrestricted commercial-use release. Hosted/API services and
commercial or production use require the permissions specified by Prior Labs.
PFN-DiD is not an official Prior Labs product.
The DiD score is motivated by Sant'Anna and Zhao,
[Doubly Robust Difference-in-Differences Estimators](https://arxiv.org/abs/1812.01723).
That reference does not by itself establish finite-sample interval validity
for the fitted neural nuisances in this release.