Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attribution

I want provenance filtering to work, and I don't think it survives contact with a real pipeline. Attribution holds on raw generations, then mostly evaporates after a paraphrase pass — which is step one of nearly every prep script I've written. If the signal dies at the first transform, lineage tracking is a lot of bookkeeping for a label that's already gone.

The title is the honest part: knowing a passage came from a model tells you nothing about whether it's worth training on. Before I wire lineage into anything, I'd want the ablation — provenance filtering against a plain quality filter plus dedup, matched on data volume and compute. If it doesn't win there, it's metadata, not a filter.

That comparison is in the paper. Section VII runs the provenance rule against a reference-model perplexity rule, which is the plain quality filter you describe, with a random draw and a fixed draw as controls. Every rule keeps 80 of 160 candidates on the same fine-tuning budget, over three rounds of generation and retraining.
It did not win. The two rules chose substantially different examples, fewer than half shared at each round, but the planned comparison found no stable difference in held-out degradation (p = 0.746). There was no dedup arm. The result does not show the rules are equivalent either, only that neither score was shown to carry the information that predicts training effect. That is the point of the title.