hindsight-or-foresight: fraud early-warning models under honest evaluation

No weights here. These models are not meant to be deployed. They are instruments for measuring how much of a model's apparent skill comes from the evaluation design: each one is re-fitted for every test year on the data an analyst could have had at that time, so a single frozen set of weights would not represent them. This repository records the models instead: the classes, hyperparameters, features, preprocessing and the four evaluation designs, in model_specs.json.

These are the models of the preprint Hindsight or Foresight? Alternative Data, Early Warning, and the Measurement of Corporate-Group Failure by Thanatchon Chongmankhong (read online · PDF · DOI 10.5281/zenodo.24216338). Code: https://github.com/thanatchon36/hindsight-or-foresight. Coded cases and full results: thanatchon36/hindsight-or-foresight (dataset).

Independent personal research. Does not represent the views of any employer or institution.

Task

Score U.S. listed firm-years for the risk of later SEC enforcement for accounting misstatement (an Accounting and Auditing Enforcement Release, AAER), on the public panel of Bao et al. (2020): 146,045 firm-years, 1990 to 2014, used from 1991. Test years are 2003 to 2008 (35,166 firm-years, of which 261, 0.74%, were later sanctioned), the window of Bao et al.

Models

code	class	features	hyperparameters
gbm	HistGradientBoostingClassifier	28 raw Compustat items (at, sale, ni, rect, prcc_f, …)	300 iterations, learning rate 0.05, at most 15 leaves, min 40 samples per leaf, L2 1.0, balanced class weights, seed 20261112
logit	LogisticRegression	14 financial ratios (dch_wc, ch_rsst, soft_assets, bm, …)	C = 1.0, balanced class weights; ratios winsorised at the training 1st/99th percentiles, median-imputed and standardised

Evaluation designs (the leakage ladder)

design	training data for test year t
Random 5-fold CV	stratified folds over all years: the model sees the future and other years of the same fraud case
Out-of-time, no gap	fiscal years ≤ t − 1, although most of their labels were not yet public
Out-of-time, 2-year gap	fiscal years ≤ t − 2
Honest	fiscal years ≤ t − 2, excluding every firm-year of any enforcement case that appears in year t

Results

Test years 2003 to 2008. 95% intervals from 400 class-stratified bootstrap resamples.

design	gbm ROC AUC	gbm avg. precision	gbm precision top 1%	logit ROC AUC	logit avg. precision
Random 5-fold CV	0.833 [0.810, 0.850]	0.041	6.25%	0.690 [0.660, 0.715]	0.014
Out-of-time, no gap	0.802 [0.770, 0.828]	0.034	5.68%	0.685 [0.656, 0.709]	0.014
Out-of-time, 2-year gap	0.764 [0.732, 0.787]	0.023	4.55%	0.680 [0.651, 0.706]	0.013
Honest	0.697 [0.663, 0.721]	0.014	2.27%	0.675 [0.646, 0.701]	0.013

Measured against chance (AUC 0.5), roughly 40% of the gradient-boosting model's apparent skill is an artefact of the evaluation design. The logistic model barely moves, and under honest evaluation the two are statistically close: the flexible learner's in-sample advantage is the part most likely to be hindsight. In a 1:1 case–control sample the honest model's threshold is right 65% of the time; in the population it is right 1.4% of the time.

Files

path	content
model_specs.json	classes, hyperparameters, feature lists, preprocessing, the four evaluation designs and the metric definitions, pinned to code commit d13e605
LICENSE	MIT, as for the code

Use

git clone https://github.com/thanatchon36/hindsight-or-foresight && cd hindsight-or-foresight
pip install -r requirements.txt
bash code/get_data.sh          # the Bao et al. (2020) panel, if data/raw/ is empty
cd code && python real_data.py # E1-E5, about 30-45 minutes on one core

One rung of the ladder on its own:

import sys; sys.path.insert(0, "code")
from real_data import load, out_of_time, metrics, TEST_YEARS

d = load()
honest = out_of_time(d, TEST_YEARS, gap=2, serial_fix=True, kind="gbm")
print(metrics(honest.misstate, honest.score))   # ROC AUC about 0.697

Limitations

• U.S. accounting data and enforcement labels, not alternative data or Thai firms. The experiments measure how evaluation design inflates skill, a property of method rather than data; magnitudes elsewhere may differ.
• Enforcement labels capture detected misconduct only; undetected fraud is coded as clean.
• Not a fraud detector for production use: precision at the true prevalence is a few percent at best.

Data credit

Bao, Y., Ke, B., Li, B., Yu, Y. J. and Zhang, J. (2020). Detecting Accounting Fraud in Publicly Traded U.S. Firms Using a Machine Learning Approach. Journal of Accounting Research 58(1), 199–235. doi:10.1111/1475-679X.12292. Please cite the original paper.

Citation

@misc{chongmankhong2026hindsight,
  author       = {Chongmankhong, Thanatchon},
  title        = {Hindsight or Foresight? Alternative Data, Early Warning, and the Measurement of Corporate-Group Failure: Global Evidence and Implications for Thailand},
  year         = {2026},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.24216338},
  url          = {https://thanatchon36.github.io/hindsight-or-foresight/paper/}
}