safewaterline/risk-model

A credit model graded by the filings

TL;DR. We built 11,109 dated questions from SEC filings ("will this borrower file for bankruptcy within 365 days of this 10-Q?"), answered each with one probability read from a single token of Qwen3.8-27B, and trained a LoRA adapter with the Brier score as the loss. On 1,702 held-out filings whose windows closed after the training set ends, Brier fell from 0.0483 to 0.0267, expected calibration error from 0.088 to 0.023, AUC rose from 0.82 to 0.86, and the riskiest 5% of filings went from holding 5 of 36 bankruptcies to 12. The raw model is still above the base-rate constant (0.0207) because it over-forecasts on average; a one-parameter rescale fitted on half the test set beats the constant on the other half. All forecasts, the training script and this write-up are linked below. The run stopped at step 229 of 265 when our Tinker credits ran out, so the numbers are from the step-130 evaluation.

Safewaterline arranges asset-based loans. The question a lender asks of every borrower, every quarter, is "will this one still be paying me in a year?" We wanted a model whose answer to that question can be scored, and a training signal that is the score itself.

1. Why a graded exam, and why EDGAR

A forecast can be scored only when three things hold: the question has a date, the input contains nothing from after that date, and the answer is written later by someone other than the forecaster. SEC filings satisfy all three for free.

• Every public company files a 10-Q on a known date. That is the question's cutoff.
• If it enters bankruptcy or receivership it must file an 8-K with Item 1.03 within four business days. That is the label, with its own timestamp.
• Both are on EDGAR with stable URLs.

This is the same shape as the outcome-trained forecasting work from Lightning Rod Labs (SEC risk questions, clinical events, Future-as-Label) and the Tinker cookbook's forecasting and decisions recipes. The difference is the domain and the label: a real credit event for a real borrower, not a question an LLM wrote.

2. The question set

Universe. EDGAR full-text search over 2018 to 2026 for 10-K and 10-Q filings that describe an asset-based facility ("asset-based", "ABL facility", "borrowing base" and similar phrases). 780 filers. We dropped SIC 283x (drug developers, whose going-concern language has nothing to do with collateral) and SIC 60 to 67 (lenders, funds, REITs). 638 borrowers remain.

Questions. One per 10-Q filed by those borrowers from 2018 through September 2025, read through the EDGAR submissions API. The label is 1 if the same CIK files an 8-K with Item 1.03 within 365 days of the 10-Q filing date. Borrowers that had already filed for bankruptcy in the 730 days before the cutoff are excluded, since a second filing inside a restructuring is not a forecast.

	Count
Borrowers	638
Questions (10-Qs)	11,109
Bankruptcies	81
Questions labelled yes	220 (2.0%)

What we did not use as a label. Item 2.04, triggering of a direct financial obligation, looks like a default signal and often is one. It also fires when a merger triggers mandatory repayment of a loan, as it did when U.S. Steel was acquired by Nippon Steel. A label that fires on good news is not a label. Bankruptcy only.

Dossier. For each question we take the 10-Q as first filed, pull the management discussion and the liquidity, debt and covenant sections, replace the company's name and ticker with "the Company", and cap the text at 48,000 characters. The training prompt uses the first 10,000 characters, about 4,200 tokens, to keep the run inside budget.

3. One token, not an essay

Asking a chat model for "a probability between 0 and 1" gives you a number the model chose to write. Nothing in the training signal ties that string to the outcome unless you run RL over sampled rollouts, which costs a group of generations per question.

We used the discriminative form from the Tinker cookbook's decision model instead. The prompt ends in a fixed answer format whose replies share every token except the last:

REPLIES = {YES: "choice#Y", NO: "choice#N"}
ANSWER_FORMAT = "choice#?"

The model's P(yes) is the softmax over those two tokens at the answer position. No sampling, one forward pass per question, and the probability is differentiable. The loss is the Brier score on the outcome:

def brier_loss(choice_probs, target_probs):
    return ((choice_probs - target_probs) ** 2).sum()

loss_fn = probability_loss(brier_loss)
datums = decision_dist_datums(dr, [(make_decision(r), {YES if r["y"] else NO: 1.0}) for r in batch])
res = await tc.forward_backward_custom(datums, loss_fn)

The prompt:

You are a credit officer. Below is an excerpt from a quarterly report (Form 10-Q) that a US public
company filed with the SEC on {cutoff}. The company's name has been replaced with "the Company".
Industry: {industry}.

Question: Will this company file an 8-K reporting a bankruptcy or receivership (Item 1.03) within
365 days of {cutoff}? Use only what was public on that date.

--- Filing excerpt ---
{text}
--- End of excerpt ---

The two choices are rendered as "The company files for bankruptcy within 365 days." and "It does not." The base model, with no training, already produces a probability from this format, which is what makes it a usable control.

4. The split and the leakage question

Two defenses against the model knowing the answer.

Names removed. The model works from the numbers and the language of the filing, not from recall of a ticker.

Time split with a gap year. Training questions have cutoffs on or before 2023-06-30, so every training label (a 365-day window) was resolved before the first test question was asked. Test questions have cutoffs after 2024-06-30. Their windows close between July 2025 and September 2026.

Split	Questions	Positives	Borrowers
Train (cutoff ≤ 2023-06-30)	8,023	151	638
Gap (2023-07 to 2024-06)	unused
Test (cutoff > 2024-06-30)	1,702	36 (2.1%)	451

What the gap does not guarantee is that the base model's pretraining never saw the outcomes. The Qwen3.8-27B model card gives an August 2026 release and no knowledge cutoff. We therefore report the base model's own score as the control: if recall were doing the work, the untrained model would already score well on these anonymized filings. It does not (Brier 0.048, mean forecast 11% against a 2.1% base rate). The clean fix is a fresh test set of questions whose windows close after the release date, which accrues on its own every quarter.

Positives are repeated 4× in the training set so that a batch of 32 holds about two bankruptcies instead of half of one. We did not split by company in the final run; an earlier company-split left only 4 positives in the test set, too few to score.

5. The run

Base model	Qwen/Qwen3.8-27B (dense, Apache 2.0)
Adapter	LoRA, rank 32
Platform	Tinker (Thinking Machines)
Renderer	qwen3_8_low_reasoning, DecisionRenderer + YesNoChoiceRenderer
Batch	32 questions, ~4,200 tokens each
Optimizer	Adam, lr 1e-4, constant
Epochs	1 (265 steps planned)
Eval	all 1,702 test questions at step 0 and step 130
Tokens	26.1M by step 229
Cost	about $120 at posted rates ($4.10 per 1M training tokens, $0.70 per 1M prefill)

The run paused at step 229 when the organization's Tinker balance hit zero. The last full evaluation is step 130; the last saved state is step 220. Checkpoints were taken every 20 steps because an earlier attempt stalled on a telemetry timeout, and the loop is wrapped in a supervisor that resumes from the last state.

6. Results

All numbers are on the 1,702 held-out filings.

Metric	Base model	Trained, step 130	Always say 2.1%
Brier score ↓	0.0483	0.0267	0.0207
Brier skill vs base rate	−1.34	−0.29	0
Expected calibration error (10 bins) ↓	0.0877	0.0233	0.0003
AUC ↑	0.819	0.862	0.5
Mean forecast	0.109	0.040	0.021

Calibration. The base model put 48 filings above 60%, none of which failed. The trained model's 0.3 to 0.5 bin holds 35 filings and 8 bankruptcies (23%), so a 40% forecast there means roughly what it says.

Ranking, the lender's metric. Sort by forecast, flag the riskiest slice, count bankruptcies caught:

Flagged	Filings	Caught, base	Caught, trained	Lift, trained
Top 5%	85	5 of 36	12 of 36	6.7×
Top 10%	170	14	17	4.7×
Top 20%	340	23	27	3.8×
Bottom 50%	851	2	2

The honest part. The raw trained Brier (0.0267) is still worse than the base-rate constant (0.0207). With a 2% base rate the Brier score is dominated by the 98% of negatives, and the model still says 4% on average for them. The ranking is there; the prior is not yet. To check whether that is a one-parameter problem we fitted a logit rescale σ(a·logit(p) + b) on a random half of the test set (a = 0.5, b = −2.5) and scored it on the other half: Brier 0.0212 against a base-rate constant of 0.0218 on that half, raw 0.0257. That is post-hoc calibration and we label it as such. The remaining 36 training steps are the cheaper fix.

7. What is missing

1. Finish the epoch. 36 steps, about $25.
2. A ratio baseline. Logistic regression on the filing's own XBRL facts (cash, debt, coverage, margin, size) to show the model reads more than a spreadsheet.
3. A frontier baseline. A large general model prompted the same way, to show a small trained model matches it.
4. A wider universe. All 10-Q filers, not only asset-based borrowers, which raises the bankruptcy count from 81 to several hundred and makes the calibration bins less noisy.
5. A clean post-release test set. Questions whose windows close after August 2026, which removes the knowledge-cutoff caveat entirely.

8. Where it goes

This is the first step of a general underwriting model: the same loop (dated question, one probability, a grade from what happened) applied next to future asset value, then to risk class, and in the end to the loan-to-value a given risk supports. The filings keep the score.

Artifacts

• Every test forecast, base and trained, with the filing URL on each row: forecasts.csv
• Adapter and training script: safewaterline/risk-model
• Page with the playground: safewaterline.com/risk-model
• Method references: Tinker cookbook decisions recipe; Lightning Rod Labs, Foresight Learning for SEC Risk Prediction; Brier (1950), Verification of forecasts expressed in terms of probability.