Rendered unchanged from reports/model_eval.md. The demo serves the active model below (LightGBM, version 6).
Model card: 12-month distress model (feature set v1)
Generated by python -m tripwire_ml train on 2026-10-26. Data as of 2026-10-27. Model versions: LightGBM 6 (active), logistic 5, Altman 4.
Task
For each SEC filer at each quarter end T: will the company start a bankruptcy case (8-K Item 1.03, text-checked) within (T, T + 12 months]? Features use only data filed on or before T. Companies already in bankruptcy at T are excluded. Banks, insurers, REITs, funds and shells (SIC 6000-6799) are excluded.
Data
| Split | As-of dates | Rows | Companies | Positives | Base rate |
|---|---|---|---|---|---|
| train | 2011-03-31 to 2018-12-31 | 170,550 | 8,597 | 1492 | 0.9% |
| validation | 2020-03-31 to 2020-12-31 | 18,386 | 4,920 | 136 | 0.7% |
| test | 2021-12-31 to 2025-09-30 | 77,891 | 5,938 | 837 | 1.1% |
Label windows of different splits never overlap (as-of dates in 2019 and early 2021 are left out). The test split was used only for these numbers, never for tuning.
Results on the test split
| Model | PR-AUC | ROC-AUC | Recall @ top 5% | Recall @ top 10% | Precision @ top 5% | Brier | Median lead (months) | Bankruptcies flagged |
|---|---|---|---|---|---|---|---|---|
| Altman Z'' (rule) | 0.019 | 0.732 | 4.3% | 13.3% | 0.9% | 0.0106 | 7.2 | 15/211 |
| Logistic regression | 0.051 | 0.824 | 34.4% | 50.5% | 7.4% | 0.0106 | 13.5 | 115/211 |
| LightGBM | 0.098 | 0.889 | 45.0% | 65.0% | 9.7% | 0.0102 | 14.2 | 151/211 |
Base rate on test: 1.1% (a random ranking scores PR-AUC ≈ 0.011). LightGBM PR-AUC is 5.1x the Altman baseline.
Recall @ top k%: share of test bankruptcies whose company ranked in the riskiest k% on that quarter end. Lead time: for bankruptcies starting from 2023-01-01 (so a year of test-period history exists), months between the first quarter end on which the company was in the high tier (top 5%) and the filing; looks back at most 24 months.
Calibration (LightGBM, test)
| Decile | Mean predicted | Observed rate | Rows |
|---|---|---|---|
| 1 | 0.0% | 0.0% | 7,789 |
| 2 | 0.0% | 0.0% | 7,789 |
| 3 | 0.0% | 0.1% | 7,789 |
| 4 | 0.0% | 0.2% | 7,789 |
| 5 | 0.0% | 0.0% | 7,789 |
| 6 | 0.1% | 0.2% | 7,789 |
| 7 | 0.3% | 0.5% | 7,789 |
| 8 | 0.4% | 1.1% | 7,789 |
| 9 | 1.1% | 1.6% | 7,789 |
| 10 | 5.2% | 6.9% | 7,790 |
Rolling-origin backtest (PR-AUC by test year)
Train on as-of dates up to the end of Y-2, test on as-of dates in Y, fixed hyperparameters.
| Year | Rows | Positives | Base rate | Altman | Logistic | LightGBM |
|---|---|---|---|---|---|---|
| 2015 | 23,154 | 301 | 1.3% | 0.018 | 0.075 | 0.080 |
| 2016 | 21,915 | 253 | 1.2% | 0.015 | 0.059 | 0.104 |
| 2017 | 20,582 | 156 | 0.8% | 0.014 | 0.083 | 0.065 |
| 2018 | 19,595 | 193 | 1.0% | 0.015 | 0.066 | 0.085 |
| 2019 | 18,991 | 278 | 1.5% | 0.020 | 0.081 | 0.157 |
| 2020 | 18,386 | 136 | 0.7% | 0.010 | 0.093 | 0.132 |
| 2021 | 19,213 | 68 | 0.4% | 0.005 | 0.028 | 0.042 |
| 2022 | 20,006 | 218 | 1.1% | 0.020 | 0.062 | 0.103 |
| 2023 | 20,019 | 290 | 1.4% | 0.026 | 0.085 | 0.137 |
| 2024 | 19,235 | 210 | 1.1% | 0.021 | 0.062 | 0.123 |
Top drivers (mean |SHAP|, LightGBM, test)
| Feature | Mean |SHAP| | |---|---| | Company size (log total assets) | 0.360 | | Total liabilities / total assets | 0.314 | | Free-cash-flow margin | 0.309 | | Return on assets | 0.281 | | EBIT / total assets | 0.280 | | Working capital / total assets | 0.223 | | Director / officer departures (8-K 5.02), last 12 months | 0.206 | | Interest coverage (EBIT / interest) | 0.193 | | Change in total assets, year over year | 0.179 | | Cash / total assets | 0.171 | | Revenue trend, last 4 quarters | 0.156 | | Gross margin | 0.152 | | Debt / total assets | 0.144 | | Book equity / total liabilities | 0.128 | | Revenue scale (log trailing-12-month revenue) | 0.124 |
Label quality
2,716 Item 1.03 8-Ks of in-scope companies were classified from their text into 1,400 bankruptcy cases.
| Filing class | Filings |
|---|---|
| petition | 1,227 |
| follow_up | 1,030 |
| no_bankruptcy_language | 130 |
| unclear | 126 |
| no_item_section | 93 |
| subsidiary_petition | 41 |
| other_entity | 35 |
| receivership | 34 |
| Case started by | Cases |
|---|---|
| petition | 1,164 |
| follow_up | 159 |
| unclear | 50 |
| receivership | 27 |
Hand review of a fresh random sample (not used for rule changes): 18 of 20 petition filings were the filer's own bankruptcy (errors: a parent filed without this registrant); about half of unclear filings were new insolvencies. Estimated label precision ≈ 88%. Bankruptcies never reported on an Item 1.03 8-K are missing (recall is not measured).
Leakage check
Features are recomputed after deleting every filing, restatement and macro release dated after the as-of date; the output must be identical. It runs in the unit tests (synthetic data, including a planted leak that must be caught) and on real data during every dataset build (results in raw.ingest_runs, source ml_dataset).
Known limitations
- Book values only: no market prices, so Altman Z'' (book-equity variant) is the baseline.
- Universe is SEC XBRL filers since 2009; dominated by small caps.
- Labels rely on Item 1.03 8-Ks; companies that stop filing before bankruptcy may be missed.
- Text-derived risk signals (Phase 5) are not in feature set v1.
- Probabilities are calibrated on 2020, an unusual year; check calibration drift (Phase 7).