Public mortgage application data / Academic audit
When should a mortgage model withhold a prediction?
I evaluated a model of historical mortgage approval and denial outcomes, compared its behavior across groups and proposed reserving uncertain predictions for human review. The model predicts recorded decisions, not creditworthiness.
- Context
- Academic analysis of public 2024 Home Mortgage Disclosure Act records.
- Problem
- Understand model performance, subgroup differences, and uncertain predictions before proposing a use.
- My contribution
- Evaluation on reserved data, feature and subgroup checks, and review-band design.
- Result
- An audit of about 1.71 million records reserved for evaluation and a proposed review workflow.
What the model predicts
Using public Home Mortgage Disclosure Act (HMDA) data, I audited a gradient-boosted classifier trained to predict a historical approval-versus-denial outcome. Reproducing that label does not establish whether the original decision was appropriate or whether a borrower would repay.
I evaluated the model on held-out data: application records reserved for testing rather than training. I checked its inputs, compared its behavior across groups and examined where uncertainty would make an approval-or-denial prediction misleading.
Explaining the audit methods
The model is a gradient-boosted classifier trained on a pruned set of 21 features. Protected attributes are excluded from training, while the audit still examines proxy risk. SHAP explanations describe how inputs contribute to model predictions; they do not establish causes. Population Stability Index comparisons examine differences between data distributions. Subgroup screening compares selected groups within the audit population; its findings depend on the data and grouping choices.
Explaining the audit methods
The model is a gradient-boosted classifier trained on a pruned set of 21 features. Protected attributes are excluded from training, while the audit still examines proxy risk. SHAP explanations describe how inputs contribute to model predictions; they do not establish causes. Population Stability Index comparisons examine differences between data distributions. Subgroup screening compares selected groups within the audit population; its findings depend on the data and grouping choices.
Reserving uncertain predictions for human review
The chart below groups evaluation records by model score. The middle band, from 0.25 to below 0.75, contains 259,343 applications, about 15% of the total. Its average score and historical approval rate both sit near the middle. I proposed reserving this group for human review, without an approval-or-denial prediction.
Under the proposed workflow, predictions would be provided for about 85% of the 1.71 million evaluation records. Among that 85%, about 8% of predictions disagreed with the recorded historical decision. That disagreement rate applies only to records for which the workflow would provide a prediction.
Original audit figure: held-out application counts, historical approval rates, and mean model scores across five score bands. Held-out records were reserved for evaluation.
View full-size score-band chartCoverage of the proposed review workflow
About 1.71 million held-out records
- Prediction provided
- About 85%
- Reserved for human review
- About 15%
About 8% disagreement with the historical decision among records receiving a prediction.
What the audit established
The subgroup screen found no groups below its 0.80 screening threshold and flagged six groups in the 0.80–0.90 caution zone for monitoring. Those findings depend on the chosen population and grouping; they do not establish that the model is fair.
The result is an academic audit and a proposed way to prioritize review. It has not been deployed for credit decisions.