This was my final project for IBM's Supervised Machine Learning: Classification course: predict which telecom customers will leave, using the IBM Telco Customer Churn dataset (7,043 customers, 26.5% of whom churned). The modelling was the easy part. The interesting decision came at the end, when the best three models turned out to be indistinguishable.

A fair comparison first

Every model family was grid-searched on the same five stratified folds of the same 80/20 split, scored by ROC AUC, and then evaluated once on the held-out test set with a 95% confidence interval.

Model CV AUC Test AUC 95% CI

Gradient boosting 0.849 0.846 0.825–0.868

Random forest 0.849 0.843 0.821–0.865

Logistic regression (L2, C = 0.1) 0.844 0.843 0.821–0.865

SVM, RBF kernel 0.844 0.843 0.819–0.865

k-nearest neighbours 0.838 0.832 0.808–0.854

Decision tree (pruned) 0.839 0.831 0.807–0.854

Gradient boosting has the highest number, but its lead over logistic regression is 0.003 AUC, well inside overlapping confidence intervals.…