Most SaaS companies have a customer health score. Green, yellow, red. A number from 0 to 100, built out of logins, seats, NPS, support tickets, and weights someone picked in a workshop.
Very few of those scores have ever been tested. Not "does it look reasonable." Tested: when the score said an account was red, did that account actually churn more often than the green ones? By how much?
If a health score were a drug, it would need a trial before anyone prescribed it. Here's the trial. It takes about thirty minutes and three queries.
The mistake that ruins the test.
Don't test today's score against past churn.
An account that churned in July has a terrible score today. Of course it does: its usage went to zero when it left. Test current scores against historical churn and the score will look brilliant at predicting churn that already happened.
You need the score as it stood before the outcome. That means point-in-time snapshots: one row per account per week or month, with the score as it was that day.…