A detection arrives with a score of 0.93. Your SOAR playbook isolates the host because someone set the auto-containment threshold at 0.9. Nobody on the team can say what 0.93 means: 93 percent likely malicious, top 7 percent of events this week, or a sum of weights rescaled to look like a probability.

That gap, between what an ML score is and how your team uses it, is where most evaluations of machine learning in security products never go. The standard checks are well covered elsewhere on this blog: the questions to put to a vendor , how to run the POC , whether the product runs a model at all , and how it holds up against evasion . This framework covers four properties those checks leave out, the ones that decide whether the model is still trustworthy six months after signing.

1. Calibration: Does the Score Mean What Your Playbook Assumes?

A model can rank alerts well and still produce scores that are useless as probabilities. Guo et al.…