I Discovered That a Cancer Prediction AI Was Using the Hospital Name — Not the Tumor — To Make Its Diagnosis. So I Built a Tool to Catch It.

Built for the Build with MeDo Hackathon · #BuiltWithMeDo

The Moment That Changed Everything

Last week I was studying how machine learning models are deployed in hospitals. I stumbled across a research paper that stopped me cold.

A cancer prediction model — the kind that tells doctors whether a patient needs aggressive treatment — was tested across multiple hospitals. It worked brilliantly at Hospital A. Terrible at Hospital B. Same cancer. Same biology. Different results.

Why?

The model had learned to use hospital_site_id as its strongest predictor. Not tumor size. Not mitotic count. Not any clinical feature that a doctor would recognize. The AI had essentially memorized which hospital a patient came from and used that to predict cancer outcomes.

This is called proxy bias , and it's one of the most dangerous failure modes in clinical AI.…