I build Halal Food Scanner, an iPhone and Android app (Expo / React Native) that tells you if a food is halal. You scan the barcode, and when the product isn't in Open Food Facts or has no ingredient list there, you take a photo of the ingredients instead. The app answers Halal, Haram or Uncertain, with the reason for each ingredient.
The photo path is where most of the hard work is. This is what we learned building it, including a bug I shipped myself.
The model reads, a table rules
The first decision was the most important one: the AI does not decide if something is halal.
A photo goes to a vision model in one call. The model has one job: find the ingredient names on the pack and map each one to an id in our own table ("Gelatine" -> gelatin, "E471" / "mono- and diglycerides of fatty acids" -> e471, "Lab" -> rennet).…