Munchable runs on two hosts. munchable.app is a Next.js site with 516 URLs in its sitemap : a guide per gut condition, a page per ingredient question, recipes, legal pages. app.munchable.app is the product itself, the same Expo app that ships to phones, served on the web through react-native-web.
Both are mine, both are on Vercel, and their robots.txt files say opposite things. You can read both in a terminal in ten seconds, which is the honest way to check a post like this one.
$ curl -s https://munchable.app/robots.txt User-Agent: * Allow: / Disallow: /api/ Disallow: /auth/
Host: https://munchable.app Sitemap: https://munchable.app/sitemap.xml
Enter fullscreen mode
Exit fullscreen mode
$ curl -s https://app.munchable.app/robots.txt # app.munchable.app is the Munchable web app, not a content site.…