Menu

Our two hosts have opposite robots.txt policies, and the one that forbids everything still names a sitemap
📰

Our two hosts have opposite robots.txt policies, and the one that forbids everything still names a sitemap

DEV Community: nextjs·Daniel Pertu·about 22 hours ago
#hzYA4Fw7
#devtonextjs#devto#dev#munchable#host
Reading 0:00
15s threshold

Munchable runs on two hosts. munchable.app is a Next.js site with 516 URLs in its sitemap : a guide per gut condition, a page per ingredient question, recipes, legal pages. app.munchable.app is the product itself, the same Expo app that ships to phones, served on the web through react-native-web.

Both are mine, both are on Vercel, and their robots.txt files say opposite things. You can read both in a terminal in ten seconds, which is the honest way to check a post like this one.

$ curl -s https://munchable.app/robots.txt User-Agent: * Allow: / Disallow: /api/ Disallow: /auth/

Host: https://munchable.app Sitemap: https://munchable.app/sitemap.xml

Enter fullscreen mode

Exit fullscreen mode

$ curl -s https://app.munchable.app/robots.txt # app.munchable.app is the Munchable web app, not a content site.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.