robots.txt valid

What this means

https://YOUR_DOMAIN/robots.txt returns 200 with a text/plain content type and contains a Sitemap: line pointing at your sitemap. That file is the first thing every crawler reads, and the sitemap line is how they find the rest of your site. Failing means the file is missing, the host serves an HTML page in its place, or no Sitemap: line is present.

How we test it

We request /robots.txt on your domain, following at most 3 redirects. We require a 200 status, a Content-Type of text/plain, and at least one line beginning Sitemap: with an absolute URL. All three present passes; a 404, an HTML content type, or a missing Sitemap: line fails and we say which.

Why it matters

A missing or broken robots file blocks or mis-directs crawlers. A host that answers /robots.txt with its 404 page at status 200 hands the crawler HTML it cannot read, and a blanket Disallow: / left over from a template or a staging config removes the whole site from search while every page still returns 200. The Fix Kit treats the sitemap line in robots.txt as a precondition for the rest of its search setup, and sends you back to it if the line is missing.

The fix

Place a robots.txt at the web root (static site: the root folder; Next.js: public/robots.txt) based on templates/robots.txt, with User-agent: *, Allow: /, your own Disallow: lines for private paths such as /api/, /admin, /account and /login, and Sitemap: https://YOUR_DOMAIN/sitemap.xml. Then confirm with curl -s https://YOUR_DOMAIN/robots.txt | grep -i "^Sitemap:" that the absolute sitemap URL is served. The full runbook is in the Fix Kit, Module 02.

Prompt for your AI

Paste this into the coding agent that built the app. It reproduces the finding and stops before changing anything; the fix steps are in the kit.

Do not change anything yet. This is a verification pass only.
Run: curl -sIL --max-redirs 3 https://YOUR_DOMAIN/robots.txt | grep -iE "^(HTTP|content-type)"
Run: curl -sL --max-redirs 3 https://YOUR_DOMAIN/robots.txt
Print the full file and tell me whether the status is 200 and the content type is text/plain.
Run: curl -s https://YOUR_DOMAIN/robots.txt | grep -i "^Sitemap:"
Tell me whether a Disallow: / line applies to User-agent: * and whether it is intentional.
Find the file in the repo that produces robots.txt and show me its path and contents.
List what it would take to serve a valid robots.txt with a Sitemap: line, without editing anything.
Verify: the grep prints a Sitemap: line with an absolute https URL; if it prints nothing, report that and stop.

Copy the prompt