robots.txt fileWhat it checks and how to fix it

What it checks

Fetches /robots.txt from the site root. It fails when the file does not answer 200, is empty, or lists a Sitemap: URL that is unreachable or not a valid sitemap; it warns when the file has no User-agent line; and it passes when the file is present, has User-agent rules and every listed sitemap is valid.

Why it matters

robots.txt is the first file crawlers request, and it tells them what they may crawl and where your sitemap lives. A missing file is tolerated, but a broken one or a Sitemap: line pointing at a dead URL sends crawlers to the wrong place. A malformed file can also be read as "disallow everything", which removes the site from search.

How to fix it

Serve a plain-text robots.txt at the root of the domain with at least one User-agent group and the absolute URL of your sitemap:

User-agent: *
Disallow: /admin/
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

In Rails the file lives at public/robots.txt and is served as-is. With nginx or Apache, put it in the document root and make sure it is served as plain text:

location = /robots.txt {
  root /var/www/example/public;
  default_type text/plain;
}

Open every Sitemap: URL in a browser to confirm it returns XML with status 200. On Cloudflare, make sure no WAF or bot rule blocks /robots.txt for crawlers.