Publish a robots.txt file

robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.

Quick Reference

• Serve a valid robots.txt at /robots.txt on the production domain, returning HTTP 200
• Include a Sitemap: directive pointing to your XML sitemap
• Never disallow crawling of CSS/JS assets that render your pages
• Avoid blocking all crawlers with Disallow: / on a live site

Check

Fetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives.

Fix

Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering.