Skip to content
+91 89607 57615
Code Craft — web development and digital marketing
Lesson 726 min

Sitemaps and robots.txt

Tell Google which pages exist and where it may crawl, without accidentally hiding important pages.

Taught by Sandesh Sahetiya

The video is part of the full course

In this lesson you will

  • Write a correct robots.txt
  • Know what robots.txt can and can't do
  • Build an XML sitemap Google can use
  • Keep pages out of results with noindex, properly

robots.txt

robots.txt is a plain text file at the root of your site, such as https://www.example.com/robots.txt, that tells crawlers which paths they may request. It manages crawling, mainly to avoid wasting your server's resources. Each host and protocol needs its own file, and rules are case-sensitive.

site-before/robots.txt: a real-world mistakeDownload
User-agent: *
# Meant to block the unfinished /about-draft/ folder, but rules match by
# prefix, so this also blocks /about.html.
Disallow: /about

Sitemap: http://127.0.0.1:8080/sitemap.xml

Rules match from the start of the path, so Disallow: /about blocks /about-draft/ and also /about.html. The fix is to block the exact folder, with its trailing slash:

site-after/robots.txtDownload
User-agent: *
Disallow: /about-draft/

Sitemap: http://127.0.0.1:8080/sitemap.xml

Continue with the full course

The rest of this lesson, the narrated video, the PDF and the exercise are part of SEO Fundamentals: On-page, Technical and Local. One payment, lifetime access.

Already bought it? Sign in with the email you paid with.