
The video is part of the full course
In this lesson you will
- Write a correct robots.txt
- Know what robots.txt can and can't do
- Build an XML sitemap Google can use
- Keep pages out of results with noindex, properly
robots.txt
robots.txt is a plain text file at the root of your site, such as https://www.example.com/robots.txt, that tells crawlers which paths they may request. It manages crawling, mainly to avoid wasting your server's resources. Each host and protocol needs its own file, and rules are case-sensitive.
User-agent: *
# Meant to block the unfinished /about-draft/ folder, but rules match by
# prefix, so this also blocks /about.html.
Disallow: /about
Sitemap: http://127.0.0.1:8080/sitemap.xmlRules match from the start of the path, so Disallow: /about blocks /about-draft/ and also /about.html. The fix is to block the exact folder, with its trailing slash:
User-agent: *
Disallow: /about-draft/
Sitemap: http://127.0.0.1:8080/sitemap.xmlContinue with the full course
The rest of this lesson, the narrated video, the PDF and the exercise are part of SEO Fundamentals: On-page, Technical and Local. One payment, lifetime access.
Already bought it? Sign in with the email you paid with.
