Custom robots.txt on Blogger: what the default says and what to change

A wrong robots.txt can remove your blog from search. The default is already sensible, so change it only for a reason.

What robots.txt is

robots.txt is a small text file at the root of a site (https://yourblog/robots.txt). It tells search engine crawlers which parts of the site they may visit. Rules are grouped under User-agent: lines. Disallow: names a path to avoid, and Allow: makes an exception. The file asks crawlers politely; it is not a lock, and it does not hide a page from someone who has the address.

The default on Blogger (checked live)

On 13 November 2026 we fetched /robots.txt from Google's own Blogger blog, which uses the default:

User-agent: Mediapartners-Google
Disallow:
$
User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /
$
Sitemap: https://blogger.googleblog.com/sitemap.xml

Read it line by line. The first group is for the AdSense crawler Mediapartners-Google, which is allowed everywhere (an empty Disallow means nothing is blocked). The second group is for every other crawler: it may not visit /search (the address Blogger uses for label and search result pages) or /share-widget, and may visit everything else. The last line names the sitemap.

Change it

1. Open Settings.
2. In Crawlers and indexing, turn on Enable custom robots.txt.
3. Click Custom robots.txt, paste the text and Save.

Blogger's help says the text will be used instead of the default. So always start from the default above, with your own address in the Sitemap line.

Safe changes

• Keep the lines Disallow: /search and Allow: /.
• Add the sitemap address on its own line, for example Sitemap: https://www.yourdomain.com/sitemap.xml. Use your real address with https, as it appears in your browser. Blogger's pages sitemap is /sitemap-pages.xml (it exists on the blog we checked).
• If you want a particular post hidden, use the "No index" option in that post's settings, not robots.txt (see the header tags page). A blocked page can still appear in results without a description, because the crawler cannot read the no-index instruction.

Mistakes that hide a blog

1. Disallow: / under User-agent: * blocks the whole site for every crawler. It is the most damaging line in robots.txt. Check for it after every edit.
2. Pasting a robots.txt from another blog, with another address in the Sitemap line.
3. Blocking /p/ (pages) or /20 (posts) by accident while trying to block old archives.

Test it

Open https://yourblog/robots.txt in a private window and read it. Use the robots.txt report in Google Search Console to see how Google read it.

This is an independent guide. It is not connected with Google or Blogger. Blogger, AdSense and Google are names of Google LLC. The steps follow Blogger Help and AdSense Help as read on 13 November 2026; the Blogger screens change from time to time, so a menu may be named a little differently on your screen.