Blocked by robots.txt: What Google's Message Means and When to Fix It

Is your whole site blocked, or only a few addresses?

The quickest way to rule out the one case that matters most is the checker below. Paste your homepage address and it reads your robots.txt the way Google's crawler does, taking the group of rules meant for Googlebot, and says whether the file shuts out the whole site. It also checks the homepage's noindex setting, its canonical, the sitemap and, where it can tell, the platform the site is built on, which tells you which steps below are yours. No email is asked for.

It checks the homepage and the whole-site rule, not every page. To check one address, such as a product or service page, use URL Inspection in Search Console, shown in the next section.

What Blocked by robots.txt means

Google works in two steps. First it crawls a page: its robot visits and reads it. Then it decides whether to index the page, which means keeping it so it can appear in results. robots.txt is a plain text file at the root of a site, such as yourbusiness.com/robots.txt, that asks crawlers to skip certain addresses. Blocked by robots.txt is Google reporting that it did as asked: it skipped those addresses, so it has nothing to add.

Google's help for the Page indexing report is relaxed about it: "URLs blocked by robots.txt are probably intentionally blocked. Use your judgment when deciding whether to address a given issue." The same page says "It's fine for a URL not to be indexed for the right reasons," and names an expected robots.txt rule as one of them. The Google help pages quoted here were all re-read on October 19, 2026.

Seeing the row also tells you something good: Google has been crawling your site and found these addresses. If it arrived as a line in Google's email New reasons prevent pages from being indexed, the guide to that email reads the whole message line by line. This page covers the one reason.

Where to find the blocked addresses

1. Open Search Console and pick your site. If you have not set it up yet, the site verification guide covers the one record Google asks for.
2. In the left menu, click Indexing, then Pages, and scroll to Why pages aren't indexed.
3. Click the row Blocked by robots.txt. Its Source column says Website, which means the rule lives on your site, not at Google.
4. Read the example addresses Google lists under it, with the date of its last visit to each. Most of the time the pattern is clear at a glance.
5. To check one address, paste it into the URL Inspection box at the top of Search Console and open Page indexing. Crawl allowed? should read Yes.

Sort each address with three questions

Ask these in order for each address under the row. Most addresses stop at the first one.

1. Is it a cart, checkout, account, admin, search, filter or preview address? Then your platform blocks it on purpose. Leave it, and do not click Validate fix for it. The table in the next section lists what each platform blocks.
2. Did the site move to a new platform in the last few days? Google may still be working from the old site's file, even for the homepage. Request a recrawl of robots.txt, as shown below, and look again the next day before changing anything. If addresses you want found are still listed then, go on to the next question.
3. Is your homepage on the list, or nearly every page? Then the block likely covers the whole site. Open yourbusiness.com/robots.txt: a Disallow: / line in the group that starts User-agent: * (or User-agent: Googlebot) confirms it. The same line under another crawler's name, such as PetalBot in Wix's default file, does not apply to Google. Fix it the same day, using the steps for your platform below.

Which blocked addresses are normal on each platform

Every major platform ships its own robots.txt file, and each one blocks a short list of addresses on purpose. Google's own help for this status stops short here: "we can't provide exact guidance how to update your robots.txt file because every hosting service has its own way to do this." So the table does it. It was built from live files read on October 18 and 19, 2026: Shopify stores, Squarespace sites, Wix sites on their own domains and GoDaddy Website Builder sites, plus the code WordPress uses to write its file and the lines WooCommerce adds to it. If an address in your report matches a row, it is meant to be there.

Not sure which platform your site runs on? The free platform checker names it in a few seconds. Of 6,861 small business sites whose owners ran Theo's free audit (August 2 to September 30, 2026), 24 percent ran on Shopify, 14 percent on WordPress, 6 percent on Wix, 4 percent on Squarespace and 4 percent on GoDaddy Website Builder (the research), so the steps further down follow that order.

When a block needs a fix

Three situations are worth your time. Each one shows up in a way you can spot without reading code.

• The whole site. Your homepage is on the list, or nearly every page is, and the checker above says the file blocks the whole site. Open yourbusiness.com/robots.txt in a browser: a line Disallow: / under User-agent: * (or under User-agent: Googlebot) asks Google to read nothing.
• One page customers should find. A service, product, collection or blog page is listed. On the platforms in the table, their default files never block those pages, so a rule was added by hand, by an add-on or by whoever built the site (on Wix, a page setting can also add one).
• Pages your sitemap lists. Set the filter above the Page indexing chart to All submitted pages. If Blocked by robots.txt still shows rows, your sitemap and your robots.txt disagree: one asks Google to find those pages and the other asks it to skip them. Keep the page and unblock it, or take it out of the sitemap. The Sitemaps report shows the same disagreement under Google's label Sitemap contains urls which are blocked by robots.txt, and Google's Sitemaps report help says to confirm the address in URL Inspection, then change robots.txt to let Googlebot in.

What the checker says in each case

Run on theo.site on October 18, 2026, the checker at the top of this page answered "Allows Google in. It names a sitemap." on its robots.txt line. A file that shuts out the whole site gets a different line: "Blocks the whole site from Google. Remove the 'Disallow: /' line that applies to Googlebot." A site with no robots.txt at all gets "No robots.txt answered. That is allowed; Google treats a missing file as no restrictions."

How to fix Blocked by robots.txt on your platform

Find your platform below. Each one keeps the file in a different place, and on most of them a setting writes it for you. When you have made the change, finish with the Search Console clicks in the section after these. GoDaddy Website Builder's default file blocks only its /404 page; if a GoDaddy file blocks more than that, GoDaddy support can say which setting wrote it.

A rule does not have to name your page to block it. It covers every address that starts with what it lists, so Disallow: /services also blocks /services/drain-cleaning, and a * in a rule stands for anything at that spot. Google's help says: "Use a robots.txt validator to find out which rule is blocking your page." Paste the page's address into one, and it shows the line that blocks it. After the edit, Test live URL in URL Inspection confirms Google can reach the page.

Just moved your website? Google may be reading the old file

In one Theo support case, an owner who had just moved a site off WordPress found Search Console still reporting Blocked by robots.txt. Google was still working from the old site's file. Nothing on the new site was blocking anything.

That lag is normal. Google's robots.txt documentation says it "generally caches the contents of robots.txt file for up to 24 hours," and longer if it cannot fetch a fresh copy. A missing file is fine too: for a robots.txt address that answers 404, Google says "Google assumes that there are no crawl restrictions."

1. In Search Console, click Settings, then Open Report next to robots.txt. The report needs a Domain property or a property for the whole site, not a single folder.
2. Find your main address in the list, such as https://yourbusiness.com/robots.txt.
3. Click the three-dot menu next to it and choose Request a recrawl.
4. The next day, check Checked on and Status in the same report, then run Test live URL on your homepage in URL Inspection.

Indexed, though blocked by robots.txt: what it means and how to fix it

This is a different status, and it sits in a different place: the Improve page experience table of the Page indexing report, not the list headed Why pages aren't indexed. Google explains: "Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page." Google never read the page, but a link elsewhere told it the address exists, so the address can appear in results with little or no description.

The fix depends on the page, and there are three answers. Google is clear about the last one: "robots.txt is not the correct mechanism to avoid being indexed." Its URL Inspection help explains why: while a page is blocked by robots.txt, Indexing allowed? "will always be 'Yes' because Google can't see and respect any noindex directives."

• A platform address, such as a cart link, store search or a filtered collection: leave it. A bare address with no description does no harm, and Shopify's checkout and cart rules stay as they are.
• A page you want customers to find: unblock it with the steps for your platform above, then request a recrawl of robots.txt and request indexing for that page.
• A page you want off Google: turn on its noindex setting first (most builders call it hide from search engines), then remove the robots.txt block so Google can read that setting; the noindex guide has each platform's switch. For anything private, use a password instead.

After the fix: what to click, and the two clocks

Run these once, after a real change. Skip them for the addresses your platform blocks on purpose.

1. Request a recrawl of robots.txt in Settings, as shown above. This is the first clock: Google generally refreshes its copy of the file within a day.
2. Test the page. Paste the address into URL Inspection and click Test live URL. Crawl allowed? Yes in the live test means the block is gone.
3. Request indexing for the one or two pages that matter most. URL Inspection has a daily limit, so save it for those.
4. Click Validate fix on the Blocked by robots.txt row once, and only if every address under it was a block you have now removed. If the row also lists addresses your platform blocks on purpose, skip it: in Google's words, "validation will stop when Google finds a single remaining instance of that issue." The Validate fix timeline on the email guide shows what happens after you click.
5. Wait for the second clock. The rows clear as Google revisits each address on its own schedule, which takes days to weeks. A page you just unblocked can show up sooner after step 3, and how to get Google to index your site covers what else helps a page get picked up.

Common advice online that does not fit this status

A few tips that circulate about this label are out of date or answer a different question. Here is what holds on October 19, 2026.

• "Use the robots.txt Tester." Google has replaced it with the robots.txt report under Settings and the live test in URL Inspection.
• "The WordPress visibility box writes Disallow: /." Not since WordPress 5.3. The box adds a noindex tag, which shows up as Excluded by 'noindex' tag.
• "Shopify's robots.txt cannot be changed." It can, through the theme's robots.txt.liquid file, though the default suits most stores. No proxy or extra service is needed.
• "Add a Noindex: line to robots.txt." Google's robots.txt rules support four fields: user-agent, allow, disallow and sitemap. A noindex line does nothing; use the page's own noindex setting.
• "Block a page in robots.txt to keep it off Google." Google's introduction to robots.txt says the file "is not a mechanism for keeping a web page out of Google." Use noindex or a password.
• "Click Validate fix on every row." Only after a real change. Addresses blocked on purpose stay listed, and that is fine.

Can Theo handle this for you?

Owners who forward Google's indexing emails to Theo often ask exactly that. On a site Theo builds, yes: the robots.txt is part of the site, it lets search engines in, and nothing in it is for the owner to manage.

On Starter and Pro, the Growth Engine's SEO repair goes further. It pulls issues from Google Search Console and a direct check of the site, inspecting the homepage, the top pages and recently changed pages with URL Inspection and reading the sitemap's status. When a robots or noindex rule is keeping out a page that should be found, it drafts the fix, and the fix waits in the Growth tab for your review unless auto-publish is on. Rules that block checkout or admin addresses on purpose are left alone.

In place of page-by-page requests, Theo submits the site's sitemap when Search Console is connected, and Google revisits on its own schedule. If your site is somewhere else, the free audit reads it with no login and shows what Google can and cannot reach.

Frequently asked questions