Three related statuses in Search Console's Page indexing report all come down to instructions your site gives Google. Blocked by robots.txt means Google wasn't allowed to crawl the URL. Indexed, though blocked by robots.txt means Google indexed it anyway, without reading it. Excluded by 'noindex' tag means Google read the page and found an instruction not to index it.
Sometimes these are exactly what you want, because admin pages, cart pages and internal search results usually should be blocked or noindexed. The trouble starts when a rule catches pages you want in search, often after a site launch, a theme change or a plugin update, or when people mix the two tools and get the opposite of what they meant.
The key idea is simple: robots.txt controls crawling. noindex controls indexing. Google has to crawl a page to see its noindex tag, so blocking a page in robots.txt can actually stop the noindex from working. This guide shows how to tell which situation you're in and fix each one.
Quick answer
- Open Indexing > Pages, click each status, and decide for every URL pattern: should this be in Google or not?
- Want it indexed but it's blocked? Remove or narrow the
Disallowrule in robots.txt, then request a recrawl of robots.txt in Settings > robots.txt. - Want it indexed but it has noindex? Remove the
noindexmeta tag orX-Robots-Tagheader in your CMS, plugin or server. - Want it out of Google but it's "Indexed, though blocked"? Unblock it in robots.txt and add
noindex, so Google can crawl it and see the instruction. - Run URL inspection > Test live URL, request indexing for key pages, then validate the fix.
Labels in the report can vary. Google's help page calls the noindex reason "URL marked 'noindex'", while the report itself commonly shows Excluded by 'noindex' tag.
Before you start
You need Owner or Full user access in Search Console. For the robots.txt report, you need a Domain property or a URL-prefix property at the site root.
You also need a way to edit robots.txt and page-level SEO settings. On WordPress that's usually your SEO plugin. On Shopify it's the robots.txt.liquid theme template. Squarespace and Wix generate robots.txt for you and give you page-level options to hide pages from search. On a custom site, robots.txt is a plain text file at the root of the domain.
Back up first. Open https://www.example.com/robots.txt in a browser and save a copy of the text. One wrong line can block a whole site, so you want an easy way back.
Step 1: Sort each URL into "keep out" or "should be indexed"
Go to Indexing > Pages and look at the Why pages aren't indexed table. Click each relevant reason and scan the example URLs.
- These reasons have the source Website: they come from your own robots.txt or page code.
- Indexed, though blocked by robots.txt shows up as a warning on indexed pages rather than in the not-indexed table.
Make a simple list, and for each URL pattern, write "keep out" or "should be indexed". Typical results:
| URL pattern | Usually |
|---|---|
/wp-admin/, /cart/, /checkout/, /account/ |
Keep out |
Internal search (/?s=, /search) |
Keep out |
| Thank-you and confirmation pages | Keep out |
| Staging or test pages on the live domain | Keep out |
| Service, product, blog and landing pages | Should be indexed |
| CSS, JavaScript and image files | Must be crawlable (don't block) |
If everything on the list is a "keep out" page, there's nothing to fix, since these statuses are informational. Move on once you've confirmed it.
Step 2: Check robots.txt in Search Console
Open Settings (the gear at the bottom of the left menu) and click Open report next to robots.txt. The report shows each robots.txt file Google found for your hosts, when it was last checked and whether it was fetched.
- Fetched means Google read the file. A 404 just means no file exists for that host, which allows everything.
- Read the rules Google is actually using. This version can lag behind your live file.
Then test specific URLs. Paste an important blocked URL into URL inspection and click Test live URL. If Crawl allowed? says No: blocked by robots.txt, the live file blocks it. The legacy robots.txt Tester tool is no longer part of Search Console, so use URL inspection for per-URL checks.
Step 3: Fix "Blocked by robots.txt" for pages you want indexed
Find the Disallow line that matches the URL. Remember how Google reads rules: the most specific (longest) matching rule wins, and if an Allow and a Disallow are equally specific, the less restrictive Allow wins. * matches any characters and $ marks the end of a URL.
Common mistakes and their fixes:
User-agent: *
Disallow: /
That pair blocks the entire site, and it's often left over from a staging setup. Change it to an empty Disallow: or remove it.
User-agent: *
Disallow: /services
Without a trailing slash, this blocks /services, /services/, /services-pricing/ and anything else starting with those characters. Remove it, or narrow it to the exact folder you meant.
User-agent: *
Disallow: /*?
This blocks every URL with a query string, which can include paginated or filtered pages you want crawled. Replace it with rules for specific parameters, such as Disallow: /*?*sort=.
A safe starting point for many small business sites:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
Where to edit it
- WordPress: many SEO plugins (Yoast, Rank Math and others) include a robots.txt editor under their tools or general settings. If there's a physical
robots.txtfile in your site root, it overrides the virtual one, so edit that file through your host's file manager. - Shopify: edit the
robots.txt.liquidtemplate in your theme code. Shopify's default rules block checkout and cart pages, which is fine, so only customize if you need to. - Squarespace and Wix: both manage robots.txt for you. If a page is blocked unexpectedly, check page-level SEO settings and site visibility instead.
- Custom sites: edit the file at the web root and deploy.
After saving, open the live file in a browser to confirm the change. Then return to the robots.txt report, open the menu next to the file and choose Request a recrawl. Google normally caches robots.txt for up to 24 hours, so the recrawl request helps it pick up urgent changes faster.
Step 4: Fix "Excluded by 'noindex' tag" for pages you want indexed
The page itself is telling Google not to index it, and the instruction lives in one of two places.
In the HTML, inside the <head>:
<meta name="robots" content="noindex, follow">
<!-- or targeting only Google -->
<meta name="googlebot" content="noindex">
In the HTTP response headers, which you can't see in the page source:
X-Robots-Tag: noindex
To check both, inspect the URL in Search Console and click Test live URL. If Indexing allowed? says No: 'noindex' detected in 'robots' meta tag or mentions the X-Robots-Tag header, you've found it. Click View tested page > HTML to see the exact tag Google received.
- This line names the source: the robots meta tag or the
X-Robots-Tagheader. - View tested page shows the exact HTML Google received, so you can find the tag.
Now remove it where it's set:
| Platform | Where noindex usually comes from |
|---|---|
| WordPress (site-wide) | Settings > Reading > Discourage search engines from indexing this site is checked |
| WordPress (per page) | SEO plugin box > Advanced > "allow search engines to show this page" set to No |
| WordPress (by type) | SEO plugin settings that noindex whole post types, categories, tags or archives |
| Squarespace | Page settings > SEO > hide page from search results, or site-wide visibility settings |
| Shopify | Theme code or an SEO app adding a robots meta tag |
| Wix | Page SEO settings that turn off indexing for that page |
| Any site | A staging, coming-soon or maintenance mode plugin |
| Server or CDN | An X-Robots-Tag header rule in Apache, Nginx, Cloudflare or your host |
The WordPress site-wide checkbox is a very common cause right after a launch, so check it first.
- Untick Discourage search engines from indexing this site under Search engine visibility.
- Click Save Changes, then clear any caching plugin so the old HTML stops being served.
To check headers from the command line:
curl -sI https://www.example.com/services/seo-audit/ | grep -i x-robots
If you see X-Robots-Tag: noindex, look in your server config, .htaccess file, CDN rules or a security plugin. It's common on staging environments that were promoted to production.
Labels in platform menus change often, so check your platform's help center if the option has moved.
Step 5: Fix "Indexed, though blocked by robots.txt"
This one confuses people. In short, Google found links to the URL, couldn't crawl it because of robots.txt, and indexed the bare URL anyway. In search it may appear with no description, or a note that no information is available.
Decide what you want for each URL before you change anything:
- You want it indexed properly. Remove the robots.txt rule (Step 3), then request indexing. Google will crawl the content and replace the bare listing.
- You want it out of Google. Remove the robots.txt block and add
noindexto the page (Step 4 in reverse). Google has to crawl the page to see thenoindex. Leave it crawlable; if you re-block it later, Google can no longer see thenoindex.
Here's the wrong setup. Google can't see the noindex, so the URL stays indexed:
User-agent: *
Disallow: /thank-you/
Delete that Disallow line and put the tag on the page instead:
<!-- On /thank-you/ -->
<meta name="robots" content="noindex">
For something urgent, like private data, use the Removals tool in Search Console to hide the URL temporarily while the noindex takes effect. Removals only last about six months, so the noindex (or password protection) is what keeps it out for good. Google's own docs say robots.txt is not a way to keep a page out of search.
Still not working? Common causes
| Symptom | Likely cause | Fix |
|---|---|---|
| Whole site shows Blocked by robots.txt | Disallow: / left over from staging |
Remove the rule; request robots.txt recrawl |
| Pages blocked after a redesign | New theme or plugin added a robots.txt rule | Compare with your backup; restore the right rules |
| robots.txt edits don't show up | A physical file overrides the plugin's virtual one, or a CDN caches the old file | Edit the physical file; purge CDN cache |
| Live test still shows noindex after removal | Page cache or CDN serves the old HTML | Clear the site cache and CDN; retest |
| Noindex on every page | WordPress Discourage search engines box is checked | Uncheck it under Settings > Reading |
| Noindex not in source but still reported | X-Robots-Tag HTTP header |
Check headers with curl; remove the server or CDN rule |
| Category or tag pages noindexed | SEO plugin taxonomy settings | Allow indexing only for the taxonomies you want in search |
| Page stays indexed after adding noindex | Robots.txt still blocks it, so Google can't see the tag | Unblock it in robots.txt |
| Pages render blank in the live test | CSS or JS folders blocked in robots.txt | Allow theme, plugin and asset folders |
| robots.txt returns 5xx | Server error on the file | Fix it fast; Google stops crawling for a period when it can't fetch robots.txt |
Don't use robots.txt to remove duplicates
Blocking duplicate or parameter URLs hides their canonical tags from Google. Use canonicals instead, as explained in our guide on duplicate and canonical issues.
Watch out for robots.txt errors
If your robots.txt file returns a server error, Google's docs say it stops crawling the site for the first 12 hours and then relies on the last good copy for a while. A file that returns 404 is treated as "no restrictions". So a broken robots.txt is worse than a missing one.
How to confirm it's fixed
- Open the live robots.txt in a browser and confirm the change is there, with no CDN serving an old copy.
- Check the robots.txt report in Settings. The Checked on date should be after your edit. If not, use Request a recrawl and check again the next day.
- Run URL inspection > Test live URL on several affected pages. You want Crawl allowed? Yes and Indexing allowed? Yes.
- Request indexing for your most important pages, then click Validate fix on the reason in Indexing > Pages.
- Wait. Validation usually takes up to about two weeks but can take longer. Pages Google rarely visits can take longer to be recrawled.
- For pages you wanted removed, re-inspect them after a recrawl. They should show Excluded by 'noindex' tag instead of Indexed, though blocked by robots.txt.
If new pages still aren't indexed after the blocks are gone, the cause may be elsewhere. Start with our guide to getting Google to index your site.
FAQ
Should I block pages in robots.txt or use noindex?
Use noindex to keep a page out of search results. Use robots.txt to stop crawling of URLs that waste crawl time and have no search value. Don't use both on the same page.
Can I put noindex in robots.txt?
No. Google doesn't support noindex rules inside robots.txt, and it stopped honoring unofficial ones years ago. Use the meta tag or the X-Robots-Tag header.
How long until Google sees my robots.txt change?
Google generally caches robots.txt for up to 24 hours. Requesting a recrawl in the robots.txt report can speed that up, but it isn't guaranteed.
Is "Excluded by 'noindex' tag" bad?
Only if the page should be in search. For thank-you pages, logins and internal search, it's exactly right.
Should I block CSS and JavaScript files?
No. Google needs them to render your pages properly. Blocking them can make pages look broken or empty to Google.
Stuck on this one? We fix these for clients every week. Book a free audit
