Indexed but Not Crawled: Should I Be Concerned?

2 responses to “A page is indexed but not crawled – should this be a concern?”

1. When it comes to pages that are indexed but blocked from being crawled, understanding the implications is crucial, especially from an SEO perspective. Let’s break down the situation:

   1. Understanding Indexing vs. Crawling:

   • Indexing: This is when search engines include a page in their database to show in search results.
   • Crawling: This is the process by which search engines discover the content on a page by following links and reading the content.

   2. Your Situation:

   • Blocked by robots.txt: You have used the robots.txt file to block certain pages from being crawled by search engines. This is a common practice for pages that contain duplicate content, are under development, or are not relevant for search engine results.
   • Indexed but not crawled: This means the search engine knows about the page (and may show it in search results), but it cannot retrieve or understand its content because it cannot crawl it.

   3. Is It a Concern for SEO?

   • Relevance: Only you can truly assess if these pages should be visible in search results. You mentioned they are login pages, which typically do not need to be indexed due to their lack of value to search users and potential privacy concerns.

   • Crawl Efficiency: Blocking crawling but allowing indexing can lead to inefficient allocation of the search engine’s crawl budget, potentially taking resources away from more important pages.

   • User Experience: Indexed but non-crawlable login pages can appear in Google search results, leading users to dead ends or confusion when clicking through to login or restricted access pages.

   4. Recommendations:

   • Meta Robots Tag: If you wish to prevent a page from appearing in search engine results altogether, use the noindex directive via a meta robots tag in the HTML of those pages:

   html
<meta name="robots" content="noindex">

   • Adjust Robots.txt: While robots.txt controls crawling, it’s also advisable to simultaneously ensure that less important pages are not indexed at all, using the above tag.

   • Understand Your Crawl Budget: Particularly if you have a large site, ensuring that search engines aren’t wasting resources on unimportant pages is key. Focus on allowing indexing and crawling for pages of unique, relevant, and high-quality content.

   • Regular Monitoring: Keep

   Reply

2. This is a great topic to discuss! It’s important to remember that while a page being indexed but not crawled may not be an immediate concern, it does have implications for your overall SEO strategy.

   When search engines discover a URL, they can index it even if they don’t have the ability to crawl and see its content, which is often the case for pages blocked by `robots.txt`. In your scenario where these are login pages, it’s likely they’re not hurting your site’s SEO significantly since they aren’t vital for user navigation or content visibility. However, ensuring that only necessary pages are indexed is crucial.

   One potential concern could arise if search engines start associating your site with a higher number of irrelevant indexed pages, which might dilute the authority of your valuable content. Additionally, it’s a good idea to periodically review your `robots.txt` settings and consider if there are any critical pages inadvertently getting blocked that you want indexed.

   You might also want to explore alternative methods, such as using a `` tag for those specific pages, which clearly communicates to search engines to not index these pages while still allowing them to crawl.

   Overall, while it’s good to keep an eye on these types of issues, focusing on high-quality, relevant content will always be the best SEO strategy in the long run!

   Reply