Special Issue 19: Crawling & Indexing 1st April 2025

Thought Pieces

Technical SEO: Don’t rush the process

In an era where efficiency is key, many businesses question the time and resources spent on technical SEO audits.

However, cutting corners in this critical area can lead to incomplete insights and missed opportunities.

Let’s dive into why technical SEO deserves a firm investment in both human effort and time, starting with the often-overlooked challenge of crawl time.

Google's Mueller Predicts Uptick Of Hallucinated Links: Redirect Or Not?

AI tools are creating fake URLs causing 404 errors. Google's John Mueller offers guidance to navigate this issue.

AI creates fake URLs that lead to 404 errors.
Google's Mueller predicts this will become more common.
He advises focusing on more meaningful SEO metrics.

Google On Search Console Noindex Detected Errors

Google’s John Mueller answered a question on Reddit about a seemingly false ‘noindex detected in X-Robots-Tag HTTP header’ error reported in Google Search Console for pages that do not have that specific X-Robots-Tag or any other related directive or block. Mueller suggested some possible reasons, and multiple Redditors provided reasonable explanations and solutions.

Google’s Advice on Fixing Unwanted Indexed URLs

An SEO posted details about a site audit in which he critiqued the use of a rel=canonical for controlling what pages are indexed on a site. The SEO proposed using noindex to get the pages dropped from Google’s index and then adding the individual URLs to robots.txt. Google’s John Mueller suggested a solution that goes in a different direction.

Google's Martin Splitt Warns Against Redirecting 404s To Homepage

Google has released a new episode in its “SEO Office Hours Shorts” video series, in which Developer Advocate Martin Splitt addresses a question many website owners face: Should all 404 error pages be redirected to the homepage?

Events

SEO Tools Demo Day · Saturday, April 12

Event Description: Tool Showcase for SEO Enthusiasts Join us for an exciting event where innovative tool founders will showcase their latest solutions for SEO,…

Studies & Data

Data Suggests Google Indexing Rates Are Improving

New research of over 16 million webpages shows that Google indexing rates have improved but that many pages in the dataset were not indexed and over 20% of the pages were eventually deindexed. The findings may be representative of trends and challenges that are specific to sites that are concerned about SEO and indexing.

Guides & Recommendations

A guide to web crawlers: What you need to know

Understanding the difference between search bots and scrapers is crucial for SEO.

Website crawlers fall into two categories: - First-party bots, which you use to audit and optimize your own site. - Third-party bots, which crawl your site externally – sometimes to index your content (like Googlebot) and other times to extract data (like competitor scrapers).

This guide breaks down first-party crawlers that can improve your site’s technical SEO and third-party bots, exploring their impact and how to manage them effectively.

How to use robots.txt, meta robots, and canonical tags correctly

Do you know how and when to correctly use robots.txt, meta robots, and canonical tags? Discover a structured approach to implementing them.

Guide to Canonical Tags & How to Audit Them | Sitebulb

Have you ever searched in Google Search Console and noticed issues like “alternate page with proper canonical tag”, or “ Duplicate, Google chose different canonical than user”? Or have you ever worked on an e-commerce website that sells products with multiple variants (dimensions, colours e.g.) and noticed duplication issues?

The magical solution to solving duplication, which Google Search Console is talking about, is a technical SEO factor called the ‘canonical’ tag.

How To Audit Crawl Depth & Improve Crawl Efficiency | Sitebulb

In this article, we’ll explore various strategies for auditing your website's crawl depth and discuss different tactics to enhance crawl efficiency. I have focused on uncommon information in this article, since you probably already know things like giving importance to internal link structure, speeding up the website, fixing broken links, etc.

How To Use XML Sitemaps To Boost SEO

While most of us recognize the importance of submitting sitemaps to Google Search Console and Bing Webmaster Tools, as well as in the robots.txt file – for faster content discovery and refresh, more efficient crawling of SEO-relevant pages, and valuable indexing reporting to identify SEO issues – the finer details of implementing sitemaps to improve SEO performance may be missed.

How to fix the 'Server error (5xx)' error in Google Search Console

Have you been hit by a 5xx server error in Google Search Console?

500 errors are an HTTP status code that indicates you messed up something and need to start a late-night debugging session.

500 errors are offensive. I can only compare it to eating fermented shark in Iceland – something you’ll want to spit out almost immediately.

500 server errors create a poor user experience and can reduce your crawl budget. If they persist, Google may start ignoring your site altogether. Your website should be commitment-worthy.

If you’re an SEO professional, you’ve likely stayed up until 3 a.m. with coffee and your DevOps team trying to fix a 500 error. You’ll want to keep reading.

How to fix the 'Page with redirect' error in Google Search Console

The “Page with redirect” error in Google Search Console shows a page on your website is redirected to a different URL when the user or Googlebot attempts to access the URL.

This means all the pages listed in the report are not showing in search results.

Nothing new, right?

At first glance, this error in Google Search Console may not seem like the “nectar of the gods.”

But after reading through 138 questions in the Search Console Help community and seeing that Stack Exchange saw 278 views on a similar question, I realize there are probably many SEO professionals who would leave their entire life fortunes to solve this issue if they could.

In the name of very serious SEO needs, I had to investigate.

How to fix ‘Blocked by robots.txt’ and ‘Indexed, though blocked by robots.txt’ errors in GSC

Confused by these Google Search Console errors? Learn what they mean, why they happen, and how to fix them effectively.

SEO News

Google Explains How CDNs Impact Crawling & SEO

Google published an explainer that discusses how Content Delivery Networks (CDNs) influence search crawling and improve SEO but also how they can sometimes cause problems.

Google Publishes New Robots.txt Explainer

Google published a new Robots.txt refresher explaining how Robots.txt enables publishers and SEOs to control search engine crawlers and other bots (that obey Robots.txt). The documentation includes examples of blocking specific pages (like shopping carts), restricting certain bots, and managing crawling behavior with simple rules.

Google Updates Robots Meta Tag Document To Include AI Mode

Google updates help document with details on managing how content appears in AI Overviews and the new AI Mode.

Google updated its robots meta tag help document. Nosnippet and max-snippet directives now apply to AI-powered search features. Publishers can use these tag to limit how much content appears in AI responses.