Skip to content

Cyber Sale: Save big on Moz Pro! Sign up by Dec. 6, 2024

Search engines 5511dd3

Google's Faster Crawl Rate: No Thanks

Casey Henry

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Table of Contents

Casey Henry

Google's Faster Crawl Rate: No Thanks

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Recently I was working with a client and I decided to do a little experiment, because sometimes I have nothing better to do.  I wanted to know whether changing the crawl rate in Google’s Webmaster Tools really made a difference.  Part of me felt Google just put it there to make people feel that Google will come to their site more often, but part of me wanted to trust Google.

I quickly wrote a script that would stored all the user agent info, page visited, and date visited in a database.  I let it run for 200 days, 90 of which were set on the “Faster” crawl speed (the rest were on “Normal”).  I also tracked the time from which a new blog was posted and when Google first visited that post.  I tracked the site for 55 days on “Normal,” 90 days on “Faster,” and then the last 55 on “Normal” again.

Here is some background information before we get into the data I collected.  The website had 45 pages cached in Google and was over a year old when I began this experiment.  New blog posts were added between 2 and 3 times a week, adding fresh new content to the site.  The site ranks well for its brand and a few other keywords related to its field.  

During the course of the experiment over 80 new blog posts were added to the site.  While the crawl rate was set at “Normal” it took Google an average of 3.4 days to first visit the page.  When the crawl rate was set at “Faster” it took Google an average of 2.9 days to first visit the page.  That was not much of an improvement in my book, but interesting nevertheless.  Note: This is not the time when the page first appeared cached in the index, but when the bot first visited the page itself.

 Faster Crawl Rate

Normal Crawl Rate

While the crawl rate was set to “Normal,” Google visited the site an average of 6 times a day and favored visiting the site on Tuesdays, Fridays, and Sundays.  When the crawl rate was set to “Faster,” Google visited the site an average of 7.5 times a day and favored visiting the site on Wednesdays, Thursdays, and Saturdays.  Over 90 days it visited 135 pages more than if I would have left it set to “Normal.”

 Crawl Rate

If you look at the graph above you will see there is slight improvement while the crawl rate is set to “Faster,” but only for a portion of the 90 days.  After seeing the results I have come to the conclusion that changing the crawl rate in Google’s Webmaster Tools makes no substantial difference to this site.

Back to Top
Casey Henry

I have a passion for both learning and helping people. I'm often found up late at night coding or designing something for my clients or one of my many secret projects. I'm a huge fan of tracking and trying to figure out what Google is up too! Feel free to follow me on Twitter: @caseyhen

With Moz Pro, you have the tools you need to get SEO right — all in one place.

Read Next

Google HCU: What Can You Do? — Whiteboard Friday

Google HCU: What Can You Do? — Whiteboard Friday

Oct 18, 2024
The Helpful Content Update Was Not What You Think

The Helpful Content Update Was Not What You Think

Sep 05, 2024
How to Optimize for Google's Featured Snippets [Updated for 2024]

How to Optimize for Google's Featured Snippets [Updated for 2024]

Aug 20, 2024

Comments

Please keep your comments TAGFEE by following the community etiquette

Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.