Skip to content

Cyber Sale: Save big on Moz Pro! Sign up by Dec. 6, 2024

Search engines 5511dd3

An Exemplary Experiment in Spider Behavior

Rand Fishkin

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Table of Contents

Rand Fishkin

An Exemplary Experiment in Spider Behavior

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Drunk Men has put together the results of a one-year long experiment to monitor spider activity from the major three search engines. If you're watching crawling performance on big content sites, this is must read. Note their graph (in log-based scale) of relative levels of performance over time:

pageviews in time

It may come as a surprise to many in SEO that Yahoo! is the fastest and deepest crawling bot for this content. I'd argue that in the last 2-3 months, I've been seeing MSN overtake Yahoo!'s crawler in both depth and breadth, and I've also seen Google, once induced by strong external links, become the most rampant and hungry (if not speediest) bot of all.

My favorite part of their results is actually this graphical representation of how Yahoo! crawled:

Yahoo! binary tree

Like a beautiful, black hat fractal...

Back to Top

With Moz Pro, you have the tools you need to get SEO right — all in one place.

Read Next

How to Optimize E-commerce Sitemaps with 1M+ Pages — Whiteboard Friday

How to Optimize E-commerce Sitemaps with 1M+ Pages — Whiteboard Friday

May 17, 2024
7 Ways SEO and Product Teams Can Collaborate to Ensure Success

7 Ways SEO and Product Teams Can Collaborate to Ensure Success

Apr 24, 2024
6 Things SEOs Should Advocate for When Building a Headless Website — Whiteboard Friday

6 Things SEOs Should Advocate for When Building a Headless Website — Whiteboard Friday

Apr 19, 2024

Comments

Please keep your comments TAGFEE by following the community etiquette

Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.