An Exemplary Experiment in Spider Behavior
The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.
Drunk Men has put together the results of a one-year long experiment to monitor spider activity from the major three search engines. If you're watching crawling performance on big content sites, this is must read. Note their graph (in log-based scale) of relative levels of performance over time:
It may come as a surprise to many in SEO that Yahoo! is the fastest and deepest crawling bot for this content. I'd argue that in the last 2-3 months, I've been seeing MSN overtake Yahoo!'s crawler in both depth and breadth, and I've also seen Google, once induced by strong external links, become the most rampant and hungry (if not speediest) bot of all.
My favorite part of their results is actually this graphical representation of how Yahoo! crawled:
Like a beautiful, black hat fractal...
Comments
Please keep your comments TAGFEE by following the community etiquette
Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.