Skip to content

Cyber Sale: Save big on Moz Pro! Sign up by Dec. 6, 2024

Search engines 5511dd3

Diagrams for Solving Crawl Priority & Indexation Issues

Rand Fishkin

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Table of Contents

Rand Fishkin

Diagrams for Solving Crawl Priority & Indexation Issues

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Yesterday night I stayed up way too late authoring a post on Google's Indexation Cap. Today, despite getting up way too early, I wanted to follow up and answer some of the questions from the comments, Twitter and my email. I think SEOs who read the post rightly asked for more direction in solving this problem - a fair request. Below, I've done my best to tackle these problems visually, as I believe we all think about site architecture and crawling issues in a visual structure.

First off, here's a sample site hierarchy to set down the concept and give the colors I'm using in the following diagrams more context:

A Sample Site Architecture

Next, I've illustrated in a more representative fashion, how those hierarchies might look on a website, and noted the external link potential of each:

Typical Site's Link Earning Potential by Content Section

In this next piece, I'm trying to explain a very important concept and something that's frequently misunderstood by SEOs. Once upon a time, search spiders would crawl the web largely recursively - hitting a homepage that had been submitted to its index (remember way back when search engines had submission?!), then crawling in an outward fashion based on the links they discovered there. That hasn't been the case for a long time, and as we all see with crawl paths (if you're looking at the requests Google/Yahoo!/Bing make to your domain), multiple entry points are nearly universal and crawling pushes "outward" from those priority URLs. It looks a bit like Minesweeper, right? :-)

Spider Crawl Priority Paths Graphic

Finally, I've got a graphic to help understand how to positively approach these problems and solve them.

Methods to Improve Crawling, Indexing & Ranking

There are certainly more recommendations that can be provided around these issues, and I look forward to a discussion of them in the comments.

p.s. I covered site architecture and navigation in a good bit of detail at the PRO Training this summer, but I like this image format so much, I think I might re-craft something new for next year. It feels like structuring sites properly is still a big pain point for SEOs (but possibly that's less to do with lack of knowledge and more to do with lack of influence during the design phase?)

Back to Top

With Moz Pro, you have the tools you need to get SEO right — all in one place.

Read Next

How to Optimize E-commerce Sitemaps with 1M+ Pages — Whiteboard Friday

How to Optimize E-commerce Sitemaps with 1M+ Pages — Whiteboard Friday

May 17, 2024
7 Ways SEO and Product Teams Can Collaborate to Ensure Success

7 Ways SEO and Product Teams Can Collaborate to Ensure Success

Apr 24, 2024
6 Things SEOs Should Advocate for When Building a Headless Website — Whiteboard Friday

6 Things SEOs Should Advocate for When Building a Headless Website — Whiteboard Friday

Apr 19, 2024

Comments

Please keep your comments TAGFEE by following the community etiquette

Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.