How We Index The Web
Frequently Asked Questions
What's Covered?
In this guide you’ll learn more about how we crawl and index the web for our links index which powers Link Explorer and the Links section of Moz Pro Campaigns.
Quick Links
Moz's Link Index Crawler
Our Link index data is gathered by crawling and indexing links, just like Googlebot does to populate Google’s search results. This data allows us to understand how Google rankings work and calculate metrics like Page Authority and Domain Authority.
Our web crawler, Dotbot, is built on a machine learning-based model that is optimized to select pages like those that appear in our collection of Google SERPs. We feed the machine learning model with features of the URL, like the backlink counts for the URL and the PLD (pay-level domains), features about the URL, like its length and how many subdirectories it has, and features on the quality of the domains linking to the URL and PLD. So the results are not based on any one particular metric, but we're training the crawler to start with high-value links.
How Often Does the Moz Link Index Update?
The index that powers Link Explorer is constantly updating to provide fresh link data. This includes updating the data which powers each section of Link Explorer, including Linking Domains, Discovered and Lost, and Inbound Links. When discovered or lost links are found, we'll update our database to reflect those changes in your scores and link counts. We prioritize the links we crawl based on a machine learning algorithm to mimic Google's index. This does not mean that DA and PA will change with every data update; it will only change if we find new links to a respective site.
How Old is Moz Link Index Data?
Links which are newly discovered by our crawlers should be populated in Link Explorer and the Links section of your Campaign within about 3 days of being found. In addition, links will not drop out of the index due to age, although they will show as Lost if we are no longer able to access or find that link when we go to recrawl the page on which it was originally found. We currently try to re-crawl all high-quality pages at least every 90 days. Most links in the index come from pages that were crawled less than 6 months ago.
Why Am I Not Seeing Expected Links to My Site?
There are a few reasons why links to your pages might not be showing up in Link Explorer or the Links section of your Moz Pro Campaign. You can read about some troubleshooting tips in our guide here.
If you’d like to keep an eye on links that point to your site which haven’t been indexed yet, add them to a Link Tracking Lists.
Not Discovered
We may not have discovered those links yet. Our index is large but it doesn’t cover the whole web. Pages with high-value links pointing to them will be crawled and indexed faster and more often.
Newly discovered links have the ability to be populated into our index in about 3 days. It may take longer for us to discover backlinks to your site based on factors like crawlability of the referring pages, as well as the quality of the links and the referring pages.
Not Indexed or Crawled the Referring Page
If you know that there is a page on the web with an inbound link pointing to your site that isn’t being indexed you can check to see if we've indexed the referring page on which that link is found.
A quick way to check if a page that you know is linking to you has been indexed is to enter the URL into Link Explorer. You’ll be able to see if we've got data for that page. If this page is not yet indexed, then we won't yet have crawled and indexed the link to your site.
Non-Canonical Content
The behavior of our crawlers is to ignore non-canonical content. Pages missing canonicalization or pages that are not canonicalized to themselves are indexed, but not crawled for content.
Robots Exclusion & Meta Noindex
If crawlers encounter a noindex meta tag or has a robots.txt file that has a Dotbot exclusion, we won’t display this data in our index. Noindex tags mean that most search engines will not show that page in their results. Noindex tags can be page or site specific and can specify all bots or just specific bots. You can read more about the impact of robots.txt exclusions on our blog.
How Do I Get Backlinks Indexed By Moz?
If you’d like to keep an eye on links that point to your site which haven’t been indexed yet, add them to a Link Tracking Lists to help monitor the discovery of your links.
- Read more on why Moz hasn't indexed your backlinks.
How to Use Domain Authority & Page Authority
We recommend comparing your site's DA with competitors’. By measuring the distance between these scores, and tracking the change over time, you'll be able to identify if you're improving relative to the sites that really matter — your direct SEO competitors. We also recommend keeping track of how many linking root domains you have from index update to index update, as this will be a quick way to confirm possible reasons for an increase or decrease in the score.
Related Articles
Was this article helpful?
Yes! Amazing!
Yes! It was what I needed.
Meh. It wasn’t really what I was looking for.
No, it wasn’t helpful at all.
Thanks for the feedback.