On-Demand Crawl Overview

In this guide you’ll learn more about the On-Demand Crawl tool. If your crawl has failed or if you're having trouble with your crawl completing, please see our troubleshooting guide.

Quick Links

• Run a One-Off Site Crawl
• How Does On-Demand Crawl Work?
• Track in a Campaign
• Crawl Report
• Issues Found
• Pages Crawled
• Recommended Fixes
• All Issues CSV Data
• Pages Crawled CSV Data

Are you scoping out a potential client and want a quick SEO audit of their site? We've got a tool for that! On-Demand Crawl is available with your Moz Pro subscription and you can find it under the Moz Pro dropdown menu.

Please note: if you're already tracking a site in a Moz Pro campaign you can trigger a recrawl with a Medium Moz Pro plan or above.

To start a crawl, enter the URL and click Analyze. You have the option to specify if you’d like us to crawl an entire root domain or just the subdomain you have entered.

As soon as you click Analyze, we’ll send rogerbot over to crawl up to 3,000 pages, starting with the URL entered. Following links from page to page, our bot crawls the page code to report on potential technical SEO issues like missing Title tags or slow load times, and locate critical issues like 4xx and 5xx errors and much, much more.

When your crawl is complete we’ll indicate the status in the interface and you’ll receive an email notification. You’ll also see how many issues we found and the date that you requested the crawl.

Want to remove a crawl? Mark the crawl using the checkbox on the left and select the Delete option on the right.

If you want to crawl more than 3,000 pages, and get weekly updates, click Track in a Campaign from the On-Demand Crawl overview page or from within the crawl report itself.

Campaigns crawl your site weekly, functioning more like a health monitor vs. a one-off health check. With Campaign Site Crawl you'll find out when new issues appear and have the option to ignore certain issue types, or mark issues as fixed as you make updates to your site.

The ability to start a Campaign from the On-Demand Crawl tool is dependent upon space within your subscription limits. If you are already at your maximum number of Campaigns, you won’t be able to create a Campaign from this tool. You can view and manage your campaigns here.

• Learn more about creating a new Campaign

Open your Crawl Report to see an overview of the crawl, including the number of pages we crawled and the number of issues we found.

Below the On-Demand Crawl overview, you’ll see a list of all the issues types we found and how many pages are associated with those issues. Here, you’ll also find the option to export a list of all the issues found to a CSV.

We’ll then show you a list of all the pages we crawled. You can sort by URL or partial URL match and you can also filter by status code and/or issue type. This information is also able to be exported as a CSV.

By clicking the arrow in the “View Issues” column, you’ll be able to see the issues we identified for that specific page in the crawl.

Below the outline of the pages we crawled, you’ll see a list of all the things Moz recommends fixing. We’ll show you how many pages have been identified as having those issues, and if you click the arrow to the right, we’ll tell you more about the issue, why it’s important, and how to fix it!

We’ll identify the importance of recommended fixes using the symbols to the left.

Red exclamation point inside a box - Indicates this is something super important that we think you should look into this fixing right away.

Yellow exclamation point inside a box - Indicates we think you should definitely pay attention to these things and should consider fixing them next.

Blue “i” inside a circle - Indicates these are important things that may be affecting your overall SEO efforts, but they aren’t as critical as the exclamation points.

You can export All Issues found in your On-Demand Crawl to CSV using the button on the top right, just below the On-Demand Crawl summary.

The following data will be included in the All Issues CSV export:

A. Issue Type - The type of issue associated with this URL, i.e. Duplicate Content, is 4xx, or Missing Title.

B. Status - Identifies whether this issue is new or existing based on the last crawl’s data.

C. Affected Pages - How many pages are affected by this issue group. Issues which have more than one affected page include Duplicate Content, Duplicate Title Tags, Redirect to 4xx, and Redirect Chains.

D. Issue Group Identifier - The unique identifier which designates the pages associated with an issue group. All pages with the same numerical value for Issue Group Identifier are considered part of the same issue group and are flagged as part of the identified issue.

E. URL - The URL crawled.

F. Referring URL - Where the crawler found the crawled URL. This may be another redirected URL or a page on your site.

G. Redirect Location - If the URL crawled redirects to another page or URL, the redirect location is where that redirect points. For example, if the URL http://mysite.com redirects to http://mysite.com/home, the redirect location would be noted as http://mysite.com/home.

H. Status Code - The status code returned to our crawler when it attempted to access the URL.

I. Page Speed - The loading time for this page in seconds.

J. Title - The title tag found in the source code of the page. A blank cell indicates no title tag was found.

K. Meta Description - The meta description found in the source code of the page. A blank cell indicates no meta description was found.

L. Page Authority - The Page Authority for this page. Page Authority is a Moz-proprietary metric between 1 and 100 which predicts how well a page will rank in Google based on a machine-learning algorithm of link metrics.

M. URL Length - The character count of the URL.

N. Title Pixel Length - The length of the page title in pixels.

O. Title Character Count - The length of the page title in characters.

You can export the Pages Crawled for your On-Demand Crawl using the button to the right, just above the filter options. If you have filtered your Pages Crawled, those filters will be reflected in your CSV export.

The following data will be included in the Pages Crawled CSV export:

A. URL - The URL crawled.

B. Referring URL - Where the crawler found the crawled URL. This may be another redirected URL or a page on your site.

C. Title - The title tag found in the source code of the page. A blank cell indicates no title tag was found.

D. Meta Description - The meta description found in the source code of the page. A blank cell indicates no meta description was found.

E. Status Code - The status code returned to our crawler when it attempted to access the URL.

F. Page Speed - The loading time for this page in seconds.

G. Canonical URL - The canonical URL found in the source code of the page. A blank cell indicates no canonical tag was found.

H. Redirect Location - If the URL crawled redirects to another page or URL, the redirect location is where that redirect points. For example, if the URL http://mysite.com redirects to http://mysite.com/home, the redirect location would be noted as http://mysite.com/home.

I. Link Count - The number of internal links we crawled on this page.

J. Page Authority - The Page Authority for this page. Page Authority is a Moz-proprietary metric between 1 and 100 which predicts how well a page will rank in Google based on a machine-learning algorithm of link metrics.

K. Issue Count - The number of Site Crawl issues identified for this page/URL.

L. Word Count - The number of words counted in the body of the page.

M. Crawl Depth - How many links/pages the crawler followed from the home page to get to this URL. The home page or seed URL of your Campaign will have a crawl depth of 0.

N. No Follow - Notes whether the page is marked as nofollow using a TRUE or FALSE value, where TRUE indicates the page is flagged as nofollow.

O. No Index - Notes whether the page is marked as noindex using a TRUE or FALSE value, where TRUE indicates the page is flagged as noindex.

P. Robots - If found, the contents of the meta robots tag will be noted here.

Q. X-robots - If found, the contents of the X-robots-tag will be noted here.

R. URL Length - The character count of the URL.

S. Duplicate Content Group - If the page is flagged as duplicate content, the Duplicate Content Group will be noted in this column by a numerical value. All pages with the same numerical value for Duplicate Content Group are considered part of the same issue group and are flagged as duplicates of one another.

T. Duplicate Title Tag Group - If the page is flagged as having a duplicate title tag, the Duplicate Title Tag Group will be noted in this column by a numerical value. All pages with the same numerical value for Duplicate Title Tag Group are considered part of the same issue group and are flagged as having duplicate title tags.

U. Redirect 4xx Group - If the URL redirects to a 4xx page, the Redirect 4xx Group will be noted in this column by a numerical value. All pages with the same numerical value for Redirect 4xx Group are considered part of the same issue group and are flagged as redirecting to 4xx.

V. Redirect Chain Group - If the URL is part of an identified redirect chain, the Redirect Chain Group will be noted in this column by a numerical value. All pages with the same numerical value for Redirect Chain Group are considered part of the same issue group and are flagged as part of the identified redirect chain.