Many of our keen members observed that late last week, Linkscape's index updated (this is actually our 27th index update since starting the project in 2008). This means new link data in Open Site Explorer and Linkscape Classic, as well as new metric data via the mozbar and in our API.

Index 27 Statistics

For those who are interested, you can follow the Linkscape index update calendar on our API Wiki (as you can see, this update was about a week early).

Although we've now crawled many hundreds of billions of pages since launch, we only serve our uber-freshest index. Historical data is something we want to do soon - more on that later. This latest index's stats feature:

• Pages - 40,152,060,523
• Subdomains - 284,336,725
• Root Domains - 91,539,345
• Links - 420,049,105,986
• % of Nofollowed Links - 2.02%
• % of Nofollows on Internal Links - 58.7%
• % of Nofollows on External Links - 41.3%
• % of Pages w/ Rel Canonical - 4.3%

These numbers continue the trend we've been seeing for some time where internal nofollow usage is declining slightly while rel canonical is down a bit in this index but up substantially over the start of the year (this likely has more to do with our crawl selection than with sites actually removing canonical URL tags.

Comparing Metrics from Index to Index

One of the biggest requests we get is the ability to track historical information about your metrics from Linkscape. We know this is really important to everyone and we want to make this happen soon, but have some technical and practical challenges to overcome. The biggest of which is that what we crawl changes substantively with each index, both due to our improvements in what to crawl (and what to ignore) and with the web's massive changes each month (60%+ of pages we fetched 6 months ago are no longer in existence!).

For now, the best advice I can give is to measure yourself against competitors and colleagues rather than against your metrics last month or last year. If you're improving against the competition, chances are good that your overall footprint is increasing at a higher rate than theirs. You might even "lose" links in a raw count from the index, but actually have improved simply because a few hundred spam/scraper websites weren't crawled this time around, or we've done better canonicalization with URLs than last round or your link rotated out of the top of a popular RSS feed many sites were reproducing.

Measuring against other sites in your niche is a great way to compare from index to index

If you've got more questions about comparisons and index modifications over time, feel free to ask in the comments and we'll try to dive in. For those who are interested, our current thinking around providing historical tracking is to give multiple number sets like - # of links from mR 3+ pages, # of links from mR 1-3 pages, etc. to help show how many "important" links you're gaining/losing - these fluctuate much less from index to index and may be better benchmarking tools.

Integration with Conductor's Searchlight Software

SEOmoz is proud to be powering Conductor's new Searchlight software. I got to take a demo of their toolset 2 weeks ago (anyone can request one here) and was very impressed. See for yourself with a few exclusive screenshots I've wrangled up:

And at the bottom of the series is Seth Besmertnik, Conductor's CEO, during the launch event (note the unbuttoned top button of his shirt with the tie; this indicates Seth is a professional, but he's still a startup guy at heart). Searchlight already has some impressive customers including Monster.com, Care.com, Siemens, Travelocity, Progressive and more. I think many in the SEO field will agree that moving further into software is a smart move for the Conductor team, and the toolset certainly looks promising.

Conductor's also releasing some cool free research data on seasonality (request form here). Couldn't resist sharing a screenshot below of the sample Excel workbook they developed:

mmm... prepopulated

SEOmoz's Linkscape index currently powers the link data section of Searchlight via our API and we're looking forward to helping many other providers of search software in the future. We're also integrated with Hubspot's Grader.com and EightFoldLogic's (formerly Enquisite) Linker, so if you're seeking to build an app and need link data, you can sign up for free API access and get in touch if/when you need more data.

The Link Juice App for iPhone

We're also very excited about the popular and growing iPhone app - LinkJuice. They've just recently updated the software with a few recommendations straight from Danny Dover and me!

The LinkJuice folks have promised an Android version is on its way soon, and since that's my phone of choice, I can't wait!

If you've got an app, software piece or website that's powered by Linkscape, please do drop us a line so we can include it. I've been excited to see folks using it for research - like Sean's recent YOUmoz post on PageRank correlations - as well as in many less public research works.

Oh, and if you somehow missed the announcement, go check out the new Beginner's Guide to SEO! It's totally free and Danny's done a great job with it.