Over the past couple of years, we've seen a number of new web-based SEO crawlers (including our own Pro App here on SEOmoz). They're great tools and can provide a lot of insight, but sometimes you've just got to get down and dirty into a big site, and you need a desktop crawler application. For the past few years, that's pretty much meant Xenu's Link Sleuth. Recently, a new entrant has thrown its hat into the ring – Screaming Frog SEO Spider.

I was amazed, on social media, how many people still hadn't heard of Screaming Frog. Actually, a few people still don't know about Xenu. So, I thought it would be a good time to put both programs through their paces.

Xenu's Link Sleuth

In this corner (that doesn't really work when you can't see me pointing, does it?), we have Xenu, a PC-based desktop app that bills itself as a tool for checking broken links. Over the years, enterprising SEOs have found it's capable of a lot more. For such a popular tool, it's funny how little we know about the creator, a German [engineer?] named Tilman Hausherr. All I can tell you is that he wrote Xenu because he was bored, he doesn't want your money, and he really dislikes Scientology.

Pro: Xenu is Free

Let's start with a few pros and cons. You really can't beat that Xenu is completely free. We all owe Tilman a hearty "thank you" for putting it out there over a decade ago.

Pro: Xenu is Fast

While a large-scale site can take a while to fully crawl, Xenu is a machine, and it's an impressively multi-threaded app, considering when it was built. It can power through 1000s of pages pretty quickly.

 Con: It's Not Intuitive

Xenu really takes some getting used to, and it's clearly built by an old-school software designer. The basics are easy enough, but getting into the real SEO applications of Xenu takes a bit of a learning curve. To be fair, it wasn't built for what most of us try to do with it.

 Con: No Mac Version

This is the biggie for many people – Xenu only runs on Windows PCs. There's no Mac or Linux version, and the author doesn't plan to release one.

What Can Xenu Do?

At its core, Xenu really is just a link checker, churning through your site to test internal and (optionally) external links:

Since Xenu operates like a crawler, though, it can really help test crawl paths and find holes in your internal linking. Xenu also reports useful stats, like the crawl level, outbound links from a page, and inbound links to a page. It also returns TITLEs and META descriptions, which can be handy at-a-glance.

One really useful feature in Xenu is that, once you find a broken link (page or image), you can just right-click on it, select "URL Properties", and you'll get a pop-up like the one below showing you all the pages linking to that broken file:

Xenu is a work-horse, and it's still a great tool for churning through links fast. It's only real fault is that we've tried to push it beyond what it was designed for, and it wasn't really designed for SEO.

Screaming Frog SEO Spider

That's where Screaming Frog comes in, a desktop crawler specifically designed for SEOs by a UK search marketing firm. Because it's built specifically for SEOs, it has a lot of bells and whistles that Xenu doesn't. Let's start with the quick list of pros and cons.

Pro: It's Built for SEOs

We'll dive into this one below, but Screaming frog has been built by SEOs for SEOs, and that fact is obvious as soon as you launch the application.

Pro: It's User-friendly

I found Screaming Frog very easy to dive into. The advanced uses will take some time to discover, I'm sure, but any experienced SEO should be able to get up and running on it quickly.

Pro: It Runs on Your Mac

Screaming Frog supports both PCs and Macs. If you have a Mac shop, that's a big one.

Con: It's Not Free

Ok, here's the down side. It's £99 per year (that's about $162, currently) to run the full version of Screaming Frog. The free version will only crawl 500 URIs. The free version is nearly fully functional, though, so you can get a good idea if it's worth the price for you.

What Can Screaming Frog Do?

On the surface, Screaming Frog doesn't look too much different than Xenu:

Like Xenu, Screaming Frog tracks the crawl level, internal links, outbound links, TITLE tags, and META descriptions. It also adds:

• HTTP status codes
• TITLE tag length
• META description length
• H1, H2 tags and lengths
• Canonical tags

Many of these elements are also broken into their own tabs at the top, so that you can easily focus in on problem areas. You can right-click on any URL for many more options, including checking indexation in Google, Yahoo, or Bing and checking backlinks in Open Site Explorer, Yahoo Site Explorer, or Majestic SEO:

The paid version includes one incredibly useful feature – the ability to custom filter and search HTML source code across a site. Digging into all of the features is beyond the scope of this post, but Branko Rihtman has a nice Screaming Frog review over at SEO Scientist. If you want to dig deeper into Xenu, Tom Critchlow had a good Xenu recap here on SEOmoz last year.

And The Winner Is...

Sorry, I just can't do it. I've loved Xenu (the app, not the galactic overlord) for years, and even if it weren't free, I'd have gotten my value out of it many times over. I'm excited about Screaming Frog and what it can do, though, and I'm especially excited to see new products and innovation finally coming to desktop crawlers. This is a win-win for technical SEOs everywhere, and I'm looking forward to the evolution of both desktop and web-based crawlers.