A common question about big websites is “what do I index?” This question is further complicated with eCommerce websites that have a larger demand for better categorization and user features to help consumers find what they are looking to buy.

The technical architecture of a website is the foundation for a good Internet marketing campaign. Determining what pages are crawled by Google to avoid having duplicate content and to keep Google bots happy are paramount concerns. Obviously, you want as many relevant pages indexed on your website as you can, but you also want to keep all the pages that aren’t adding any value uncrawled, and out of Google's index.

In an effort to help ease the headache that I have found myself suffering through many times when working on large eCommerce websites, this is a guide to faceted navigation, filters, categories, and what not to index.

eCommerce Overview

For those of you who may not have dealt with an eCommerce website, or are newly experiencing working on one, here are some of the attention points. An eCommerce website is unlike a standard website in that there are multiple versions of each page of content which can add hundreds, thousands, and even sometimes hundreds of thousands of pages from a single product or group of products. With products coming in all sorts of colors, sizes, brands and prices you need to know when to draw the line as far as the types of pages that Google should be adding to their index.

This line needs to be drawn so that users are getting all of the information they need, and the search engines are not being over-whelmed or confused by pages being indexed. To bridge the need for giving the consumer the best experience while they find their desired products, and keeping the Google bots happy, we need to make the best use of facets, filters and categories. Building on what Rand Fishkin talked about over 3 years ago in a Faceted Navigation White Board Friday.

How to Easily Distinguish eCommerce Categories

Categorization cannot be determined by a cookie cutter model. Every website, and vertical, is a bit different when it comes to their needs for categorization. When determining categorization, there are two major things to keep in mind; the first is to keep it simple enough for the user to easily navigate, and the second is to make sure you have an intuitive structure that adds value (and the least amount of work).

Simple Categories - Rule of three

Keeping categories simple for users will be very important, but often on eCommerce websites you will also have the URL reflect the categories you use. From a technical SEO standpoint, URLs that are too long are hard for Google to read and are bad for your website's SEO. Any options you might have to keep a concise URL, and still have good categorization, will be important to take advantage of. As a general rule, I try to keep all of my products within three categories or less. Example: I have www.my-example.com and the product I am categorizing is a Nike running shoe. I might categorize this shoe like so:

• www.my-example.com>Running shoes(general distinguishing category)>Men's(sub category)>Nike(specifically distinguished)

My specific product url would look something like:

• www.my-example.com/running-shoes/mens/nike/nike-air-max-mens-running-shoe

This would be my static product URL. Any URL that is 115 characters or longer is considered a no-no, so keep it simple. If you want to check, I use the Meta Length tool. I know it isn't intended for counting URL characters, but it does work and it's free. If you have the option in your website to shorten the URL, I highly suggest using it. For example:

• www.my-example.com/running-shoes/mens/nike/nike-air-max-mens-running-shoe

Could easily be shortened to:

• www.my-example.com/running-shoes /nike-air-max-mens-running-shoe

It honestly just depends on what options you have to manipulate the URL and the categories, but still remember the rule of three. Too much will confuse the user, and too little will leave them without a roadmap to find the products they need.

Building Structured Categories

When building or rebuilding your categories you need to think in a hierarchical way as if you are building a pyramid; starting from your homepage, and mapping out to your various products. Often, I use breadcrumb navigation links to see how my structuring will be represented (a large white board is also a useful tool). This allows me to ensure that the structure makes sense going from a broad or main category to more specific ones. Narrowing down while adding value can be hard, but doing so will provide a better user browsing experience. If you're seeing a lot of duplicate categories you might not need as many total categories. Below is an example. As you can imagine the hierarchy can get quite complex and can get more specific using a filter to help the consumer find the right products.

When a Category is Too Specific

There is certainly a gray area as to when you have categories that are too specific, and thus actually infringe on your products themselves. A category should lead to a broader product list, and not just to a single product. Consider this when building the categories: less will be more concerning category structure, and always create the category to help your consumer find that product on your website. If the categories are too specific, this is where facet navigation can greatly help.

Category Checklist

• Are my categories simple enough to understand?
• Do they have good structure? (Main Category, Sub Category, Specific Category)
• Does each category add value for the user, not just rearrange products?
• Are my URLs still short enough and pretty enough to be “SEO-friendly”?

Faceted Navigation

For eCommerce websites, facets provide a way to help the user find products more easily. Faceted navigation is normally found in the sidebar of a website and contains facets (or indexed categories) and filters (which are not indexed-more on that later).

These are sidebar links, but as Google may encourage this type of navigation for the convenience of the user, they have a very specific idea of just how they should be indexed, or not indexed.

Faceted navigation is often not very SEO friendly if it isn’t done properly. The saying “write for humans, optimize for robots” applies to this topic. The most important thing is to have the navigation there to help the consumer narrow, sort, and find the products intuitively and without hassle. On the technical side, this can create duplicate content, and make a headache for a web developer. You might be again asking, "what do we make crawlable and where do I draw the line?" Read on...

Using Facets

Facets are sidebar links which allow consumers to simplify product results, without the need for ugly siloed categorization. A facet category should be considered an extension of your categories and ARE indexed by Google. If these options are indexed, they should add a unique value for every selection. Every facet on a website should change the on-page content, thereby making the consumer happy by delivering better results, and making Google bots less confused. Determining your facets can be difficult at first, but these will likely reflect the specific categories you already have. Facet categories are indexed because they help to specify product listings. This is a fine line to walk when considering that filters are not to be indexed by search engines.

Qualifying as a Facet Category

• Is your facet category a usable extension of your already existing categories?
• Does it help to specify a product result?

If you answered yes to both of these questions, you have a facet category, and the links and content should be indexed.

No Indexed Filters

Filters can create some serious on-page issues for technical SEO, but are very necessary for the user. A filter is used to sort or narrow the content of a page, which often creates either duplicate content, or additional URLs that add no value. But you still need to know what on your site IS a filter. If you happen to be sorting your content differently, by price or specific feature (ex. color), and these options aren't fundamentally changing the content but just narrowing it in some way, this is a filter. Specific product features might just narrow the product pool.

Other examples of content that shouldn’t be indexed are different options that might sort the content differently just as a consumer’s browsing preference. This could be a feature as simple as list and grid view modifiers. As a disclaimer, you will need one type of browsing preference crawled for Google to index the content. However, you will want to either disallow, or set the other versions of browsing as a canonical to the preferred indexed copy.

For example:

www.my-example.com/running-shoes/mens-shoes would display all men's running shoes, but maybe I want a specific size of 12. I select the filter and get my results. Since there's a chance that all of those shoes have a size 12 this could appear as duplicate content for a robot, but for a user it might narrow the selection enough to help them find the product they want.

Filter Qualifying Checklist

• Does this option sort my content in a different manner (ex. list/grid or by price)?
• Does this option narrow the results specifically (to a set of products) for a user?

Solutions for No Indexing Facet Navigation Filters

There are a number of solutions to make sure your filters aren't being indexed. Below are 4 solutions to help you no-index filters using development, SEO techniques, and Webmaster Tools. Again, every web platform and website is a little different. So, there is no solution that will work for every instance concerned.

AJAX/Scripts – Preferred

This is the most ideal solution for faceted navigation as you can easily provide the usability for the user while satisfying Google bots as they will only index the content that matters. This can often be a more expensive solution to faceted navigation and you would still want to apply the identifiers of categories and filters to the navigation. For More information on AJAX Faceted Navigation see this post from Mike Pantoliano about Building Faceted Navigation That Doesn't Suck.

Rel=canonical Directing your Indexation

Many of you may be familiar with canonical tags and how to use them. In this case, they are a little less straight forward than usual. Typically, a rel="canonical" tag would be used to refer a similar or duplicate page of content toward the content you want to or prefer to be indexed. We can easily apply this to some websites and filters as we can set the canonical to the previous category before the filter is applied. In addition, it can be used for any sort options, like grid and list views for example. Just remember to prefer the highest converting version of the page. This is a simple solution to help direct the robots toward the content you really want crawled while keeping the content that is helping users find products. Do keep in mind that canonicalization is simply a suggestion to the search engines of how to index content, as opposed to disallowing indexation of a page via the robots meta file. For more information on implementing rel="canonical" tags see SEOmoz's Canonicalization page

Robots.txt - Disallowing Crawling

The infamous robots.txt file can help you remedy a situation, but only if your filters have a defined path. For some eCommerce platforms, this won’t work if the filters are implemented dynamically. A robots file can disallow or exclude pages from being indexed by the crawlers. This will give a more efficient crawl for the robots. Just make sure everything you’re disallowing is in fact something you don’t ever want indexed. Many times, this will work for sites. If you are using the robots file too much, you might want to look for a better long term solution for your faceted navigation. For more information on a Robots.txt file or uses look at SEOmoz's Robots.txt and Meta Robots.

Google Webmasters is Good for Dynamic Websites

Using Google Webmaster Tools is a choice that is really seldom mentioned in regards to faceted navigation, but it can actually help to create temporary fixes while building a better faceted navigation. This method should be thought of as a last resort in most cases, because you aren’t physically fixing the site for better crawling. In Webmaster Tools, there is a selection under configuration called URL parameters. This tab will allow you to mark up certain dynamic modifiers.

For example:

?color=blue or ?size=large

These are filters that can easily be marked up in URL configuration to not be crawled. I would encourage anyone who is going to use this method to watch the video on configuring URL parameters from Google. It's quite helpful and can help to avoid making scary mistakes.

Healthy Sites Have Happy Robots and Consumers

Taking into consideration the categories and how they are utilized on a website can greatly affect your site's health and usability. Using faceted navigation properly with filters can provide a great benefit to any consumer looking for specific products within your website, and keep the Google bots happy. Every platform for eCommerce is a little different, and should be individually diagnosed when determining what needs to be improved for the user and for the search engines. Taking time to define what helps the site get found, and what is just helping individual consumers is important. Happy consumers AND happy Google bots are the recipe for success in eCommerce navigation.

As they say "necessity is the mother of invention". This post was entirely inspired by blindsaver.com an extensive and ongoing eCommerce project dealing with categories, facets and filters.