Measuring the frequency of Googlebot visits on following listing pages

Some time ago we conducted a research concerning indexation of subsequent listing pages. It was proven that the farther from the main category page (first page of the listing), the less chance that the page will be indexed, and that increasing the number of linked pages on the main page improved their indexation (tested on pages: 5th, 10th and 15th).

Nevertheless, in case of frequently updated listings (for example with classifieds) not only the level of their indexation is important but also the frequency of Googlebot’s visits as the more often it comes to the site, the more listed pages (classifieds) it can find and crawl.

To gather more information about the issue, I conducted another test aimed at measuring the frequency of Googlebot visits on various listings pages. The test has already been partly described in the previous article, but this time I would like to focus on subsequent listing pages only.

Googlebot visits further listing pages few hundred times less often than the first page

The first conclusion of the research is that Googlebot’s visits to pages 2, 3 and following of the most popular categories are extremely rare. For example, the first page of the category "Dogs for sale" was visited 550 times (in 37 days), the second page – 5 times, while the third - only once. For the top 50 category pages (first pages of various categories), the number of Googlebot’s visits was more than 140 times greater than for pages 2 and 3.

Similar conclusions can be drawn when taking into account all categories (5,024 unique category and category+region pages visited by Googlebot during the experiment). All the first pages were visited by the Googlebot ca. 1000 times every day, while the second ones only 10 times, third - 7 times and so on.

This means that even though following listing pages are indexed, they may not improve indexation of listed items at all. At least as far as highly dynamic content is concerned (like hundreds of new classifieds added daily).

Linking further listing pages from the first page increases Googlebot visit frequency

The second conclusion is that having pages listed on the first category pages (within pagination links) increases Googlebot interest in them. Pages number 2, 3, 4, 5, 10, 15 are few times more often visited by Googlebot than others:

This also proves conclusions from our previous experiment - the one about testing how crawl priority works. But still, even though linking more pages from the first category page increases their indexation and Googlebot visit frequency, these improvements are so little that may not have any influence on the listed elements indexation (remember that the first pages were visited by 36,000 times in total during whole experiment, and the second pages - only 400 times as you can see above). So linking even 20 following pages on the first category page may not improve indexation of the listed elements.

How to improve indexation of highly dynamic content?

According to the research, we can conclude that pages of listings following the first are very rarely visited by Googlebot, so chances of finding a link on them decreases with increasing page number. This means that in order to improve the indexation we should look for other methods than improving the listings.

If we have trouble with indexing dynamic content, first of all (if possible) we should increase the number of elements listed on the first pages of each category. In my research every page contained 8 ads. This number can be easily increased to 20-25, and in case of less important pages (for example, archives) up to 100.

A good way of improving indexation would also be improving content categorization and crosslinking by introducing more categories, regions, or tags and their proper linking. So even if an element is moved beyond the first category page before Googlebot's visit, it can be found on the first page of more specific categories, tag searches or within "see similar" lists.