Welcome to the Q&A Forum

Hi! - I wrote this guide a few years ago on penalty recovery which may help you as it contains a lot of methods around auditing the links - https://moz.rankious.com/_moz/blog/ultimate-guide-to-google-penalty-removal

If we were to approach a product with 70k URLs. We'd do the following steps:

1. Pull all the URLs into a Spreadsheet
2. Split the URLs into domains
3. Filter the URLs are search for common spammy words. e.g 'Link', 'Best', 'Free', 'Cheap', 'Dir', 'SEO' etc (mark as spam accordingly)
4. Run contact finding across all URLs using a tool such as URL Profiler with Whois Lookups
5. Filter by contact name and find duplicates (mark as spam accordingly)
6. Filter by website type and mark as spam accordingly
7. Manually check remaining links

By working through by domain, you'll rule out thousands of spammy links very quickly. Though 70k will ultimately take a few solid days of work.

Hope this helps,

Lewis

As Logan pointed out, the most likely cause of this is because of the canonical tags pointing to the HTTP variation. I'd probably suggest carrying out the following:

Download a tool like Screaming Frog SEO Spider and crawl your full website - https://www.screamingfrog.co.uk/seo-spider - Once the crawl has run, export the data and check both the URLs and the Canonicals across all pages to ensure only HTTPS variants appear.

If you find http variants, then it's likely that the redirect rules you have in place aren't working. This is a global rewrite rule that can be applied to force the https:// version to show at all times, so you shouldn't need to make the changes once by one. I'd also make sure that if you're linking to any pages internally, that you're linking to them using a relative path, or straight to the https:// version. If your canonicals are causing issues, just ensure they're always choosing the HTTPS version and you shouldn't have any issues!

Finally, in GSC, just make sure you've got the HTTPS version of the site setup and that your sitemap includes only HTTPS URLs.

Hope this helps!

I'm not sure the answers previously presented are related to the issues you're having. Having worked with Magento for a long time, this can be an issue that occurs over and over again.

To answer your initial question, truncating your core_url_rewrite table will remove all of these URLs, but it'll only delay the problem until it reoccurs again in the future (unless you've had a problem in the past which has been rectified). You're also correct in that any rewrites in the system previously there will disappear, so you'll probably end up with a lot of crawl issues appearing in Search Console.

Your best move would be to find out why you have so many URLs in there in the first place. Do you have a huge product catalog with multiple stores? Or is this something to do with an issue in your Magento version or some setup issues. The most common time this usually occurs is if two products get added to your site with the same URL Key. Every time the reindex process runs, your core_url_rewrite table will grow. You could check this by looking at the number of rows in the table, reindexing the site and if it grows further, then it's likely to be the problem. The quickest way to fix this is to ensure all URL key are unique.

There's also an article here about duplicate keys - https://firebearstudio.com/blog/magento-url-reindex-core_url_rewrite-duplicates-patch.html - this should hopefully clear the issue.

I hope this helps! If it doesn't solve the problem, then sending over a little more information around the number of stores, catalog site and the split between system generated URL rewrites and custom URL rewrites would be great so we can try to help further!

Thanks,
Lewis

Hi!

We do a lot of consultancy for Magento projects and this is a question that comes up quite regularly as it can't really be handled perfectly straight out of the box with Magento.

Every implementation is a little bit different, but I'll put together some recommendations below based on the information available at the moment.

For your faceted navigation, you ideally don't want to index any of these pages, unless you believe that you'll rank in your own right for specific filters (e.g. Colour, like you pointed out in your last message).

That then comes with some additional complications. In Magento, if you have 3 colours available in the faceted nav, you'll have all the different variations indexed in each combination.

For example:

Blue
Black
Red

Blue + Black
Blue + Red
Black + Red
Black + Blue
Red + Blue
Red + Black

Magento as standard doesn't always keep the filters in the same order, so you can end up with literally thousands of pages ending up in the index for a relatively small number of attributes being shown on your pages.

There are a few recommendations here:

1. Go and look at the MageWorx Ultimate SEO Suite Plugin - http://www.mageworx.com/seo-suite-ultimate-magento-extension.html - For $249, it solves a lot of issues Magneto has straight out of the box and gives you ultimate control over your meta titles.

What you want to do is set all of your facets to 'NOINDEX,FOLLOW' where possible. This will reduce the number of URLs in the index gradually. An example of this would be adding ?min=* and mode=* etc (grid/list variants).

2. For your canonicals, you're probably best setting the canonical to the current filtered page (for example, if you're on a category page with colour = blue selected in your faceted nav, you'd have this URL as your canonical). Some sites we work on have it setup so the canonical points to the category URL (like you currently have).

Finally, you probably want to build an extension to allow you to inject content into the filtered content pages. If you're using an extension like ManaDev for your facet navigation, this can be achieved fairly easily and allows you to add a block of text to each filter applied on a page.

You should also look to request each of the incorrectly indexed URLs is removed from the index (although this does take a long time if you have a lot!).

We wrote a really long guide around launching a Magento website last month which may be of interest - https://www.pinpointdesigns.co.uk/the-definitive-guide-to-launching-a-magento-website/. We've also done a guide on Common Magento SEO Issues here - https://www.pinpointdesigns.co.uk/common-magento-seo-issues/ and I previously wrote a guide on setting Magento up for Search Engines on Moz - https://moz.rankious.com/_moz/ugc/setting-up-magento-for-the-search-engines (Although this is likely to be a little outdated now)

I hope this helps!
Lewis

Hi Luke,

What data are you trying to get from Google Webmaster Tools?

They have an export for features like backlinks that you can put into something like Google Docs, but it depends on what data you're wanting to collect. I would guess that depending on your requirements, someone will have built something to pull the information out into an external system.

They've also got an API available which would allow you to develop your own application and pull out a selection of data fairly easily - https://developers.google.com/webmaster-tools/

Many thanks,
Lewis

You're right in with the **Disallow: /fish **in the robots file blocking all those initial links, but if you wanted to block everything inside the /en/ folder, you would need to do disallow: /en/fish

You could use a wildcard in the robots.txt file to do something along the lines of Disallow: /BTS-*

This _'should' _work, but it's always worth checking using a tool to make sure it's all implemented correctly. Distilled did a post a while back about a JS tool which allows you to test if robots.txt files work correctly which can be found here - http://www.distilled.net/blog/seo/js-bookmarklet-for-checking-if-a-page-is-blocked-by-robots-txt/

In addition to this, you could also use the 'blocked URLs' tool in GWT to see if the pages are successfully blocked once you've implemented the code.

Hope this helps!

Hi all,

I've just tried adding a new campaign to our Moz analytics account but it's saying that it already exists. I actually removed all of the campaigns we had in Moz a while ago, so it doesn't already exist.

Any one else had the same issues and if so, how did you fix it?

Thanks,
Lewis

Hi Manan,

If you go to http://pro.moz.com/campaigns - then go to Global Settings, you can upload your logo for the branded reports.

Someone posted a similar message last week and I think they were referring to Moz Analytics. I can't find the Q&A post now but I'm sure a member of staff responded saying that Branded reports weren't available as of yet on Moz analytics so that might be where your issue is.

Hope this helps,
Lewis

Hi Zach,

Yes, you should be indexing these pages and ensuring you add them to your sitemap. On the basis the content is unique and provides a benefit to the people reading your blog, then there would be no reason to not index the pages.

Hope this helps.
Lewis

Have you received any notifications in your Google Webmaster Tools account with regards to a penalty? You can usually find this by going to https://www.google.com/webmasters/tools/ and clicking on 'Search Traffic > Manual Actions'. Alternatively, you might have a message in there.

This would only show for a manual penalty, so if you've received a penalty due to an algorithm, this is a bit harder to spot. The easiest way to do this is look at the date your traffic dropped and comparing this with the updates that Google have released - http://moz.com/google-algorithm-change - The moz link there has a list of all known updates.

Alternatively, you could look at the following plugin which will overlay over a chart over the top of your analytics results and give you an indication of when penalties have come in along with your traffic drops.
https://chrome.google.com/webstore/detail/chartelligence/njhdcfdiifemfnfddhfjmfbkajajceag?hl=en

I wouldn't have thought that moving hosting provide / IP would cause you any huge drops in the search engines. Maybe a little bit of fluctuation, but nothing significant. Where is your new server located in relation to your store location?

EDIT: I started writing my response before any other comments were posted, so looks like Peter beat me to it on most of the points

Hi Zach,

Yes, you should be indexing these pages and ensuring you add them to your sitemap. On the basis the content is unique and provides a benefit to the people reading your blog, then there would be no reason to not index the pages.

Hope this helps.
Lewis

Have you received any notifications in your Google Webmaster Tools account with regards to a penalty? You can usually find this by going to https://www.google.com/webmasters/tools/ and clicking on 'Search Traffic > Manual Actions'. Alternatively, you might have a message in there.

This would only show for a manual penalty, so if you've received a penalty due to an algorithm, this is a bit harder to spot. The easiest way to do this is look at the date your traffic dropped and comparing this with the updates that Google have released - http://moz.com/google-algorithm-change - The moz link there has a list of all known updates.

Alternatively, you could look at the following plugin which will overlay over a chart over the top of your analytics results and give you an indication of when penalties have come in along with your traffic drops.
https://chrome.google.com/webstore/detail/chartelligence/njhdcfdiifemfnfddhfjmfbkajajceag?hl=en

I wouldn't have thought that moving hosting provide / IP would cause you any huge drops in the search engines. Maybe a little bit of fluctuation, but nothing significant. Where is your new server located in relation to your store location?

EDIT: I started writing my response before any other comments were posted, so looks like Peter beat me to it on most of the points

Hi Manan,

If you go to http://pro.moz.com/campaigns - then go to Global Settings, you can upload your logo for the branded reports.

Someone posted a similar message last week and I think they were referring to Moz Analytics. I can't find the Q&A post now but I'm sure a member of staff responded saying that Branded reports weren't available as of yet on Moz analytics so that might be where your issue is.

Hope this helps,
Lewis

Like Alejandro has explained, the header.phpxxx isn't actually being used. .phpxxx is not a valid php extension, so they've added the xxx in so it isn't used in the wordpress theme.

Looking at the code in header.php, your title tag is here:

<title><?php if( is_home() || is_front_page() ) bloginfo('name'); else wp_title(); ?></title>

This basically means:

If you're on the homepage or on the front page of the website, show the blog name (which is pulled from your wordpress settings). If not, show the wordpress title (this will be pulled from the name of the page which the user is on).

If you wanted to make it so the order of the title page is different, for example, if you wanted your title tag to show 'Page Title Here | Site Name', you could go into your wordpress settings and should be able to change the order in which this is done... Alternatively, download a plugin such as Yoast's SEO plugin and you'll be able to customise everything there!

Hi Luke,

What data are you trying to get from Google Webmaster Tools?

They have an export for features like backlinks that you can put into something like Google Docs, but it depends on what data you're wanting to collect. I would guess that depending on your requirements, someone will have built something to pull the information out into an external system.

They've also got an API available which would allow you to develop your own application and pull out a selection of data fairly easily - https://developers.google.com/webmaster-tools/

Many thanks,
Lewis

Hi Iris,

It's hard to debug the problem without being able to see the issue, but it sounds like it's an incorrect file path. Open up your index.php file and find the following:

require('./wp-blog-header.php'); change it to this:

require('wp-blog-header.php');

Hopefully that will help fix your issue!

Lewis

It should just be the one inside the standard public_html folder as this is what is referenced in the PHP error (/home2/damian/public_html/index.php on line 17).

Hope this helps! Let us know how you get on.

Glad it worked!

Yeah, you just need to make the same change to the following index.php file and that will solve that problem too:

public_html/villasdiani.com/wp-content/themes/decorum/index.php

That should get you all up and running.

Sorry, I copied the wrong file path. I meant this one:

public_html/villasdiani.com

No problem - Glad I was able to help.

Best of luck in the future!
Lewis