Welcome to the Q&A Forum

Hey Moosa,

Since the value of footer links is very low anyway, the likelihood that they may be seen as manipulative would be of far more concern to me - no following them is unlikely to make any sort of negative dent in search visibility. It will, however ensure that the site does not become an easy target for Penguin.

Given the business your client is in, I imagine the most important thing for them is bringing new qualified traffic to their site. The potential for referral traffic from a nofollowed link with anchor text properly crafted to reinforce the brand and entice the click should be the prize they have their eye on.

Hope that helps,

Sha

Hi Zippy-Bungle,

To understand first why the 803 error was reported:

When a page is called, the web server sends header details of what's to be displayed. You can see a complete list of these HTTP header fields here.

One of the headers sent by the web server is Content-length, which indicates how many bytes the rest of the page is going to send. So let's say for example that content length is 100 bytes but the server only sends 74 bytes (it may be valid HTML, but the length does not match the content length indicated)

Since the web server only sent 74 bytes and the crawler expected 100 bytes the crawler sees a TCP close port error because it is trying to read the number of bytes that the webserver said it was going to send. So you get an 803 error.

Now browsers don't care when a mismatch like this happens because Content-length is an outdated component for modern browsers, but Roger Mozbot (the Moz crawler, identified in your logs as RogerBot) is on a mission to show you any errors that might be occurring. So Roger is configured to detect and report such errors.

The degree to which an 803 error will adversely affect crawl efficiency for search engine bots such as Googlebot, Bingbot and others will vary, but the fundamental problem with all 8xx errors is that they result from violations of the underlying HTTP or HTTPS protocol. The crawler expects all responses it receives to conform to the HTTP protocol and will typically throw an exception when encountering a protocol-violating response.

In the same way that 1xx and 2xx  errors generally indicate a badly-misconfigured site, fixing them should be a priority to ensure that the site can be crawled effectively. It is worth noting here that bingbot is well known for being highly sensitive to technical errors.

So what makes the mismatch happen?

The problem could be originating from the website itself (page code), the server, or the web server. There are two broad sources:

1. Crappy code
2. Buggy server

I'm afraid you will need to get a tech who understands this type of problem to work through each of these possibilities to isolate and resolve the root cause.

The Moz Resource Guide on HTTP Errors in Crawl Reports is also worth a read in case Roger encounters any other infrequently seen errors.

Hope that helps,

Sha

Hi fryguy,

A couple of Quick things to check in Google Webmaster Tools:

Index Status: Google Index > Index Status should show you visually whether the site has ever been indexed or not.

Robots.txt tester to check that your robots.txt file is correctly configured.

Fetch as Googlebot Tool to see whether Googlebot is able to successfully crawl your site. You can also use Fetch & Render to see what Google sees on your pages.

Once you get a handle on what might be happening it will be way easier to find a solution

Hope that helps,

Sha

Hi Healeyman,

Tom has pretty much nailed the answer above.

Just one thing to add. If the blogs in question meet all of the criteria Tom has mentioned in his first paragraph and are likely to bring you referral traffic, perfect. If you are just thinking about the possibility that they might influence search engine rankings, your time would probably be better spent looking for opportunities that have a potential to bring people to your site.

Sha

Sorry Marie,

Should have included you in my comments, but could not see that you had commented since I was dealing with a "disappearing answer" catastrophe by composing elsewhere and pasting in.

Why is it only the long ones that do that?

Sha

Hi Elchanan,

Well Eyepaq is batting a thousand today!

Eyepaq is quite correct. The only way to "transfer" the bulk of the link equity is to redirect the domain which would inevitably result in a transfer of the manual action as well. In fact, it is worse than that. In recent times a number of domains have been dealt a manual action by association without a redirect even being in place. These manual actions have been applied because the Webspam team believes that the sites are related and are part of a larger scale manipulative effort, or indeed an effort to get out from under a penalty. Matt Cutts talked about this at SMX West, stating that people should not be able to "move down the road" to avoid a manual action.

If you genuinely needed to build a new site before the manual action and have not put your life savings and years of effort into building a brand, there could be a business case for starting again with a brand new domain, BUT remember you will be starting with nothing - even less than you will have if you successfully clean up the existing domain. This has to be a careful business decision and any new site would need to be completely unique and without any connection to the penalized site. Personally, starting over would be my last resort unless the site was fundamentally broken (and the domain name was a poor choice) to start with.

There are generally five broad reasons why a reconsideration request may fail:

Insufficient data - maybe links in the backlink profile have not been surfaced in the data gathering stage. Incomplete data is common and is best remedied by using as many data sources as possible and in some cases by pulling multiple samples over a period of days or weeks.
(Remember that the links returned by the Webspam team when a reconsideration request fails are "examples". They are intended to point you toward other links in the backlink profile which follow the same patterns or use the same unnatural linking tactics.)

Mistakes in Analysis - If links have been misclassified as natural and are kept, the reconsideration request will fail. Sometimes this happens because people rely solely on algorithmic analysis tools to determine which links to keep or remove and results are not 100% accurate. I would always argue that a real human should be the primary tool when doing analysis because I believe there is no room for mistakes in a job that your livelihood depends upon!
Sometimes human analysis can go wrong too - most often because people forget that this is about "unnatural" linking. That means links that were created rather than earned.
Another mistake that people make at this point is to try to just remove the worst of the unnatural links to preserve some of the benefits that were gained from unnatural linking. Omitting unnatural links from the cleanup effort because you think they are not so bad is a big mistake for two reasons:

1. It will mean leaving unnatural links in the backlink profile - on their own they could cause the manual action to be upheld, but even in the rare instance that this might be allowed to scrape past on reconsideration, retaining them leaves the site vulnerable to the Penguin algorithm
2. It immediately shows the Webspam team reviewer that the site owner's manipulative mindset has not changed. Making a case for reconsideration requires that they are able to trust the site owner will never employ those kinds of tactics again.

Incomplete or ineffective Disavow submissions - As mentioned above, it is always best to disavow at the domain level to ensure that any links you are unaware of do not remain in play and sabotage your efforts. The only exceptions to this rule are unnatural links on high value domains you might reasonably love to have "natural" links from. In these extremely rare cases you should disavow the specific URLs to ensure that any natural links are preserved and any natural links you might earn from that domain in the future will be accorded their rightful value. Also - a red light went on for me when I saw "Google ignored some domains in the two disavow files we submitted". This causes me to wonder whether you have uploaded two completely separate files to the Disavow Links Tool? If so, then this could be the problem. The Disavow Links Tool submission is an overwrite, not an update. This means that you need to combine any existing disavow list with the new list before uploading. If you don't do this then you are effectively re-avowing all of the domains or links that were in the existing file.
If you need to update an existing disavow file with a new list, you can use this free tool to make it easy. Once you have created a free account you can upload your existing list, then upload any new list in the future to create an updated disavow file. When you upload a new list the tool will combine the data, remove duplicates and add date notations so that you can keep track of when domains were added. The tool also ensures that your new disavow file is within the 2Mb file size limit and generates it in the correct text format, ready for submission.

Insufficient effort in the cleanup - Sometimes this is actually just that there is insufficient evidence provided that the work has been done. Most common mistakes here:

1. Omitting domains completely from the cleanup effort because a WhoIs email address is not available
2. Including domains that do not have a WhoIs email address, but not bothering to look any further to find a method of contact. If there are email addresses or contact forms available on the site, a "good faith effort" will require that you have used them to attempt to contact the domain owner.
3. Being unhelpful when requesting that links are removed. The more that can be done to help the domain owner easily locate and remove the links, the better the success rate for the entire link removal campaign.The Webspam team needs to see that a "significant proportion" of the links have been removed. The better the cleanup rate, the smoother the path to getting a manual action revoked.
4. The Bullying approach. Link removal requests should always be written with three things in mind a) You are asking someone to do you a favor b) threats or demands are unlikely to make someone feel that they want to be helpful c) the Brand is at stake here - whatever impression is created by the request will reflect heavily on the Brand. When people get this incredibly wrong, flow-on results can be catastrophic.
5. Not recording and providing evidence of link removal efforts for domains that have not been successfully cleaned up. Keep good records. Provide evidence where domain owners have refused to help or requested payment.Provide  evidence where on-site forms do not function. Make it easy for the Webspam team to make an assessment by providing good documentation.

Not making a case for reconsideration - Site owners need to demonstrate that they understand where they went wrong and will not repeat the same mistakes. In addition to this they need to convince the Webspam team that they have made a "good faith effort" to remove the links. Also, if there are links that are known to be natural, but may look suspicious, address them. Give a reasonable explanation as to why links have been retained (as long as there IS a reasonable explanation). You can use this checklist to make sure you have covered the most important things in your reconsideration request.

This Slide Deck provides an overview that might be helpful.

Any or all of these things can be playing a part in a failed reconsideration effort. It is not uncommon for it to take multiple attempts to have a penalty revoked, but the more of these potential problems we can eliminate by following best practice from start to finish, the more predictable the results.

Best of luck with resolving the manual action and getting things back on track.

Hope that helps,

Sha

Well Hello Mr Fisher

So good to get back into Q&A and to catch up with you too!

Hope it's all good with you and your team,

Sha

Hey William!

I should say thanks for leaving that open for me

I have so been hanging out for the opportunity to get back in and get a Q&A fix lately, but the support phone always takes precedence. It was so good when I made it back in to find some questions that I could contribute to.

Glad you love rmoov!

That started my day off with a big grin

Sha

Hi Kate,

Good call

The nofollow tag will eliminate any unnatural linking issue and in your case the potential to bring qualified traffic to your site by far outweighs any value the links would ever bring in terms of visibility.

Sha

Hi Jampaper,

Just to preface, I spend my days wading through the unnatural links sewer looking at the mess people have gotten themselves into because they thought they were smarter than Google or had that "how would Google ever know" thought in their heads.

EGOL is spot on with his response.

The criteria for undesirable links is not "how would Google ever know it's unnatural?", but "is it unnatural?"

On the "How", here are some things to consider:

1. Google's reach and ability to mine and interpret data (accurately or not) is so far outside our comprehension that it is probably better we don't even think about it.

2. Reviewers have a habit of unitentionally sharing information or creating patterns in the way they do things that are a clear red flag for orchestrated reviews

3. "These reviews always point to inner pages" ...Ooops! There's a pattern

4. "We're obviously targeting authoritative sites which do do reviews" ...Ooops! another pattern

5. Unnatural links on "Authoritative sites" would be more likely to enrage me if I were a member of the Webspam team than those on less influential sites. Let's face it, nobody ever sent me an email suggesting they could sell me links on a crap site

6. (and this you should take as very tongue in cheek, but perhaps give some thought to implications)
   This site has upwards of 400,000 community members. One of them is a guy who is currently on leave from his job at G, but occasionally comments on Moz blog posts that interest him (that's the tongue in cheek part as while it is possible, I seriously doubt he or any of the other Googlers who might be members spend time combing through this site looking for extra work!)

   However, it doesn't take much imagination to think there may be other people out there who could be made aware and if they were a certain kind of person might be likely to look into a backlink profile and perhaps lodge a report. Once the manual review process comes into play, the cleverness of the algorithm is irrelevant.

When you have a great product your customers will always be your best sales force! Do things that make THEM want to tell people how THEY feel about you. If you do that enough, even those Authoritative sites will be checking you out for themselves and gifting you natural links

Hope that helps,

Sha

Hi Zippy-Bungle,

To understand first why the 803 error was reported:

When a page is called, the web server sends header details of what's to be displayed. You can see a complete list of these HTTP header fields here.

One of the headers sent by the web server is Content-length, which indicates how many bytes the rest of the page is going to send. So let's say for example that content length is 100 bytes but the server only sends 74 bytes (it may be valid HTML, but the length does not match the content length indicated)

Since the web server only sent 74 bytes and the crawler expected 100 bytes the crawler sees a TCP close port error because it is trying to read the number of bytes that the webserver said it was going to send. So you get an 803 error.

Now browsers don't care when a mismatch like this happens because Content-length is an outdated component for modern browsers, but Roger Mozbot (the Moz crawler, identified in your logs as RogerBot) is on a mission to show you any errors that might be occurring. So Roger is configured to detect and report such errors.

The degree to which an 803 error will adversely affect crawl efficiency for search engine bots such as Googlebot, Bingbot and others will vary, but the fundamental problem with all 8xx errors is that they result from violations of the underlying HTTP or HTTPS protocol. The crawler expects all responses it receives to conform to the HTTP protocol and will typically throw an exception when encountering a protocol-violating response.

In the same way that 1xx and 2xx  errors generally indicate a badly-misconfigured site, fixing them should be a priority to ensure that the site can be crawled effectively. It is worth noting here that bingbot is well known for being highly sensitive to technical errors.

So what makes the mismatch happen?

The problem could be originating from the website itself (page code), the server, or the web server. There are two broad sources:

1. Crappy code
2. Buggy server

I'm afraid you will need to get a tech who understands this type of problem to work through each of these possibilities to isolate and resolve the root cause.

The Moz Resource Guide on HTTP Errors in Crawl Reports is also worth a read in case Roger encounters any other infrequently seen errors.

Hope that helps,

Sha

Hi Eric,

This should give you the basics:

• Add Keywords to Product Titles and Descriptions
• Add UPC Codes (required)
• Add Shipping cost info (required)
• Add Seller Reviews
• Add Rich Snippets
• Optimize your Product_Type column
• Only use Google's product taxonomy for google_product_category

Remember not to confuse product_type and google_product_category. Both should be used, but the latter must use the correct taxonomy. Taxonomy is downloadable from the Google Merchant Center.

Do some careful reading on that page and throughout the Merchant Center. This is one area where crossing t's and dotting i's is VERY important!

Even though google_product-category is not required for all countries and products, recommendation is to use it for ALL products. Taxonomy is available for download as both .txt and excel spreadsheet.

Make sure your feeds are clean and push them every day.

Best of Luck!

Hi Ryan,

All great advice here (wearing out my thumb up guys!).

Just wanted to suggest that you try running the keyword terms it help and computer help. I just ran them in the Google Adwords Keyword Tool (exact match) and it returned some related terms with decent search and low/medium competition that could work for you.

Also, any time you are struggling to come up with the right terms for a site, try just asking people you know what they would search for if they needed ".................". Sometimes the bleedingly obvious can be the last thing we think of and other times people come up with what I think are the weirdest terms that are frighteningly common!

Once you have chosen some terms that might be suitable, you can also use the Keyword Difficulty & SERP Analysis Tool to help identify the strengths & weaknesses of sites already ranking in the Top 10 for that term. If you have not used the tool before, Rand's post The Best Kept Secret in the SEOmoz Toolset gives a good explanation of how you can use it.

Hope that helps,

Sha

Hi Charles,

I would say

1. Keep an eye on #Mozcon and connect with some people before you get to Seattle
2. Stay over and take a Tour of the Mozplex
3. Spend some real time checking out the Speaker photos on this page so you know who they are and what they look like...there is a real chance you will find yourself standing next to them in the lunch queue, sitting down to breakfast or lining up to sing Karaoke with them!
4. Be at the conference early every day and try to sit with new people as well as catch up with friends you have already made.
5. Read this post from Zeph , hit him up on Twitterand get him to hook you up for a spot at the informal Meetup on the night before the conference
6. Make a list of awesome people that you would like to meet while there and keep it close handy
7. Make a donation to the Myrtle Point Library in honor of Michael Cottam's parents at the conference and help send love to Michael and his family from everyone in the Moz "family"
8. Get some tips on where to find the Best coffee in Seattle from Jonathon Coleman
9. Arm yourself with Rand's Restaurant and Bar Guide to Seattle and the The Not-so-Short Shortlist of Moz's Top Seattle Restaurants, Bars, and Activities for MozCon 2013
10. Bring business cards and make sure you make notes on the cards people give you so you can easily put names to faces and shared experiences when you get home.

Oh...and you might consider joining a PosSEO, becoming a PosSEM or something like that, but whatever you do, say G'day!

Sha

Hi Eric,

A quick look in Majestic SEO shows nearly 34,895 external backlinks from 260 linking root domains with large numbers of links being added during March and April.

As mentioned by Bradley and Chris, there is a lot of suspect anchor text (including cialis, adipex, tramadol etc) in there and many of the referring sites are not places would really want to be getting your links from regardless of the anchors.

It seems your site has attracted some unwanted attention which has dramatically lowered Google's trust. While the obvious conclusion is that Penguin might be responsible, it is clear from the screenshot you provided that the dropoff actually happened around 5/6 May (Dr Pete has talked about "something" happening with the algorithm around 5 May and that would seem to correlate with your problems). Of course, it is possible that this was a test rollout of Penguin and your site was captured as a result, or it may be something completely different (which Google has given no information about).

Either way, it is easy to see that the linking behavior is harming your site's trustworthiness and unfortunately, this is likely to also affect your visibility in Bing and other Search engines as well.

Dealing with the backlink problem is the most critical place to start ... site updates will come later I'm afraid.

Hope that helps,

Sha

Hi Rob,

First of all ...this is not a domain redirect.

What they are actually doing is pulling the content of your site into their own using an iframe.

They are not able to do anything through your site by doing this as the content is rendered by the browser and not their server.  So, the question is why they would do it.

Best guess: This could be someone who is planning to set up some kind of low quality site (possibly full of ads), but wants to build up backlinks for the domain. They can go to blogs, forums etc and leave comments & posts with their URL. The webmaster checks the URL and sees your site, so approves the comment or post...after a few months of doing this, BAM! they remove the iframe and let loose their real content.

hmmm...

Sha

Hi Croozie,

Awesome work once again from Ryan!

Since your question feels like a request for suggestions on "how" to create a solution, just wanted to add the following.

When you say "classified listings" I hear "once off, here for a while, gone in 45 days content".

If that is the case, then no individual expired listing will ever be matched identically with another (unless it happens to be a complete duplicate of the original listing).

This would mean that it would certainly be relevant to send any expired listing to a higher order category page. If your site structure is such that you have a clear heirarchy, then this is very easy to do.

For example:

If your listing URL were something like http://www.mysite.com/listings/home/furniture/couches/couch-i-hate.php, then you can use URL rewrites to strip out the file name and 301 the listing to http://www.mysite.com/listings/home/furniture/couches/, which in most cases will offer a perfectly suitable alternative for the user.

There is another alternative you could consider if you have a search program built in - you could send the traffic to a relevant search. In the above example, mysite.com/search.php?s=couch.

Hope that helps,

Sha

Hi waspman,

No, you are absolutely not doing anything wrong

The simple answer is that you are acquiring links primarily to the sub-domains and not the root domains.

To explain:

"mozRank is SEOmoz's global link popularity score. It compares the relative link value (ranking power) between URLs on the Internet. It is similar in premise to Google's original PageRank metric but is updated more frequently and offers greater precision."

You can read more about all of the SEOmoz Metrics at the Open Site Explorer explanatory page.

So, if you go to Open Site Explorer and run a query for your sub-domain you will see the number of external links to that URL. If you then remove the www and run the query for the root domain you will see that there are 0 links to that URL.

Since mozRank = link popularity - it should be higher for the sub-domain because that is where the links are pointing, and of course you want to ensure that your links are always consolidated to just one domain. So, things are exactly as they should be.

Hope that helps,

Sha

Hi Rick,

If you wish to use the robots.txt method to disallow all or part of your site's https protocol, you simply need to load two separate robots.txt files.

The http and https protocols are basically viewed by bots as if they were two completely separate root domains (which I guess you already know as you have mentioned the fact that port 443 is used for the secure protocol).

Google's advice is that to use this method, you should have a separate robots.txt file for each protocol with code as follows:

For your http protocol (http://www.startuploans.org/robots.txt

User-agent: *
Allow: /

For the https protocol (https://www.startuploans.org/robots.txt

User-agent: *
Disallow: /

However, blocking crawlers with robots.txt is not the most reliable method for excluding pages from Search engines. The reason for this is that the page will continue to be indexed if it happens to be found via a link from another page. Basically, the robots.txt is the sign on the front door that says "Please stay out of our house", but it is never seen by the people who enter via the rear exit or climb in a window!

The most reliable method of excluding pages is to add the noindex meta tag as suggested by MagentoWebDeveloper and Alan.When a bot encounters the noindex meta tag it will send a signal to the search engine to de-index the page and there is no further problem.

I would generally use noindex, follow rather than noindex, nofollow as the nofollow tag will stop the flow of link value through your site. In most cases, as long as the noindex is in place, there is no reason to be worried about the links on the pages being followed.

You should NEVER use both methods at the same time.

Hope that helps,

Sha

Hi Paul,

Great answers from Ryan and EGOL.

The other thing to keep in mind is that traffic doesn't automatically equal business.

While it may seem attractive to be "free from Google" and other search engines, there is an important consideration that must not be forgotten.

If the Lion's share of your traffic is perhaps from RSS subscribers or referrals from more general sites, how many of those visits are actually delivering you people who need a lawyer?

When I need a lawyer for myself or a family member (when my need is immediate and I am ready to contact the one that seems best able to help me), what am I going to do?

Will I:

• Look for a blog to subscribe to?
• Surf around and read other websites looking for links to lawyers that might help me?
• Go to my favorite search engine and search on a term that will take me straight to a list of local law firms in that practice area?

While diversity of traffic sources is important, the reality is that if they were not the most useful means of finding what you want on the web, search engines would not still hold the place they do.

I have a number of clients who are lawyers and would agree that the approach Ryan has described for you is the best way to go. In addition to that I would say that the key in developing content for them has always been remembering the mindset of the client. A person who needs a lawyer is a person who needs help. The practice area and seriousness of the situation will dictate what type of help.

Hope that helps,

Sha