But suddenly - this week - my rankings for ALL my keywords on ALL search engines disappeared. The nice page breakout and descriptions in search results that I had earned for services, contact and other pages that had been stewing online for the last 7 years were gone too. SHIZZ NITS!
So Karebear started freaking out. Went into my Google Webmaster Tools to see an alert, from this last week, to inform me that Google could no longer access my site. Do what? I had made no changes to this file.
I've either been a victim of a hack, or perhaps negative SEO. So let's investigate this Robots.Txt Mystery.
Nutshell - What is a robots.txt file? This is a text file in the root directory of your site that tells Google and other search engines whether they have permission to crawl your pages. It looks something like this:
User-agent: *
Disallow: /directory-you-do-not-want-to-be-crawled/
Disallow: /file-you-do-not-want-to-be-crawled.html
Sitemap: http://www.yoursite.com/sitemap.xml
It is a best practice to have a robots.txt file, even if you allow your entire directory to be crawled. Here's what I found last night:
User-agent: *
Disallow: /
This file is telling Google, and ALL other search engines - NOT to crawl ANY of my site. Disallow from the root directory onward. Not good.
So I went into panic mode, and made some edits and am now hoping, praying that this resolves itself quickly and will be watching closely over my analytics accounts. I'd like you to say a prayer too, please. :)
So now, my next question is - why did this file change? First thing in head - could file have been hacked? Short answer - yes. Could a competitor hack your robots.txt file? Yes, by hacking your robots.txt file so it blocks the site and then submitting a removal request to Google - your competitor could theoretically block your site from search results. Entirely.
First, there's this article on gaining 'competitive intelligence' by reviewing directories of competitor websites. And even better - is this story from a few years back on how a hacker used proxy hacking to effect a site other than their own. But this one, takes the cake. It's about negative SEO and how it can dramatically reduce your rankings on the internet. Here's an excerpt written by Scott Langdon, contributor/author from www.HigherVisibility.com:
For those who are unfamiliar, negative SEO is when one website either hacks into your website and makes negative changes, reports your website to Google for no reason, or purchases spam links and then points them to the sites of competitors. The idea is that your competitor will be penalized by Google; thus allowing your site to rise above. This is seen as unethical and in most cases it is considered worse than any of the other black hat SEO tactics.
What can you do?
Your first step is to understand that robots.txt is an important file - and if not done correctly, or if there is a change - it can effect your SERPS in a HUGE way. That is, eventually dropping you completely if bots can't access your content.
Then - DO NOT let your robots get lonely. Or your website reporting for that matter. CHECK Analytics and Webmaster Tools Site Health regularly. If you see any changes, or decrease in traffic - as usual - take appropriate measures.
Use your webmaster tools to check your back links and do searches on yourself and your company. People are mean, try to stay on top of them. If people are wrongly linking your site, stealing your work (I just pasted in the first paragraph of my homepage in quotes to Google and found my copy stolen VERBATIM on another web designer's site. Nice.) or trying to sabotage your efforts in some way. A good side benefit of this is also checking reviews and for online reputation management.
Bottom line: I'm not sure yet what happened. I only have my guesses. I'll certainly report back and let you know everything I find out. If you find you've been a victim - it could be too late as it takes time to index and rank sites. You've been working hard for your positions - and especially if you depend on your rankings for income, this can be devastating to your business. So to me - the most important part of this is to stay on top of your positioning. If you notice a drop, investigate. Check your analytics and webmaster tools regularly, and if it's been several weeks - seek the help of a professional.
**UPDATE** - some nice folks on the Google Webmaster Tools Forums advised me that it is VERY rare for someone to hack your site and only change a robots.txt file. They would be doing other things, and wreaking havoc elsewhere. Negative SEO and Google Proxy stuff wasn't mentioned - and it's too over my head anyway. They did mention that the google bots 'crawl' your site and it has nothing to do with indexing. I guess this means that they just can't access the information to display, and that hopefully the index / search results will not be effected since I caught it early. They pretty much chalked it up to user error. Whatever the case - I will from now on check my rankings and analytics VERY regularly - so I can stay on top of this.
