Competition Sabotage? Hacker? What You Need To Know About ROBOTS.TXT and SEO

 

Does your site have a robots.txt file? Mine does. In fact, my website has had the same robots.txt file since 2008, barring adding or removing a non-relevant directory from the search bots.

But suddenly - this week - my rankings for ALL my keywords on ALL search engines disappeared. The nice page breakout and descriptions in search results that I had earned for services, contact and other pages that had been stewing online for the last 7 years were gone too. SHIZZ NITS!

So Karebear started freaking out. Went into my Google Webmaster Tools to see an alert, from this last week, to inform me that Google could no longer access my site. Do what? I had made no changes to this file.

I've either been a victim of a hack, or perhaps negative SEO. So let's investigate this Robots.Txt Mystery.

Nutshell - What is a robots.txt file? This is a text file in the root directory of your site that tells Google and other search engines whether they have permission to crawl your pages. It looks something like this:

User-agent: *
Disallow: /directory-you-do-not-want-to-be-crawled/
Disallow: /file-you-do-not-want-to-be-crawled.html
Sitemap: http://www.yoursite.com/sitemap.xml 

It is a best practice to have a robots.txt file, even if you allow your entire directory to be crawled. Here's what I found last night:

User-agent: *
Disallow: / 

This file is telling Google, and ALL other search engines - NOT to crawl ANY of my site. Disallow from the root directory onward. Not good.

So I went into panic mode, and made some edits and am now hoping, praying that this resolves itself quickly and will be watching closely over my analytics accounts. I'd like you to say a prayer too, please. :)

So now, my next question is - why did this file change? First thing in head - could file have been hacked? Short answer - yes. Could a competitor hack your robots.txt file? Yes, by hacking your robots.txt file so it blocks the site and then submitting a removal request to Google - your competitor could theoretically block your site from search results. Entirely. 

First, there's this article on gaining 'competitive intelligence' by reviewing directories of competitor websites. And even better - is this story from a few years back on how a hacker used proxy hacking to effect a site other than their own. But this one, takes the cake. It's about negative SEO and how it can dramatically reduce your rankings on the internet. Here's an excerpt written by Scott Langdon, contributor/author from www.HigherVisibility.com:

For those who are unfamiliar, negative SEO is when one website either hacks into your website and makes negative changes, reports your website to Google for no reason, or purchases spam links and then points them to the sites of competitors. The idea is that your competitor will be penalized by Google; thus allowing your site to rise above. This is seen as unethical and in most cases it is considered worse than any of the other black hat SEO tactics.

What can you do? 

Your first step is to understand that robots.txt is an important file - and if not done correctly, or if there is a change - it can effect your SERPS in a HUGE way. That is, eventually dropping you completely if bots can't access your content.

Then - DO NOT let your robots get lonely. Or your website reporting for that matter. CHECK Analytics and Webmaster Tools Site Health regularly. If you see any changes, or decrease in traffic - as usual - take appropriate measures.

Use your webmaster tools to check your back links and do searches on yourself and your company. People are mean, try to stay on top of them. If people are wrongly linking your site, stealing your work (I just pasted in the first paragraph of my homepage in quotes to Google and found my copy stolen VERBATIM on another web designer's site. Nice.) or trying to sabotage your efforts in some way. A good side benefit of this is also checking reviews and for online reputation management.

Bottom line: I'm not sure yet what happened. I only have my guesses. I'll certainly report back and let you know everything I find out. If you find you've been a victim - it could be too late as it takes time to index and rank sites. You've been working hard for your positions - and especially if you depend on your rankings for income, this can be devastating to your business. So to me - the most important part of this is to stay on top of your positioning. If you notice a drop, investigate. Check your analytics and webmaster tools regularly, and if it's been several weeks - seek the help of a professional.

**UPDATE** - some nice folks on the Google Webmaster Tools Forums advised me that it is VERY rare for someone to hack your site and only change a robots.txt file. They would be doing other things, and wreaking havoc elsewhere. Negative SEO and Google Proxy stuff wasn't mentioned - and it's too over my head anyway. They did mention that the google bots 'crawl' your site and it has nothing to do with indexing. I guess this means that they just can't access the information to display, and that hopefully the index / search results will not be effected since I caught it early. They pretty much chalked it up to user error. Whatever the case - I will from now on check my rankings and analytics VERY regularly - so I can stay on top of this.

Pinterest Holding Interest

It's the newest hottest thing on the internet - it's Pinterest. An online 'pin board' or, as I like to call it - a 'vision board' - where you can identify pictures, and in some cases, 'do' - things, visually - as well as share and interact with others.

Who uses it? It appears that women, in their 30's-50's - in general, are highly addicted to Pinterest. And - with good reason. It's a gold mine for interior design ideas, recipes (here's where the DO comes into play), crafting, hobbies, hairstyles, beauty and fashion and more.

What's great about Pinterest, is that it's not so much 'personal' - i.e. - pictures of the family dog and family outings - but things you can share freely and comment on without worry that you've barged in on someone's privacy. If it's on Pinterest, it's free for the pinning.

Any business value? For most businesses - I think that trying to incorporate Pinterest into your social media strategy would depend 100% on what you do. Also - it MUST be kept friendly, light, family-friendly and social. Certain businesses just do not fall into this category, and are not a good fit for Pinterest. For example - here are some industries/businesses that would do well to consider utilizing Pinterest as a vehicle for viral exposure:

1/ Hair salons/independent stylists: Showcase your work! Get permission from your clients to present your newest creations on Pinterest. Much inspiration for that next great hair-do comes from here.

2/ Interior Designers/Artists/Designers: Here's where you show off your brands, design ideas, colors, palettes, layouts and also offer great ideas to those who are crafty like to do-it-yourself. Even though they do-it-themselves - they remember where they got help and support - and if/when the time comes, maybe your name will pop back into mind. "How-tos" for art/craft projects and interior decor are a hugely searched - so participate with helpful content and gain some (real) fans.

3/ Chefs, Restaurants, Cooks: Share recipes on Pinterest - it's a great source of people who cook EVERY SINGLE DAY - and like above, they'll remember where they got the recipe. What's great about Pinterest - is the viral shareability of this content. Your recipe has the potential to get shared to hundreds of people, within minutes. Talk about free exposure!

4/ Photographers/Scrapbookers: Pinterest is a place for the crafty!! You can not only showcase your work, but also provide those handy-well searched-how-to's to get even more exposure. Play up the holidays and special occasions - Valentine's Day, Birthdays, Weddings - as people are continually searching for ideas and maybe - vendors - for these special occasions.

5/ E-Commerce: If you're selling something in the world of beauty, fashion or style via an online store (of course you know we specialize in e-commerce website design - close shameless plug) - then Pinterest will allow you to 'pin' individual items. Do this when you get something particularly relevant to these industries, or new items - or even if you have something on sale.

6/ Food, Fashion Writers/Mom Bloggers - ladies, unite. 'Nuff said.

And that's not to say men aren't using Pinterest. They are. Just not like the ladies. Plus, for the most part, I know better how ladies think than I do men; and I know this is a smorgasboard of home, beauty and family inspiration. It is what it is, and right now, it's more about the women on Pinterest.

Tips for using Pinterest: Pin Often, pin relevant, take special care to categorize and beef up your specific boards - and add your own description to reach image. Then, share it on Facebook and Twitter, Linked In and anywhere else you can.

There's not a lot of SEO value for Pinterest pins, because each pic pinned has an id number, and nt a real 'title' per-se - which makes sense, because there's just so much content. It's the loyal followers, share-ability and genuine inspiration that exists on Pinterest that make it so valuable.

If you know your market, and pin for them, (and yourself) it'll be a fun and interesting way to increase exposure for your business or your name.

SEO, Social Media and Your Website

JUST SAYIN'...

SOOOO many clients ask me about search engine optimization, and then I spend an hour typing out this extensive twenty-three paragraph email on "tips and tricks" to help their site be seen on the search engines. While my typing skills are great, (ok, maybe not great - but good enough) this was taking a lot of my time, and I don't have much to spare. So, I just decided to write a blog post about it.

When I do a website for any client (informational website) I do my best to make sure that the markup is up to W3C Standards, I will plug in the META data, try to use useful file names and all that jazz. And certainly - you can hire me to help you with site optimization.

In fact, I am going to put together a basic "SEO/Social Networking Support" package in the next few weeks to support my clients with who have SEO and social networking support needs. While I can NOT guarantee rankings - I CAN guarantee that EVERY SITE OWNER should be already be making sure these simple (but time consuming) things are done - because along with valid markup and META data, they are things that will help keep your site in the natural running for rankings.

So, before I get to putting together a package - and - if you are a 'do-it-your-selfer' - you can use these tips to start a basic SEO/Internet Marketing plan for your website.

Everything I learned - I learned from trial and error, fiddling around with my Google accounts, and books. Great websites like SEO BOOK (GREAT tutorials, forums, articles tools and such!). Testing what people like, what they don't like - what they click on - what they don't. It's been several years in the process, putting websites online, watching how search engines pick them up, index and rank them - and the parameters that control rankings constantly change and evolve. And - every site and it's needs, target audience, etc - is different. But there are some basic things that I can tell you EVERY website owner should be doing to promote new traffic to their website.

1. Get This Book.
The Ultimate Guide To Search Engine Optimization - this book speaks volumes about both the basics, and tons of tricks to help enhance your site's visibility on search engines. What's best about it - is that it's written in easy-to-read language. You don't have to be a rocket scientist (or HTML expert) to read it. Read it from start to finish - because it covers the basics of beginning with an accessible, optimized website, then on to cover the basics of SEO, (META data, titles, alt tags, keywords, linking) PPC management, (Pay-Per-Click) internet marketing, article writing, and conversion testing and tracking. Jon Rognerud - the author, is quite the expert. And while you're there - sign up to download his FREE SEO Guide, and his FREE SEO TIPS by email. They're gems.

2. Get Online.
Use the free stuff, BABY! Get a BLOG (we can design a really nice blog design for you) - and get on Facebook, Twitter, LinkedIn, Bizzy.com - create a page (or, lens) on Squidoo - and then (THIS IS IMPORTANT!) spend a few hours every week doing something with it! For your blog - write something about your business. Write something about what you do. Offer a special, a valuable piece of information, educate your reader or audience on something they want to hear. But keep it relevant! Don't be writing about tennis - if you're selling telephones. Unless of course, your telephone is a special telephone that is just for tennis players. Then, share your blog post on Facebook, Twitter - anywhere you can! Scream it to the world!

AND HERE'S A BONUS - Jon Rognerud - the author of aforesaid gotta-have SEO book - is offering a FREE GUIDE to Facebook called FAST FACEBOOK right now on his website SocialMediaYeah.com - Learn how and why Facebook can promote your business on the internet.

3. Get Listed.
Submit your site to directories (again - relevant ones!). Get listed in the general directories (DMOZ is free, Yahoo! Directory has an annual fee) in the proper categories. There are also content-specific directories where you can get listed. For example - Kari and Company is listed in the Website Design Directory. Obviously, I wouldn't want to be listed on the top Hungarian Basket Weaving sites. Find directories of websites that are relevant to your business and check out their listing options. Some are free, some are paid. But even the paid ones are usually pretty reasonable.

4. Use the SEARCH ENGINE's FREE tools!
I am talking about Google Analytics, Webmaster Tools - and most importantly - if you have a brick and mortar - Google Places! (formerly Local Business Center). Sign up for your free account, integrate these free tools (and any others you can figure out!) and do the same thing on Bing.com, and anywhere else you can do it.

Most people don't have time to do all this stuff, and for some people - it doesn't come easy. That is where I have decided to put a package together to support people who own websites and don't have time to track the traffic, write blogs, network and promote. Until then - hope these tips helps the do-it-yourselfers.

Holidays BLEW by!

I'm sorry I haven't written much lately - the holidays just BLEW BY - and now we're into 2009 already! I've got some great stuff to share about SEO, so I'll have to organize it and post it ASAP for you. Also - some great new sites are now live!

So, look for me to post again this week on some things about natural & organic SEO and the new sites Kari & Co. was working on over the 2008 Holiday!

So far for 2009, I've got a few Volusion customers that I'm working with to help them get their site BE-AU-TI-FIED!

Hope everyone is great, had a GREAT holiday season - and we're looking forward to a successful 2009!!