1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Why Would Someone Keep Looking at my Robots.txt File?

Discussion in 'White Hat SEO' started by TribeGal, Jan 18, 2012.

  1. TribeGal

    TribeGal Newbie

    Joined:
    Nov 21, 2011
    Messages:
    8
    Likes Received:
    0
    I apologize in advance if this is posted in the wrong section. Why would someone keep doing this? I can see in StatPress that is a user. The only thing listed in this file is the * under allow and a link to my sitemap. What could the person be looking for? I just ask because it's kind of creepy.

    Also, anything I should do to this file to add more security to my site?
     
  2. PhillyLuke

    PhillyLuke Junior Member

    Joined:
    May 18, 2011
    Messages:
    101
    Likes Received:
    37
    Occupation:
    Fulltime SEO/IM
    Location:
    Philadelphia, PA
    I might be a crawler/spider of some sort going through your site

    If it is a person I recommend making a scary "I'm watching you" face out of ASCII characters within your Robots.txt file and beating them at their own game.
     
  3. hotwicked

    hotwicked Regular Member

    Joined:
    Jul 28, 2008
    Messages:
    238
    Likes Received:
    51
    I would say a crawler of some sort. Not much they can do with it.
     
  4. SpeakToTman

    SpeakToTman Regular Member

    Joined:
    May 31, 2009
    Messages:
    249
    Likes Received:
    158
    Location:
    BHW
    Home Page:
    Either a crawler, or someone looking for a "hidden" directory on your site where you may be storing digital products that you sell...
     
  5. XIIII

    XIIII Registered Member

    Joined:
    Jan 19, 2010
    Messages:
    63
    Likes Received:
    8
    The simplest way to identify the website platform - robots txt...
    If your site is making decent money, you could consider to do get a better hosting with good support team. You will not to do it all by yourself. Believe me, handling dedicated servers is hell's work - you need to know DNS, IP, Firewal, Antivirus, server performance, compression, etc etc etc.

    My advice - focus on making money until you can hire people to monitor the site for you,
     
  6. TribeGal

    TribeGal Newbie

    Joined:
    Nov 21, 2011
    Messages:
    8
    Likes Received:
    0
    Thanks guys for your replies, you all are probably right, maybe I'm just paranoid at the moment. Right before I posted this a I was reading about at semi-popular blogger who got hacked! Freaked me out!

    Haha, that's funny! I'd be silly enough to do something like that.

    Thanks for the tips, and yes this is my ultimate goal once the money starts rolling in!

    Thanks again guys!