What is interesting with robots.txt

Discussion in 'Blogging' started by zeeipad, Jul 22, 2011.

  1. zeeipad

    zeeipad Registered Member

    Joined:
    May 27, 2011
    Messages:
    71
    Likes Received:
    8
    I was looking at my logs and I see a company looked at the robots.txt page of my wordpress site a few times over a few weeks. What would they be looking for there? It is a telephony services Company. Apparently they also ran a whois on the site too.

    Should I just block the Ip or something?
     
    Last edited: Jul 22, 2011
  2. m4dm4n

    m4dm4n Regular Member

    Joined:
    Sep 15, 2010
    Messages:
    223
    Likes Received:
    92
    Occupation:
    /dev/full
    Location:
    /dev/urandom
    did you check the user-agent?
    usually robots.txt are checked for 2 reasons:
    1. a bot crawled your site or tried to get some info on your site
    2. a malicious user tried to see urls that you've blocked bots from.
    Something like : Disallow: /admin
     
  3. zeeipad

    zeeipad Registered Member

    Joined:
    May 27, 2011
    Messages:
    71
    Likes Received:
    8
    Oh...well then I am not worried about it. It's all whitehat anyways... except for spun articles.

    I am really not big enough to need telephony services...lol.