1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What is interesting with robots.txt

Discussion in 'Blogging' started by zeeipad, Jul 22, 2011.

  1. zeeipad

    zeeipad Registered Member

    Joined:
    May 27, 2011
    Messages:
    71
    Likes Received:
    8
    I was looking at my logs and I see a company looked at the robots.txt page of my wordpress site a few times over a few weeks. What would they be looking for there? It is a telephony services Company. Apparently they also ran a whois on the site too.

    Should I just block the Ip or something?
     
    Last edited: Jul 22, 2011
  2. m4dm4n

    m4dm4n Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 15, 2010
    Messages:
    221
    Likes Received:
    92
    Occupation:
    /dev/full
    Location:
    /dev/urandom
    did you check the user-agent?
    usually robots.txt are checked for 2 reasons:
    1. a bot crawled your site or tried to get some info on your site
    2. a malicious user tried to see urls that you've blocked bots from.
    Something like : Disallow: /admin
     
  3. zeeipad

    zeeipad Registered Member

    Joined:
    May 27, 2011
    Messages:
    71
    Likes Received:
    8
    Oh...well then I am not worried about it. It's all whitehat anyways... except for spun articles.

    I am really not big enough to need telephony services...lol.