Implications to ignoring robots.txt disallow rule

Discussion in 'General Programming Chat' started by happyhatter, Apr 21, 2009.

  1. happyhatter

    happyhatter Registered Member

    Joined:
    Jan 26, 2009
    Messages:
    61
    Likes Received:
    3
    What would be the implications, if any, if I ignored the disallow rule? I'm using a Visual Basic application to scrape a site once a day for 15 seconds.
     
  2. crashed

    crashed Senior Member

    Joined:
    Aug 13, 2008
    Messages:
    958
    Likes Received:
    1,201
    Occupation:
    Guru-slayer
    Location:
    Behind the VPN...
    You could download all the stuff that the webmaster doesn't want in the search engines :D

    They may block your IP though :)
     
  3. drey2k

    drey2k Power Member

    Joined:
    Jan 4, 2009
    Messages:
    567
    Likes Received:
    174
    Occupation:
    IM Master
    Location:
    USSR 1943
    Sorry to bother...

    But what is scraping a site?
     
  4. crashed

    crashed Senior Member

    Joined:
    Aug 13, 2008
    Messages:
    958
    Likes Received:
    1,201
    Occupation:
    Guru-slayer
    Location:
    Behind the VPN...
    It means taking the content from the website you are scraping.