1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Implications to ignoring robots.txt disallow rule

Discussion in 'General Programming Chat' started by happyhatter, Apr 21, 2009.

  1. happyhatter

    happyhatter Registered Member

    Joined:
    Jan 26, 2009
    Messages:
    61
    Likes Received:
    3
    What would be the implications, if any, if I ignored the disallow rule? I'm using a Visual Basic application to scrape a site once a day for 15 seconds.
     
  2. crashed

    crashed Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 13, 2008
    Messages:
    958
    Likes Received:
    1,198
    Occupation:
    Guru-slayer
    Location:
    Behind the VPN...
    Home Page:
    You could download all the stuff that the webmaster doesn't want in the search engines :D

    They may block your IP though :)
     
  3. drey2k

    drey2k Power Member

    Joined:
    Jan 4, 2009
    Messages:
    551
    Likes Received:
    169
    Occupation:
    Finance guy
    Location:
    USSR 1943
    Sorry to bother...

    But what is scraping a site?
     
  4. crashed

    crashed Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 13, 2008
    Messages:
    958
    Likes Received:
    1,198
    Occupation:
    Guru-slayer
    Location:
    Behind the VPN...
    Home Page:
    It means taking the content from the website you are scraping.