Implications to ignoring robots.txt disallow rule

H

happyhatter

Registered Member

Jan 26, 2009

61

3

• Apr 21, 2009

• #1

What would be the implications, if any, if I ignored the disallow rule? I'm using a Visual Basic application to scrape a site once a day for 15 seconds.

 

C

crashed

Senior Member

Aug 13, 2008

958

1,209

• May 5, 2009

• #2

You could download all the stuff that the webmaster doesn't want in the search engines

They may block your IP though

 

drey2k

Power Member

Jan 4, 2009

567

174

• May 5, 2009

• #3

Sorry to bother...

But what is scraping a site?

 

C

crashed

Senior Member

Aug 13, 2008

958

1,209

• May 5, 2009

• #4

drey2k said:

Sorry to bother...

But what is scraping a site?

Click to expand...

It means taking the content from the website you are scraping.