Scraping, how much is to much.

M

mward

Newbie

Oct 19, 2010

43

3

• Oct 24, 2010

• #1

Simple question, what would you say is a sensible amount of scrapes of a site per hour or min? If you didn't want to get banned or found out.
Posted via Mobile Device

 

C

cjw10

Junior Member

Jan 7, 2010

116

32

• Oct 24, 2010

• #2

Use proxies and it doesn't matter if your banned.

 

B

bhprince

Registered Member

Oct 19, 2010

56

8

• Oct 24, 2010

• #3

you can get either private proxies or just use scapebox to find some public ones.

 

C

CyrusVirus

BANNED

Aug 20, 2009

1,109

693

• Oct 24, 2010

• #4

lol wierd that you posted this in the proxies section, are you asking how much is too much to scrape for private proxies?
or just in general without proxies at all.

if you have just 1 site, then 2 proxies should do it.
in general, throw in a few free proxies and just scrape.

 

M

mward

Newbie

Oct 19, 2010

43

3

• Oct 24, 2010
• Thread Starter Thread Starter

• #5

Sorry my fault, I have private proxies but still feel I should limit the amount of request as if the proxies get banned I will have to buy more
Posted via Mobile Device

 

SMMSEO

Regular Member

Oct 14, 2009

250

47

• Oct 24, 2010

• #6

I think it depends on how old your domain is. If it's a new domain you'll get sandboxed very fast.

 

M

mward

Newbie

Oct 19, 2010

43

3

• Oct 24, 2010
• Thread Starter Thread Starter

• #7

My domain? 10 years
Posted via Mobile Device

 

proxygo

Elite Member

Executive VIP

Jr. VIP

Nov 2, 2008

59,090

26,637

• Oct 25, 2010

• #8

thats why the golden rule is

use public proxies for scraping
private ones for posting
thought most people new that

 

M

mward

Newbie

Oct 19, 2010

43

3

• Oct 25, 2010
• Thread Starter Thread Starter

• #9

public ones are fair to unreliable for that we need, it need to work everytime, no just on an off, we tested it with public ones and it failed, but thanks anyway