Proxies needed to scrape PageRank?

E

el_gring0

Newbie

Jun 22, 2011

3

0

• Sep 5, 2011

• #1

Hello,

I'm finding public proxies terribly slow to scrape PageRank (using scrapebox). I'd like to scrape somewhere over 1000 per minute (I can do this without proxies, though quickly get a timeout).

Is that sort of speed possible with public proxies? What should I do that minimizes cost/effort to get proxies capable of this?

Thanks

 

darshan1994

BANNED

Oct 9, 2009

654

320

• Sep 5, 2011

• #2

Find google allowed proxy. Many providers on this forum that give you few hundred google clear proxy a day. If not go with 10 private proxy and harvest like 20-30k PR before temp ban and repeat.

 

E

el_gring0

Newbie

Jun 22, 2011

3

0

• Sep 5, 2011
• Thread Starter Thread Starter

• #3

Also, how does having more proxies help scrapebox lookup PR faster? AFAI understand, it can only use one proxy per PageRank request, so I don't see why more would neccesarily be better.

 

E

el_gring0

Newbie

Jun 22, 2011

3

0

• Sep 5, 2011
• Thread Starter Thread Starter

• #4

darshan1994 said:

Find google allowed proxy. Many providers on this forum that give you few hundred google clear proxy a day. If not go with 10 private proxy and harvest like 20-30k PR before temp ban and repeat.

Click to expand...

Thanks. Can you recommend a good source for private proxies? I find plenty by Googling but a lot of the sites look pretty sketchy.

 

G

gbomb45

Newbie

Jun 5, 2011

39

2

• Sep 6, 2011

• #5

Type in proxy list in google or somethin

 

Fathom

Power Member

Jul 1, 2011

518

282

• Sep 6, 2011

• #6

if you are adding a huge amount of url's to scrapebox it will be slow, break them down into 10k chunks and the speed should be just fine

 

Autumn

Elite Member

Nov 18, 2010

2,197

3,073

• Sep 6, 2011

• #7

el_gring0 said:

Also, how does having more proxies help scrapebox lookup PR faster? AFAI understand, it can only use one proxy per PageRank request, so I don't see why more would neccesarily be better.

Click to expand...

I'm not a SB user but generally if you're scraping anything using proxies then you can multithread and use one proxy per thread. So if you have 50 proxies you can run 50 threads at a time. If you try to bombard anything you're trying to scrape with 50 threads from the same IP, most services are going to block you pretty quickly.

With my own scraping system, I have about 50 threads running 24/7 and I'll spike it up to 500-1000 if I need to do a big scrape in a hurry. However slow and steady generally wins the race when it comes to serp scraping.

 

NIXMY

Regular Member

Sep 26, 2010

481

322

• Sep 6, 2011

• #8

Fathom said:

if you are adding a huge amount of url's to scrapebox it will be slow, break them down into 10k chunks and the speed should be just fine

Click to expand...

This is the reason I work on linux. It's typical to windows tools when you add more than normal amounts, they either crash, get slow and/or unreliable.

Does any1 have idea how many PR request you can do with one IP before it will be banned temporarily?

 

madoctopus

Supreme Member

Apr 4, 2010

1,388

3,709

• Sep 6, 2011

• #9

I use shared proxies from http://buyproxies.org/ for this and fallback on dedicated proxies from them and then fallback on public proxies and then fall back on direct connection. I fall back a lot as you see

They work well and are cheap but obviously are not free.

 

M

mmBob

Newbie

Feb 16, 2009

31

6

• Sep 8, 2011

• #10

Alternatively you could use the easy api (theeasyapi dot com). They have a PR scraping service.

 

NIXMY

Regular Member

Sep 26, 2010

481

322

• Sep 9, 2011

• #11

Thanks for the idea. Look like I must do some testing on PR scraping to see how fast a single IP gets banned.