Did Scrapebox URL harvester get a lot slower all of the sudden?

deviatus

Power Member
Joined
May 25, 2007
Messages
517
Reaction score
388
I used to scrape at a speed of 1000 urls per second, it was awesome. Now it's down to 150 per second no matter which proxy system I use, and it's collecting a smaller number of urls per keyword also.

Blog commenting has remained the same good speed.
 
Honestly, do you really need 1000 urls per second?

But anyway, are all your threads, timeout, proxy settings the same?
 
Honestly, do you really need 1000 urls per second?

Saves time, and time adds up. :D

But anyway, are all your threads, timeout, proxy settings the same?

All different configurations tried, and I'll get roughly the same speed with G, Y, and B. I really can't pin it on any other variable except something with the harvester itself.
 
Hmm, maybe the search engines did some sort of update that's throwing off Scrapebox?

Or a Scrapebox update itself could have done this...

Eh, who knows?
 
Ironically, mine has improved. Earlier, I had a 512k connection, and did an occasional 8 URLs/s.

Now, with a 2mbit connection, I get ~50 per sec.

My connections are set to 40 for each SE, for more than that, SB throttles.

Maybe, you should try by scraping each SE individually.

TheMatrix
 
Ironically, mine has improved. Earlier, I had a 512k connection, and did an occasional 8 URLs/s.

Now, with a 2mbit connection, I get ~50 per sec.

My connections are set to 40 for each SE, for more than that, SB throttles.

Maybe, you should try by scraping each SE individually.

TheMatrix

40?!

I have 100mbit connection and 15 connections gives me the best results.
 
40?!

I have 100mbit connection and 15 connections gives me the best results.

Well, I actually meant that I'm on a 1 gbps line shared amongst several users.

attachment.php


So, sometimes when I'm lucky, I get even upto 70 per sec.

TheMatrix
 

Attachments

  • Screen shot 2010-08-21 at 4.53.14 PM.png
    Screen shot 2010-08-21 at 4.53.14 PM.png
    71.1 KB · Views: 69
Back
Top