ScrapeBox Optimal Connections Settings
- Thread starter whitez
- Start date
antsaoo
Supreme Member
- Oct 1, 2008
- 1,288
- 650
Was wondering this and been trying optimize for last 8 hours lol.
100mbs
Harvester 350 per Engine works fine
But the link checker hangs no matter how less threads i put in . Then it goes really fast for little time and slows down again. Without proxies. This is killing me as im trying to think how to get it working better. Win server 2003.
Fast Poster 100 threads - 90s timeout
100mbs
Harvester 350 per Engine works fine
But the link checker hangs no matter how less threads i put in . Then it goes really fast for little time and slows down again. Without proxies. This is killing me as im trying to think how to get it working better. Win server 2003.
Fast Poster 100 threads - 90s timeout
jheyslow
BANNED
- May 11, 2010
- 891
- 718
jheyslow
BANNED
- May 11, 2010
- 891
- 718
Umm i don't see any point using proxies if checking links. :/ Am i missing something ?
Ops I thought it needs proxy, Try asking support about it.
- Jan 25, 2009
- 6,378
- 3,774
Dedi Servers Internet connection: 1Gbit/s
Url Harvester: depends on engine and proxies.
Yahoo -10 connections with no proxies
Google and AOL - Use private proxies at 20% of my proxies. So if I use 100 private proxies I set connections to 20. Can harvest for quite a while like this, and private proxies are so fast, I find my urls/s come out more then more public proxies, plus IMHO you don't have to mess around with failed ones, as 99.9% of them succeed first time with private proxies
Fast Poster: 500 - 60 sec timeout
Link Checker: 500 - 90 sec timeout on small runs, lower connections seem to be faster on my server when the list goes over 400K
Proxy Harvester: 100 - 15 sec timeout.
~~~~~~
When I used 100Mbit dedi servers
Link checker 100 - 90 sec timeout
Fast Poster - 200 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.
~~~~~~
Home internet - 18Mbit
Link checker 100 - 90 sec timeout
Fast Poster - 100 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.
I didn't see much difference in many settings as far as optimal from my home 18Mbit connection to my server 100Mbit except that the server just did everything faster. When I jacked connections success seemed to fall.
When I upgraded to my 1 Gigabit I am able to max everything and it sails smooth as can be.
What I do is start low, and turn it up until speed and results tend to fall off rather then increase and thats about where you need to be. Of course its subjective, cause the larger the file you work with, the more the processor has to compensate with, even if there is enough RAM, and of course if there is not enough RAM then you have issues.
This happens to me even on my 1Gbit server when checking links at 500 connections. So long as your not getting lots of timeout errors then its fine. Basically what happens is you have a lot going on there. Windows uses a message Que to tell the scrapebox window to repaint (refresh what your viewing) and when you have tons happening your pc may not have enough resources to do everything Windows and scrapebox and whatever else you have running, are asking it to do. So it can't process those messages fast enough and the screen "hangs" or ghosts or even says not responding. Even my 6 core servers with 8GB of ram on 1Gbit lines do this and sometimes even for 10 mins at a go. I have learned don't touch it, the pc is doing what it needs, let it be. (or buy a faster pc for instant gratification lol - I spend too much as it is).
If you are getting lots of timeout errors, then your inet connection pipe isn't big enough to handle it and it times out before it has a chance to load the page. When you have 100 pages that all time out at a time it will seem "jerky" in motion as it skips. In which case just turn your connections down/timeouts up.
So basically not a lot of timeout errors, leave it be. Lots of timeout errors, then turn your connections down and timeouts up.
Oh, and no you don't need proxies when link checking, although its good to use the randomize setting so you aren't link checking 300 pages on 1 domain at a time.
Url Harvester: depends on engine and proxies.
Yahoo -10 connections with no proxies
Google and AOL - Use private proxies at 20% of my proxies. So if I use 100 private proxies I set connections to 20. Can harvest for quite a while like this, and private proxies are so fast, I find my urls/s come out more then more public proxies, plus IMHO you don't have to mess around with failed ones, as 99.9% of them succeed first time with private proxies
Fast Poster: 500 - 60 sec timeout
Link Checker: 500 - 90 sec timeout on small runs, lower connections seem to be faster on my server when the list goes over 400K
Proxy Harvester: 100 - 15 sec timeout.
~~~~~~
When I used 100Mbit dedi servers
Link checker 100 - 90 sec timeout
Fast Poster - 200 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.
~~~~~~
Home internet - 18Mbit
Link checker 100 - 90 sec timeout
Fast Poster - 100 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.
I didn't see much difference in many settings as far as optimal from my home 18Mbit connection to my server 100Mbit except that the server just did everything faster. When I jacked connections success seemed to fall.
When I upgraded to my 1 Gigabit I am able to max everything and it sails smooth as can be.
What I do is start low, and turn it up until speed and results tend to fall off rather then increase and thats about where you need to be. Of course its subjective, cause the larger the file you work with, the more the processor has to compensate with, even if there is enough RAM, and of course if there is not enough RAM then you have issues.
This happens to me even on my 1Gbit server when checking links at 500 connections. So long as your not getting lots of timeout errors then its fine. Basically what happens is you have a lot going on there. Windows uses a message Que to tell the scrapebox window to repaint (refresh what your viewing) and when you have tons happening your pc may not have enough resources to do everything Windows and scrapebox and whatever else you have running, are asking it to do. So it can't process those messages fast enough and the screen "hangs" or ghosts or even says not responding. Even my 6 core servers with 8GB of ram on 1Gbit lines do this and sometimes even for 10 mins at a go. I have learned don't touch it, the pc is doing what it needs, let it be. (or buy a faster pc for instant gratification lol - I spend too much as it is).
If you are getting lots of timeout errors, then your inet connection pipe isn't big enough to handle it and it times out before it has a chance to load the page. When you have 100 pages that all time out at a time it will seem "jerky" in motion as it skips. In which case just turn your connections down/timeouts up.
So basically not a lot of timeout errors, leave it be. Lots of timeout errors, then turn your connections down and timeouts up.
Oh, and no you don't need proxies when link checking, although its good to use the randomize setting so you aren't link checking 300 pages on 1 domain at a time.
Last edited:
antsaoo
Supreme Member
- Oct 1, 2008
- 1,288
- 650
Ye but the problem kinda is its not taking even 10% of processor or memory. Guess its the connection opening / closing.
Also to mention i noticed funny glitch while scraping. When you start harvester. Open up view changelog for example while scraping and i got to 2k/s with google 2.5k urls/s with yahoo. 2s public proxies.Dunno if its only some error on my dedi or it fucks up something. Though i tried on vps and it maxed to 10mbs at all times too :O ( mofos did not tell it was limited at their site =/)
Also to mention i noticed funny glitch while scraping. When you start harvester. Open up view changelog for example while scraping and i got to 2k/s with google 2.5k urls/s with yahoo. 2s public proxies.Dunno if its only some error on my dedi or it fucks up something. Though i tried on vps and it maxed to 10mbs at all times too :O ( mofos did not tell it was limited at their site =/)
I have a Toshiba laptop, Intel Core 2 Duo, CPU T7500, 2.20 GHz,RAM 2 MB , wireless connection 100Mbit. For fast poster I set 50 connections, 90 timeout and I get around 100 links processed per minute which I find very low as my lists are quite big.
What can be done to increase the speed of links processing? A better home computer, private proxies, etc?
Thanks!
What can be done to increase the speed of links processing? A better home computer, private proxies, etc?
Thanks!
Internet connection: 10mb/s, full duplex
Url Harvester: 30 - Timeout: 10 sec
Fast Poster: 150 - Timeout: 40 sec (5-7% more success @ 30 connections, 60s timeout, but 4x the time)
Link Checker: 50 - Timeout: 40 sec
Proxy Harvester: 50 - Timeout: 3 sec
Comp
AMD x6 1055T @ 4ghz
8gb ram
Win7 64 Ultimate
Url Harvester: 30 - Timeout: 10 sec
Fast Poster: 150 - Timeout: 40 sec (5-7% more success @ 30 connections, 60s timeout, but 4x the time)
Link Checker: 50 - Timeout: 40 sec
Proxy Harvester: 50 - Timeout: 3 sec
Comp
AMD x6 1055T @ 4ghz
8gb ram
Win7 64 Ultimate
I use 500 threads on my 40MBPS (house internet)
It'd be nice if SweetFunny added support for more threads one day
You can edit it yourself at YOUR-SCRAPEBOX-DIR/scrapebox.ini
ningning
Supreme Member
- Mar 22, 2010
- 1,262
- 309
I have a pretty similar setup with 10 private proxies.
I can get ~200 per minute. But the then failure rate is 50%...
Any suggestions?
webwowster
Newbie
- May 17, 2009
- 6
- 0
When you say 100mb's do u really mean 1mb = 1000kbps?. I am on the fastest speed adsl2 through internode and when i did a speed test i am only getting 3872 kbps so in other words nearly 4mbs?? I am running on 3ghz system with 2gb ram. No idea what to put my settings at!
I am running on 3ghz system with 2gb ram. No idea what to put my settings at!
jharr53301
Compromised Account
- Apr 10, 2011
- 551
- 57
This is very helpful info thanks alot
bstsystems
Regular Member
- May 17, 2011
- 464
- 82
How would I tell if I have the url harvester connections set too high, would it just not work or what?
kokoloko75
Elite Member
- Jan 1, 2011
- 1,627
- 1,950
For Proxy Checker, I use 200 connections with 2x Re-check Failed.
Proxygo said 50 connections is the best choice for the proxy checker, to get real results.
Beny
EDIT : So old thread ! Wtf to bump this ?!
Proxygo said 50 connections is the best choice for the proxy checker, to get real results.
Beny
EDIT : So old thread ! Wtf to bump this ?!
Last edited: