ScrapeBox Optimal Connections Settings

Hi Guys,
What is your maximum/optimal connection settings for ScrapeBox?

Mine is:

Internet connection: 9mb/s
Url Harvester: 10 - Timeout: 30 sec
Fast Poster: 50 - Timeout: 35 sec
Link Checker: 35 - Timeout: 60 sec
Proxy Harvester: 20 - Timeout: 15 sec

Internet connection: 30-60 mb/s
Url Harvester: 30 - Timeout: 50 sec
Fast Poster: 150 - Timeout: 90 sec
Link Checker: 150 - Timeout: 90 sec
Proxy Harvester: 50 - Timeout: 30 sec[/QUOTE]

That is is the settings I used now.

Was wondering this and been trying optimize for last 8 hours lol.

100mbs

Harvester 350 per Engine works fine

But the link checker hangs no matter how less threads i put in . Then it goes really fast for little time and slows down again. Without proxies. This is killing me as im trying to think how to get it working better. Win server 2003.

Fast Poster 100 threads - 90s timeout

antsaoo said:

Was wondering this and been trying optimize for last 8 hours lol.

100mbs

Harvester 350 per Engine works fine

But the link checker hangs no matter how less threads i put in . Then it goes really fast for little time and slows down again. Without proxies. This is killing me as im trying to think how to get it working better. Win server 2003.

Fast Poster 100 threads - 90s timeout

Click to expand...

Try using free proxies.

Umm i don't see any point using proxies if checking links. :/ Am i missing something ?

I am wondering the same as well. I just bought 5 private proxies today and pretty much only plan on using fast post, I have a 10mbps connection. What would you guys recommend for connections and timeout?

antsaoo said:

Umm i don't see any point using proxies if checking links. :/ Am i missing something ?

Click to expand...

Ops I thought it needs proxy, Try asking support about it.

Dedi Servers Internet connection: 1Gbit/s

Url Harvester: depends on engine and proxies.

Yahoo -10 connections with no proxies
Google and AOL - Use private proxies at 20% of my proxies. So if I use 100 private proxies I set connections to 20. Can harvest for quite a while like this, and private proxies are so fast, I find my urls/s come out more then more public proxies, plus IMHO you don't have to mess around with failed ones, as 99.9% of them succeed first time with private proxies

Fast Poster: 500 - 60 sec timeout

Link Checker: 500 - 90 sec timeout on small runs, lower connections seem to be faster on my server when the list goes over 400K

Proxy Harvester: 100 - 15 sec timeout.


~~~~~~

When I used 100Mbit dedi servers

Link checker 100 - 90 sec timeout
Fast Poster - 200 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.


~~~~~~

Home internet - 18Mbit

Link checker 100 - 90 sec timeout
Fast Poster - 100 - 50 sec timeout
Proxy Harvester - 100 - 15 sec timeout
Url Harvester - 75 per engine, I only ran 1 engine per scrapebox instance.



I didn't see much difference in many settings as far as optimal from my home 18Mbit connection to my server 100Mbit except that the server just did everything faster. When I jacked connections success seemed to fall.

When I upgraded to my 1 Gigabit I am able to max everything and it sails smooth as can be.




What I do is start low, and turn it up until speed and results tend to fall off rather then increase and thats about where you need to be. Of course its subjective, cause the larger the file you work with, the more the processor has to compensate with, even if there is enough RAM, and of course if there is not enough RAM then you have issues.

antsaoo said:

Was wondering this and been trying optimize for last 8 hours lol.

100mbs

Harvester 350 per Engine works fine

But the link checker hangs no matter how less threads i put in . Then it goes really fast for little time and slows down again. Without proxies. This is killing me as im trying to think how to get it working better. Win server 2003.

Fast Poster 100 threads - 90s timeout

Click to expand...

This happens to me even on my 1Gbit server when checking links at 500 connections. So long as your not getting lots of timeout errors then its fine. Basically what happens is you have a lot going on there. Windows uses a message Que to tell the scrapebox window to repaint (refresh what your viewing) and when you have tons happening your pc may not have enough resources to do everything Windows and scrapebox and whatever else you have running, are asking it to do. So it can't process those messages fast enough and the screen "hangs" or ghosts or even says not responding. Even my 6 core servers with 8GB of ram on 1Gbit lines do this and sometimes even for 10 mins at a go. I have learned don't touch it, the pc is doing what it needs, let it be. (or buy a faster pc for instant gratification lol - I spend too much as it is).

If you are getting lots of timeout errors, then your inet connection pipe isn't big enough to handle it and it times out before it has a chance to load the page. When you have 100 pages that all time out at a time it will seem "jerky" in motion as it skips. In which case just turn your connections down/timeouts up.

So basically not a lot of timeout errors, leave it be. Lots of timeout errors, then turn your connections down and timeouts up.

Oh, and no you don't need proxies when link checking, although its good to use the randomize setting so you aren't link checking 300 pages on 1 domain at a time.

Ye but the problem kinda is its not taking even 10% of processor or memory. Guess its the connection opening / closing.

Also to mention i noticed funny glitch while scraping. When you start harvester. Open up view changelog for example while scraping and i got to 2k/s with google 2.5k urls/s with yahoo. 2s public proxies.Dunno if its only some error on my dedi or it fucks up something. Though i tried on vps and it maxed to 10mbs at all times too :O ( mofos did not tell it was limited at their site =/)

I have a Toshiba laptop, Intel Core 2 Duo, CPU T7500, 2.20 GHz,RAM 2 MB , wireless connection 100Mbit. For fast poster I set 50 connections, 90 timeout and I get around 100 links processed per minute which I find very low as my lists are quite big.

What can be done to increase the speed of links processing? A better home computer, private proxies, etc?

Thanks!

Internet connection: 10mb/s, full duplex
Url Harvester: 30 - Timeout: 10 sec
Fast Poster: 150 - Timeout: 40 sec (5-7% more success @ 30 connections, 60s timeout, but 4x the time)
Link Checker: 50 - Timeout: 40 sec
Proxy Harvester: 50 - Timeout: 3 sec

Comp
AMD x6 1055T @ 4ghz
8gb ram
Win7 64 Ultimate

c0rup7 said:

I use 500 threads on my 40MBPS (house internet)

It'd be nice if SweetFunny added support for more threads one day

Click to expand...

You can edit it yourself at YOUR-SCRAPEBOX-DIR/scrapebox.ini

smartgirl said:

I have a Toshiba laptop, Intel Core 2 Duo, CPU T7500, 2.20 GHz,RAM 2 MB , wireless connection 100Mbit. For fast poster I set 50 connections, 90 timeout and I get around 100 links processed per minute which I find very low as my lists are quite big.

What can be done to increase the speed of links processing? A better home computer, private proxies, etc?

Thanks!

Click to expand...

I have a pretty similar setup with 10 private proxies.

I can get ~200 per minute. But the then failure rate is 50%...

Any suggestions?

When you say 100mb's do u really mean 1mb = 1000kbps?. I am on the fastest speed adsl2 through internode and when i did a speed test i am only getting 3872 kbps so in other words nearly 4mbs?? I am running on 3ghz system with 2gb ram. No idea what to put my settings at!

This is very helpful info thanks alot

How would I tell if I have the url harvester connections set too high, would it just not work or what?

What do you all use for pagerank?

For Proxy Checker, I use 200 connections with 2x Re-check Failed.
Proxygo said 50 connections is the best choice for the proxy checker, to get real results.

Beny

EDIT : So old thread ! Wtf to bump this ?!