Scrapebox crashing while using fast poster? Here's why and how to fix it!

GoldenGlovez

Senior Member
Joined
Mar 23, 2011
Messages
890
Reaction score
2,201
Does your Scrapebox crash while using fast poster? This can help!

I use Scrapebox A LOT, each day I harvest and post millions of URL's. Over the last few month's I've noticed an alarming increase in random crashes while using the fast poster to post large lists. Every time, I have to reload Scrapebox, re-import my harvest and clean from the crash dump file. After getting tired of repeating this process 2-3 times a list I started looking into the cause. What I found is that the crash occurs each time fast poster comes across a long URL string (1000+ characters in length).

So I began to look for methods on removing these long strings from my harvest before posting. After much searching, slow results and failed attempts, I was about to give up. Then a friend (and moderator) here from BHW Apekillape sent me just the bit of info I needed to get the job done fast and reliably.

If this problem affects you, here is the breakdown on what you need to do:

1. First we need to install PERL on our machine. A free and lightweight solution for Windows is 'Strawberry Perl'. You can find the download links on the main website here:

http://www.strawberryperl.com/

2. Once you have that installed, next we need to create the script for PERL to run against our lists. Open and create a new text document, inside you will copy and paste this:

Code:
my $lRow;
while ( $lRow = <STDIN> ) {
chomp ($lRow);
$lRow = substr($lRow, 0, 500);
print "$lRow\n";
}
3. Save this new file as 'urltrim.pl'. (If using Notepad, make sure to select All Files before saving; Notepad++ is recommended)

4. Now put the new 'urltrim.pl' file in the same folder/directory as the text files you would like to clean.

5. Then you will need to open an MS-DOS command prompt (Either Start > RUN, or press WindowsKey + R and type CMD and then Enter).

6. Finally, CD to the directory containing your Harvested URL's and the PERL script and run this command inside DOS:

Code:
type harvest_to_clean.txt | perl urltrim.pl > new_clean_harvest.txt
Within seconds PERL will output a new and cleaned URL list named 'new_clean_harvest.txt'

You have now successfully trimmed any URL's over 500 characters in length to a new clean file that should run through fast poster without crashes.

Hope this helps!

Regards,
GoldenGlovez
 
Awesome post!

Have you taken a look at some of those long URLs? If you have a few example I would like to take a look at them - sometimes there is a pattern - like they all contain a ? or something similar.
 
Awesome post!

Have you taken a look at some of those long URLs? If you have a few example I would like to take a look at them - sometimes there is a pattern - like they all contain a ? or something similar.

Here are a few examples:

Code:
http://ashinsirinda.com/index.php/component/k2/item/697-%E2%80%9C%E1%80%94%E1%80%AC%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7-%E1%80%95%E1%82%92%E1%80%AC%E1%80%94%E1%80%B9%E1%80%B8-%E1%80%9C%E1%80%BD%E1%80%B4%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7%E1%80%86%E1%80%BC%E1%80%99%E1%80%B9%E1%80%B8%E2%80%9D-%E1%80%9E%E1%80%AF%E1%80%AD%E1%82%94%E1%80%99%E1%80%9F%E1%80%AF%E1%80%90%E1%80%B9-%E2%80%9C%E1%80%86%E1%80%AE%E1%80%99%E1%80%BB%E1%80%96%E1%80%85%E1%80%B9%E1%80%81%E1%80%84%E1%80%B9%E1%80%80-%E1%80%A1%E1%80%AF%E1%80%94%E1%80%B9%E1%80%B8%E1%80%9E%E1%80%AE%E1%80%B8%E1%80%98%E1%80%9D%E2%80%9D/697-%E2%80%9C%E1%80%94%E1%80%AC%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7-%E1%80%95%E1%82%92%E1%80%AC%E1%80%94%E1%80%B9%E1%80%B8-%E1%80%9C%E1%80%BD%E1%80%B4%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7%E1%80%86%E1%80%BC%E1%80%99%E1%80%B9%E1%80%B8%E2%80%9D-%E1%80%9E%E1%80%AF%
Code:
http://shinglestreatment369.blinkweb.com/1/2011/08/possibly-the-first-time-an-individual-observed-the-blistering-associated-with-shingles-you-thought-you-got-bit-by-way-of-a-spider-or-had-some-sort-of-contact-dermatitis-discomfort-may-be-the-defining-characteristic-of-shingles-signs-versus-any-other-type-of-break-outs-typically-you-will-begin-having-severe-headaches-for-any-7-days-or-experience-any-phantom-discomfort-close-to-your-own-belly-or-face-often-this-kind-of-pain-is-wrong-with-regard-to-kidney-belly-or-heart-problems-initially-once-you-discover-the-actual-distinguishing-break-outs-come-in-any-heavy-band-around-one-side-of-the-face-upper-body-or-even-stomach-call-a-doctor-right-away-to-control-the-herpes-virus-just-before-it-gets-out-of-hand-do-not-hesitate-obtaining-shingles-remedy-when-a-prognosis-has-been-manufactured-you-will-begin-the-treatment-by-way-of-antiviral-medications-which-go-after-the-real-cause-of-your-problem-doctors-say-that-its-always-best-to-begin-your-own-treatment-course-within-2
Code:
http://peppersprays694.blinkweb.com/1/2011/09/as-a-matter-of-fact-pepper-became-a-popular-method-for-torturing-people-criminals-and-slaves-while-its-recommended-that-anyone-serious-about-self-defense-whether-its-utilizing-pepperspray-or-learning-to-disable-an-assailant-physically-consider-a-professional-self-defense-course-here-are-a-few-tips-and-facts-in-the-proper-use-of-defense-spray-pepper-spray-is-the-real-deal-and-yet-a-non-lethal-solution-to-protecting-yourself-the-fourth-option-is-the-standard-spray-that-has-been-made-from-an-extract-of-chili-peppers-used-for-making-oleoresin-capsicum-pepper-spray-training-useful-self-defense-tips-for-carrying-pepper-spray-pepper-spray-or-oleoresin-capsicum-oc-can-be-very-effective-in-helping-someone-out-of-a-potentially-dangerous-situation-however-possessing-a-canister-of-oc-and-using-it-improperly-could-cause-more-problems-than-solutions-and-one-more-thing-always-remember-to-check-whether-or-not-defensive-sprays-are-legal-in-your-area-before-you-buy-the-fbi-was-one-
Now that I think about it, you could probably also filter these by using regular expression in Notepad++ to search for strings containing an excessive amount of % and - characters. However, the PERL method is quick and painless with only one command (and guarantees to find all long strings).
 
It might be worth to integrate as an option in ScrapeBox, to ignore urls longer than xxx character.
 
It might be worth to integrate as an option in ScrapeBox, to ignore urls longer than xxx character.

This would certainly be a welcomed feature. It's not very often I come across such long URL's, but it does become a pain to constantly restart SB and clean my lists each time this happens.
 
It will be added in the next update.

hell ya!!!i run into this problem with large lists also.. i have a super fast core i7 with 12 gb ram and it crashes often and i never figure out when and why it crashed or how far it made it through the list because the crashdump log files are always so big i have to delete them before i can reopen scrapebox again which gets annoying.. hope this is fixed asap!! :)

Also can you please add a better proxy leecher to the proxy harvester?? i know tools like forum proxy leecher and proxy miltiply and proxy goblin do this but if scrapebox could do what they do it would be golden!!!
here is a sample list of what forum proxy leecher uses:
http://www.forumproxyleecher.com/liststat.php
i know there are several other good lists floating around!! :) thanks!!
 
Last edited:
Also can you please add a better proxy leecher to the proxy harvester?? i know tools like forum proxy leecher and proxy miltiply and proxy goblin do this but if scrapebox could do what they do it would be golden!!!
here is a sample list of what forum proxy leecher uses:
http://www.forumproxyleecher.com/liststat.php
i know there are several other good lists floating around!! :) thanks!!

You can already add your own custom proxy list URL's and scrape proxies from them, the only thing it won't work with is sites which require a login to the view the proxies, or sites which obfuscate the proxies with javascript or printing them as an image.

So most of the lists in your link will work fine.
 
It will be added in the next update.

v1.15.28 New: Ability to remove harvested URL's longer than XXX characters in the harvesters Remove/Filter button

Thanks again for this new update! Ran a test using it last night and posted 2 million blogs over the last 24 hours consecutively without a single crash.
 
v1.15.28 New: Ability to remove harvested URL's longer than XXX characters in the harvesters Remove/Filter button

Thanks again for this new update! Ran a test using it last night and posted 2 million blogs over the last 24 hours consecutively without a single crash.

Yes this feature is nice. Thanks for your original post too. :)
 
v1.15.28 New: Ability to remove harvested URL's longer than XXX characters in the harvesters Remove/Filter button

Thanks again for this new update! Ran a test using it last night and posted 2 million blogs over the last 24 hours consecutively without a single crash.

Excellent, thanks for troubleshooting the problem and coming up with the idea. :)
 
Excellent, thanks for troubleshooting the problem and coming up with the idea. :)

Hello sweetfunny sorry for the off topic post, but I have someone on the blackhat forum that wants to sell me their license to scrapebox, but can you verify that this persons license is legit?
How do I get a response from you about this as your pm box is full.
please let me know
thanks:)
 
I realize it's not a techy answer - but I load my URL's into Openoffice calc (free) run a column next to the list with the "Len" string command, ten filet out any over 512 characters.

There are, surprisingly, several in every list.
Also - it's normally buried that deep in the URL structure of the site that the link I'm losing has near zero significance - essp in relation to the 5000+ others that are going out.

Scritty
 
Hy.

My SB is still crashing also I remove all url that have more than 300 character?

Any suggestion?
 
Hy.

My SB is still crashing also I remove all url that have more than 300 character?

Any suggestion?

How large is your list? How much system memory? I've seen on the VERY rare occasion crashes even after removing long strings. Unfortunately, I've been unable to pinpoint the cause.
 
How large is your list? How much system memory? I've seen on the VERY rare occasion crashes even after removing long strings. Unfortunately, I've been unable to pinpoint the cause.

100k list and the VPS have 3g memory
 
100k list and the VPS have 3g memory

Shouldn't be any problems running a 100k list on a machine with 3GB. I'd recommend splitting the list into 20k parts and run each one (use the export and split list feature). This could help pinpoint which part of the file is causing the issue. Sometimes some link lists just don't want to behave with SB =/
 
I go on the safe side and remove all URLs with over 150 characters. It removes about 1% of the list only and I can run 1 million+ lists for days constantly without crashing on a 1gb vps, haven't tried any larger than that but it's looking good.
 
Back
Top