1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrapebox crashing while using fast poster? Here's why and how to fix it!

Discussion in 'Black Hat SEO Tools' started by GoldenGlovez, Oct 4, 2011.

  1. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    Does your Scrapebox crash while using fast poster? This can help!

    I use Scrapebox A LOT, each day I harvest and post millions of URL's. Over the last few month's I've noticed an alarming increase in random crashes while using the fast poster to post large lists. Every time, I have to reload Scrapebox, re-import my harvest and clean from the crash dump file. After getting tired of repeating this process 2-3 times a list I started looking into the cause. What I found is that the crash occurs each time fast poster comes across a long URL string (1000+ characters in length).

    So I began to look for methods on removing these long strings from my harvest before posting. After much searching, slow results and failed attempts, I was about to give up. Then a friend (and moderator) here from BHW Apekillape sent me just the bit of info I needed to get the job done fast and reliably.

    If this problem affects you, here is the breakdown on what you need to do:

    1. First we need to install PERL on our machine. A free and lightweight solution for Windows is 'Strawberry Perl'. You can find the download links on the main website here:

    http://www.strawberryperl.com/

    2. Once you have that installed, next we need to create the script for PERL to run against our lists. Open and create a new text document, inside you will copy and paste this:

    Code:
    my $lRow;
    while ( $lRow = <STDIN> ) {
    chomp ($lRow);
    $lRow = substr($lRow, 0, 500);
    print "$lRow\n";
    }
    3. Save this new file as 'urltrim.pl'. (If using Notepad, make sure to select All Files before saving; Notepad++ is recommended)

    4. Now put the new 'urltrim.pl' file in the same folder/directory as the text files you would like to clean.

    5. Then you will need to open an MS-DOS command prompt (Either Start > RUN, or press WindowsKey + R and type CMD and then Enter).

    6. Finally, CD to the directory containing your Harvested URL's and the PERL script and run this command inside DOS:

    Code:
    type harvest_to_clean.txt | perl urltrim.pl > new_clean_harvest.txt
    Within seconds PERL will output a new and cleaned URL list named 'new_clean_harvest.txt'

    You have now successfully trimmed any URL's over 500 characters in length to a new clean file that should run through fast poster without crashes.

    Hope this helps!

    Regards,
    GoldenGlovez
     
    • Thanks Thanks x 8
  2. HelloInsomnia

    HelloInsomnia Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Mar 1, 2009
    Messages:
    1,816
    Likes Received:
    2,912
    Awesome post!

    Have you taken a look at some of those long URLs? If you have a few example I would like to take a look at them - sometimes there is a pattern - like they all contain a ? or something similar.
     
  3. wannabie

    wannabie Elite Member

    Joined:
    Mar 11, 2009
    Messages:
    3,807
    Likes Received:
    2,954
    Occupation:
    Seo and Marketing Suprisingly
    Location:
    Your bedroom window
    Home Page:
    Grea t post
     
  4. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    Here are a few examples:

    Code:
    http://ashinsirinda.com/index.php/component/k2/item/697-%E2%80%9C%E1%80%94%E1%80%AC%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7-%E1%80%95%E1%82%92%E1%80%AC%E1%80%94%E1%80%B9%E1%80%B8-%E1%80%9C%E1%80%BD%E1%80%B4%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7%E1%80%86%E1%80%BC%E1%80%99%E1%80%B9%E1%80%B8%E2%80%9D-%E1%80%9E%E1%80%AF%E1%80%AD%E1%82%94%E1%80%99%E1%80%9F%E1%80%AF%E1%80%90%E1%80%B9-%E2%80%9C%E1%80%86%E1%80%AE%E1%80%99%E1%80%BB%E1%80%96%E1%80%85%E1%80%B9%E1%80%81%E1%80%84%E1%80%B9%E1%80%80-%E1%80%A1%E1%80%AF%E1%80%94%E1%80%B9%E1%80%B8%E1%80%9E%E1%80%AE%E1%80%B8%E1%80%98%E1%80%9D%E2%80%9D/697-%E2%80%9C%E1%80%94%E1%80%AC%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7-%E1%80%95%E1%82%92%E1%80%AC%E1%80%94%E1%80%B9%E1%80%B8-%E1%80%9C%E1%80%BD%E1%80%B4%E1%80%9E%E1%80%B0%E1%80%99%E1%80%9B%E1%80%BD%E1%80%AD%E1%80%90%E1%80%B2%E1%80%B7%E1%80%86%E1%80%BC%E1%80%99%E1%80%B9%E1%80%B8%E2%80%9D-%E1%80%9E%E1%80%AF%
    Code:
    http://shinglestreatment369.blinkweb.com/1/2011/08/possibly-the-first-time-an-individual-observed-the-blistering-associated-with-shingles-you-thought-you-got-bit-by-way-of-a-spider-or-had-some-sort-of-contact-dermatitis-discomfort-may-be-the-defining-characteristic-of-shingles-signs-versus-any-other-type-of-break-outs-typically-you-will-begin-having-severe-headaches-for-any-7-days-or-experience-any-phantom-discomfort-close-to-your-own-belly-or-face-often-this-kind-of-pain-is-wrong-with-regard-to-kidney-belly-or-heart-problems-initially-once-you-discover-the-actual-distinguishing-break-outs-come-in-any-heavy-band-around-one-side-of-the-face-upper-body-or-even-stomach-call-a-doctor-right-away-to-control-the-herpes-virus-just-before-it-gets-out-of-hand-do-not-hesitate-obtaining-shingles-remedy-when-a-prognosis-has-been-manufactured-you-will-begin-the-treatment-by-way-of-antiviral-medications-which-go-after-the-real-cause-of-your-problem-doctors-say-that-its-always-best-to-begin-your-own-treatment-course-within-2
    Code:
    http://peppersprays694.blinkweb.com/1/2011/09/as-a-matter-of-fact-pepper-became-a-popular-method-for-torturing-people-criminals-and-slaves-while-its-recommended-that-anyone-serious-about-self-defense-whether-its-utilizing-pepperspray-or-learning-to-disable-an-assailant-physically-consider-a-professional-self-defense-course-here-are-a-few-tips-and-facts-in-the-proper-use-of-defense-spray-pepper-spray-is-the-real-deal-and-yet-a-non-lethal-solution-to-protecting-yourself-the-fourth-option-is-the-standard-spray-that-has-been-made-from-an-extract-of-chili-peppers-used-for-making-oleoresin-capsicum-pepper-spray-training-useful-self-defense-tips-for-carrying-pepper-spray-pepper-spray-or-oleoresin-capsicum-oc-can-be-very-effective-in-helping-someone-out-of-a-potentially-dangerous-situation-however-possessing-a-canister-of-oc-and-using-it-improperly-could-cause-more-problems-than-solutions-and-one-more-thing-always-remember-to-check-whether-or-not-defensive-sprays-are-legal-in-your-area-before-you-buy-the-fbi-was-one-
    
    Now that I think about it, you could probably also filter these by using regular expression in Notepad++ to search for strings containing an excessive amount of % and - characters. However, the PERL method is quick and painless with only one command (and guarantees to find all long strings).
     
  5. softtouch2009

    softtouch2009 Senior Member

    Joined:
    Dec 2, 2009
    Messages:
    1,001
    Likes Received:
    225
    Occupation:
    Programming
    Location:
    ssdnet.biz
    Home Page:
    It might be worth to integrate as an option in ScrapeBox, to ignore urls longer than xxx character.
     
  6. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    This would certainly be a welcomed feature. It's not very often I come across such long URL's, but it does become a pain to constantly restart SB and clean my lists each time this happens.
     
  7. softtouch2009

    softtouch2009 Senior Member

    Joined:
    Dec 2, 2009
    Messages:
    1,001
    Likes Received:
    225
    Occupation:
    Programming
    Location:
    ssdnet.biz
    Home Page:
    It will be added in the next update.
     
    • Thanks Thanks x 3
  8. frederickpc

    frederickpc Regular Member

    Joined:
    Sep 14, 2010
    Messages:
    284
    Likes Received:
    111
    Location:
    Computer Chair
    Home Page:
    hell ya!!!i run into this problem with large lists also.. i have a super fast core i7 with 12 gb ram and it crashes often and i never figure out when and why it crashed or how far it made it through the list because the crashdump log files are always so big i have to delete them before i can reopen scrapebox again which gets annoying.. hope this is fixed asap!! :)

    Also can you please add a better proxy leecher to the proxy harvester?? i know tools like forum proxy leecher and proxy miltiply and proxy goblin do this but if scrapebox could do what they do it would be golden!!!
    here is a sample list of what forum proxy leecher uses:
    http://www.forumproxyleecher.com/liststat.php
    i know there are several other good lists floating around!! :) thanks!!
     
    Last edited: Oct 6, 2011
  9. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    You can already add your own custom proxy list URL's and scrape proxies from them, the only thing it won't work with is sites which require a login to the view the proxies, or sites which obfuscate the proxies with javascript or printing them as an image.

    So most of the lists in your link will work fine.
     
  10. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    v1.15.28 New: Ability to remove harvested URL's longer than XXX characters in the harvesters Remove/Filter button

    Thanks again for this new update! Ran a test using it last night and posted 2 million blogs over the last 24 hours consecutively without a single crash.
     
  11. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,372
    Likes Received:
    1,799
    Gender:
    Male
    Home Page:
    Yes this feature is nice. Thanks for your original post too. :)
     
    • Thanks Thanks x 1
  12. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    Excellent, thanks for troubleshooting the problem and coming up with the idea. :)
     
  13. wallstreeter43

    wallstreeter43 Registered Member

    Joined:
    Dec 22, 2010
    Messages:
    95
    Likes Received:
    7
    Hello sweetfunny sorry for the off topic post, but I have someone on the blackhat forum that wants to sell me their license to scrapebox, but can you verify that this persons license is legit?
    How do I get a response from you about this as your pm box is full.
    please let me know
    thanks:)
     
  14. Scritty

    Scritty Elite Member Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,807
    Likes Received:
    4,496
    Occupation:
    Affiliate Marketer
    Location:
    UK
    Home Page:
    I realize it's not a techy answer - but I load my URL's into Openoffice calc (free) run a column next to the list with the "Len" string command, ten filet out any over 512 characters.

    There are, surprisingly, several in every list.
    Also - it's normally buried that deep in the URL structure of the site that the link I'm losing has near zero significance - essp in relation to the 5000+ others that are going out.

    Scritty
     
  15. ija61

    ija61 Senior Member

    Joined:
    Mar 2, 2011
    Messages:
    960
    Likes Received:
    634
    Gender:
    Male
    Occupation:
    The first SEO economist:)
    Location:
    Romania
    Home Page:
    Hy.

    My SB is still crashing also I remove all url that have more than 300 character?

    Any suggestion?
     
  16. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    How large is your list? How much system memory? I've seen on the VERY rare occasion crashes even after removing long strings. Unfortunately, I've been unable to pinpoint the cause.
     
  17. ija61

    ija61 Senior Member

    Joined:
    Mar 2, 2011
    Messages:
    960
    Likes Received:
    634
    Gender:
    Male
    Occupation:
    The first SEO economist:)
    Location:
    Romania
    Home Page:
    100k list and the VPS have 3g memory
     
  18. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    Shouldn't be any problems running a 100k list on a machine with 3GB. I'd recommend splitting the list into 20k parts and run each one (use the export and split list feature). This could help pinpoint which part of the file is causing the issue. Sometimes some link lists just don't want to behave with SB =/
     
  19. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    I go on the safe side and remove all URLs with over 150 characters. It removes about 1% of the list only and I can run 1 million+ lists for days constantly without crashing on a 1gb vps, haven't tried any larger than that but it's looking good.