1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Over A Million WP Blogs Scraped, And Lost!! Grrr Activate Crashdump Logger If You Havent!

Discussion in 'Black Hat SEO' started by tbrad, Oct 5, 2010.

  1. tbrad

    tbrad Registered Member

    Joined:
    Aug 11, 2010
    Messages:
    99
    Likes Received:
    11
    I'm so annoyed with myself right now, I had scrapebox harvesting WP blogs for around 5 hours, last time i checked it google alone had harvested 950k, of course i know a lot of dupes, but then all of a sudden scrapebox just froze on me, and only way to get it back again was to kill the process, which of course also killed my harvested urls.

    GRRRRRR

    VERY annoying, if only i had installed the crashdump logger, I would have all my URLs, now i have to wonder wheel all the keywords again, and start all over.

    So if this has not happened to you yet, and you do not have the crashdump logger installed, please learn from my mistake and install it right now,
    it takes 10 seconds.

    Just download the addon, and then
    Options and select "Enable Crash Logging".

    You really dont want this to happen to you, i promise you that.
     
    • Thanks Thanks x 2
  2. carthom

    carthom Registered Member

    Joined:
    Sep 18, 2010
    Messages:
    77
    Likes Received:
    15
    Damn, thats alot of URL's to lose. I will have to get the dump logger to make sure this doesnt happen. Thanks for the advice :)
     
  3. boro

    boro Regular Member Premium Member

    Joined:
    Aug 6, 2008
    Messages:
    264
    Likes Received:
    161
    done thanks for the heads up
     
  4. teeniegenie

    teeniegenie Supreme Member

    Joined:
    Aug 28, 2010
    Messages:
    1,296
    Likes Received:
    662
    Location:
    The Cool Part of Vegas
    Outch...thanks for the heads up.:(
     
  5. boro

    boro Regular Member Premium Member

    Joined:
    Aug 6, 2008
    Messages:
    264
    Likes Received:
    161
    its an addon in scrapebox
    addons
    show available addons
    download it
    options
    enable crashdump logging
     
  6. almir012

    almir012 BANNED BANNED Jr. VIP Premium Member

    Joined:
    Feb 1, 2010
    Messages:
    556
    Likes Received:
    104
    Here is the one for you:

    http://www.unsavedwork.org/

    :)
     
  7. SixRox

    SixRox Newbie

    Joined:
    Sep 29, 2010
    Messages:
    24
    Likes Received:
    0
    Wow thanks for the info. Just got the plugin, that's alot of URLS to lose :(. Out of curiosity how long did that take you? I've got about 6k in the last 37 minutes.
     
  8. krzysiekz

    krzysiekz Senior Member

    Joined:
    Jul 29, 2010
    Messages:
    953
    Likes Received:
    578
    I'd be more worried about how much data usage I used wasted by losing those URL's, I imagine it would have been 1gb+ ?
     
  9. Olbol

    Olbol Junior Member

    Joined:
    Jul 22, 2010
    Messages:
    126
    Likes Received:
    24
    Occupation:
    IM
    My scrapes are normally about 5GB & 4,000,000 blogs, after duplicate domain removal eding with about 120k unique domains. Only takes a couple of hours though! Does it not save all the URLs you got to so far in the Harvester_Sessions folder? Mine are saved there automatically.
     
  10. tbrad

    tbrad Registered Member

    Joined:
    Aug 11, 2010
    Messages:
    99
    Likes Received:
    11
    ^^ Not if it crashes,

    as for data usage, im doing a scrape atm, almost 7 million harvested URLs so far, and its at around 7GB.

    All those from just 2 keywords in the wonder wheel which produced 25k more keywords!
     
  11. ducdaigia

    ducdaigia Newbie

    Joined:
    Oct 5, 2010
    Messages:
    9
    Likes Received:
    1
    Be careful next time dude
     
  12. alexy2k

    alexy2k Registered Member

    Joined:
    Aug 31, 2010
    Messages:
    55
    Likes Received:
    13
    hwo many proxies you use to get such amount of data?
    I use 10 private proxies and after somewhat 200mb. they get flagged.
     
  13. benpari

    benpari Registered Member

    Joined:
    Jan 7, 2010
    Messages:
    61
    Likes Received:
    13

    Random question-

    How are you able to scrape 7m URLS? No matter how many I scrape it only shows that one million were harvested after its finished.
     
  14. alexy2k

    alexy2k Registered Member

    Joined:
    Aug 31, 2010
    Messages:
    55
    Likes Received:
    13
    I am too only able to get 1kk blogs.
     
  15. onkelmicha

    onkelmicha Registered Member

    Joined:
    Mar 15, 2010
    Messages:
    74
    Likes Received:
    11

    Well that is one million per text file you will haev to trim them down one at the time and remove duplicate before you bash them all toghether.

    M
     
  16. thedorf

    thedorf Senior Member

    Joined:
    Oct 1, 2008
    Messages:
    1,193
    Likes Received:
    700
    Occupation:
    what? I gotta have a job?
    Location:
    BHW - Where else?
    Have just installed and activated crash dump addon. Thanks for the heads up, many of us are listening out here!
     
  17. benpari

    benpari Registered Member

    Joined:
    Jan 7, 2010
    Messages:
    61
    Likes Received:
    13
    Alright, just as misunderstanding then. I was under the impression you were harvesting 7m URLS in one go.