1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ScrapeBox Limit! How to pass the 1million scraped url's limit?

Discussion in 'Black Hat SEO Tools' started by lewi, Sep 23, 2010.

  1. lewi

    lewi Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 5, 2008
    Messages:
    2,309
    Likes Received:
    818
    Hey,

    I keep seem to be hitting the 1million scraped url cap on scrapebox with my keyword list which i have already split into 8 parts and run them one at a time in each instance of SB.

    However then when i remove the duplicate url's the number of urls shoots down yet some of the keywords (actually a big chunk) haven't even scraped anything yet dude to this 'cap' on the number of url's you can scrape.

    Only thing i can think of doing is making each keyword list that i load up shorter and have around 20 to 30 of them but still really that means if i only choose one search engine and it scrapes all 1k results then i can only have 1k keywords per keyword list. Which is hardly anything!

    And i have around 100k different keywords :S

    So how do i remove or extend this cap to over 1million in SB?

    Lewi
     
    • Thanks Thanks x 1
  2. lewi

    lewi Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 5, 2008
    Messages:
    2,309
    Likes Received:
    818
  3. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    I want to know that too! :)

    I guess we have to wait until Sweetfunny removes the limit.
     
  4. JackSparrow

    JackSparrow Supreme Member

    Joined:
    Mar 24, 2007
    Messages:
    1,469
    Likes Received:
    2,054
    Occupation:
    Semi-Employed Pirate
    Location:
    search 4 blackhatsparrow
    Home Page:
    1 Million WOW; id be happy to hit 100,000.
     
  5. jiggsaw

    jiggsaw Regular Member

    Joined:
    Dec 25, 2009
    Messages:
    342
    Likes Received:
    154
    I always hit the 1 million limit. I just remove the duplicate sites, then remove the keywords scrapebox already scraped and continue.

    P.S. You have 100k keyword list without overlap? Impressive. I'm trying to get my list really diverse so I don't get as many duplicates. If any has any ideas I'd love to hear them.
     
  6. CyrusVirus

    CyrusVirus BANNED BANNED Premium Member

    Joined:
    Aug 20, 2009
    Messages:
    1,110
    Likes Received:
    686
    maybe contacting the programmer that made SB, he may have set a limit on it, or maybe he didn't set one at all and the program itself set it. ask him if he can set a limit. to say 20 million.
     
  7. chaos_id

    chaos_id Newbie

    Joined:
    Jul 7, 2010
    Messages:
    41
    Likes Received:
    15
    Occupation:
    blackhatter
    Location:
    internet
    Home Page:
    I imagine if i can harvesting that much, although I am quite satisfied with the scrapebox work today. Yes, we all need this scrapebox feature mainstay. would be heard from another colleague.
     
  8. deltrum

    deltrum Junior Member

    Joined:
    Aug 1, 2010
    Messages:
    102
    Likes Received:
    68
    1 million links scraped urls......LMFAO....I think submitting to all of these would see you hit with the banhammer !
     
  9. krocks

    krocks Registered Member

    Joined:
    Feb 4, 2008
    Messages:
    59
    Likes Received:
    11
    Not all links get indexed immediately.
     
  10. SamuelCoats

    SamuelCoats Junior Member

    Joined:
    Feb 4, 2009
    Messages:
    105
    Likes Received:
    125
    Location:
    Australia
    wen i scrape this many urls scrapebox usually freezes :p i hit like 400k and then BAMN im out with the list not being saved. I have to stop it just before it lol. But i dont mind. I only post to a few thousand each day.
     
  11. crazyflx

    crazyflx Elite Member

    Joined:
    Nov 9, 2009
    Messages:
    1,674
    Likes Received:
    4,825
    Location:
    http://CRAZYFLX.COM
    Home Page:
    I logged in JUST to post an answer to your question Lewi :)

    First off, there is no "limitation" with SB. No .txt file on an operating system running Windows can hold more than 1,000,000 lines. Since SB is getting it's data from .TXT files, that means it's limitations are only those of Windows...

    However, that doesn't mean you can't scrape more than 1,000,000 URLs in a single run. I've scraped over 30 - 40 million URLs from 1 single list of keywords (using multiple lists of keywords, I've scraped over 200 million URLs). It's so easy, it's going to make you mad that you hadn't already done it.

    Go to Add Ons, and download the Scrapebox Crashdump Logger Add On.

    Now, go to "Options -> Enable Crashdump Logging"

    Now, what's going to happen, is ScrapeBox is going to save, in real time, all harvested URLs to a folder. So, if SB is running, all URLs are being saved as they are scraped to a .txt file. If that .txt file it's saving those URLs to surpasses the 1 million mark, it starts a new .txt file. If that one goes over, it starts another.

    I've let SB run all night long scraping URLs and woke up to 30 different .txt files each containing 1 million entries.

    The location of these "saved in real time .txt files" is this:

    Folder SB Instance is running out of -> Harvester_Sessions -> Harvester_YEARMONTHDAY_HOURMINUTESSECONDS (so if that folder were made right now, it would read "Harvestor_20100925_023532) -> batch0001.txt

    Now, if batch0001.txt goes over 1 million entries, it starts batch0002.txt (and so on and so forth)

    Now, put ALL your keywords BACK into ONE list, and scrape away Lewi! No need to have so many files/lists all over the place!
     
    • Thanks Thanks x 19
    Last edited: Sep 25, 2010
  12. lewi

    lewi Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 5, 2008
    Messages:
    2,309
    Likes Received:
    818
    Thanks man i knew there had to be some way to do it!

    And that way seems more than perfect!

    Lewi
     
  13. onlinemarketnow

    onlinemarketnow Junior Member

    Joined:
    Mar 28, 2010
    Messages:
    145
    Likes Received:
    7
    that was a great post by crazy
     
  14. aznxmtg

    aznxmtg Regular Member

    Joined:
    Jul 9, 2010
    Messages:
    327
    Likes Received:
    48
    My scrapebox always freeze (no response) if I try to search like more than 60 ish keywords at once, I have to split them up.

    Am I missing something?
     
  15. Man4Life

    Man4Life Registered Member

    Joined:
    Jan 30, 2010
    Messages:
    54
    Likes Received:
    3
    Cool Bro
     
  16. evilartemis

    evilartemis Junior Member

    Joined:
    May 30, 2009
    Messages:
    155
    Likes Received:
    13
    is there a way to resume the harvest after recovering from the crash during harvesting?
     
  17. Snoopy69

    Snoopy69 Newbie

    Joined:
    Feb 6, 2011
    Messages:
    14
    Likes Received:
    2
    yeah, it seems to be more willing to freeze up for me during the time when I reach 1 million URLs, also I restart it after a harvest, because my memory use rarely goes back down to pre harvest levels after a harvest...
     
  18. blackh4t

    blackh4t Newbie

    Joined:
    Jan 30, 2009
    Messages:
    27
    Likes Received:
    3
    LMFAO what are you idiot doing in a blackhat forum ? obviously you are some whitehat fag so piss off..
    banhammer my ass...
     
  19. hitext

    hitext Junior Member

    Joined:
    Dec 29, 2009
    Messages:
    155
    Likes Received:
    52
    Occupation:
    Getting paid too little for doing too much
    Location:
    At the top of the sales graph.
    Home Page:
    Yet another great Scrapebox contribution from Crazyfix - that is great information!