1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

problem: scrapebox cannot fit more than 1 million urls

Discussion in 'Black Hat SEO' started by nonai, Oct 6, 2014.

  1. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    63
    imagine you have a lot of keywords that you are harvesting. when scrapebox reaches one million urls, it keeps harvesting. however, at this point, the harvesting is useless because scrapebox cannot fit more than one million urls. Am I right?
    so what am I supposed to do if I have a lot of keywords and expect the harvested urls to go way over one million? it would be nice if SB would remove the duplicates behind the scenes while it's harvesting so the one million would not be so easily reached. can it do that?
     
  2. bk071

    bk071 Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Nov 24, 2010
    Messages:
    3,105
    Likes Received:
    7,917
    Occupation:
    I don't have a job
    Location:
    .............
    Install scrapebox crashdump logger and enable it.
    SB will now create a new file in the harvester folder whenever it reaches 1M URLs in the previous file e.g. if you harvest 7 million URLs, you will have 7 txt files and a million URLs in each file.
     
    • Thanks Thanks x 1
  3. Peter Ngo

    Peter Ngo Jr. VIP Jr. VIP

    Joined:
    Apr 23, 2013
    Messages:
    1,834
    Likes Received:
    1,461
    Occupation:
    I browse BHW for a living
    Location:
    The Internet
    It is a built in feature, when the scrape reach 1 million URLs, it automatically saves it to a different file.

    You should change your harvesting folder's size in setting option, so it won't interrupt your scrape.