1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Fastest way to extract backlinks of 25 million URLs?

Discussion in 'Black Hat SEO' started by ѕє∂σηιc, Oct 16, 2014.

  1. ѕє∂σηιc

    ѕє∂σηιc Registered Member

    Joined:
    May 27, 2013
    Messages:
    67
    Likes Received:
    10
    I have extracted about 25 million URLs with ScrapeBox' Link Extractor.

    ScrapeBox' Backlink Checker 2 is way to slow, to be honest - 1 URL per 10 seconds with a MozScape account.

    I also tried scraping from Google with the each URL in quotes, like "URL", and my proxies get banned with 1 connection and 15 seconds delay between each query.

    I have about 50 semi-dedicated proxies. And a good dedicated server with a 1 Gbps connection.

    What would be the fastest way to extract the backlinks of those?
     
    Last edited: Oct 16, 2014
  2. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    5,400
    Likes Received:
    3,691
    Home Page:
  3. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    The delay is pointless, the proxy will eventually get banned.
    You either need a sh*t load of proxies or a DSL connection.
     
  4. ѕє∂σηιc

    ѕє∂σηιc Registered Member

    Joined:
    May 27, 2013
    Messages:
    67
    Likes Received:
    10
    Not sure, but I think SEO Spyglass is pretty slow.

    I just wish SB still had the old Backlink Checker tool.
     
  5. prab1996

    prab1996 Elite Member

    Joined:
    Jan 8, 2013
    Messages:
    3,496
    Likes Received:
    2,027
    Occupation:
    your gf's <3 ♥♥♥♥
    Location:
    Prab1996.com
    Home Page:
    just add more accounts , with 50 proxies you can add 50 accounts and that will be about 5 urls per second.
    -=-
     
  6. hnmfvnxe

    hnmfvnxe Regular Member

    Joined:
    Nov 10, 2012
    Messages:
    281
    Likes Received:
    78
    There is your problem, your looking for free solution...lol
    Your not going to get free solution to scrape backlinks of 25 million urls.

    Your going to need ahrefs $499 or bigger plan if you want backlinks from millions of urls.
     
  7. junglist08

    junglist08 Junior Member

    Joined:
    Aug 1, 2013
    Messages:
    128
    Likes Received:
    25
    You need more proxies, the end.
     
  8. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    With 5 URLs per second it will take almost 58 days, so good luck.
     
  9. Paper-Boy

    Paper-Boy Elite Member

    Joined:
    Jun 17, 2009
    Messages:
    5,116
    Likes Received:
    1,821
    Why don't you get port proxies? There's a few bhw sellers selling these type of proxies and you can use up to 1000 proxies.
     
    • Thanks Thanks x 1
  10. ѕє∂σηιc

    ѕє∂σηιc Registered Member

    Joined:
    May 27, 2013
    Messages:
    67
    Likes Received:
    10

    Any recommendations?
     
  11. Execute

    Execute Supreme Member

    Joined:
    Aug 30, 2010
    Messages:
    1,349
    Likes Received:
    5,017
    Location:
    United Kingdom
    Have a look in the proxy section and talk to a few of the providers, they should be able to tell you straight away if you are a good fit for their service.