1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Successfully Scraped and Entire Business Directory and now...

Discussion in 'Black Hat SEO' started by lafours, Aug 7, 2013.

  1. lafours

    lafours Power Member

    Joined:
    Sep 15, 2008
    Messages:
    576
    Likes Received:
    334
    After weeks of brainstorming/trialling I finally have a list of every single listing from a major directory that has approx 4 million listings. Now my problem is this, I need to grab the external URLs from each page but there just aren't enough proxies in the world it seems. I'm using Scrapebox's link extractor and my private proxies are good for a few hundred scrapes but then get banned. Scrapebox's own proxies are mostly dead by the time you've checked them and started using them so I'm not getting much success from them. I'm stumped, can anyone offer any other methods?
     
  2. kindarthur

    kindarthur Jr. VIP Jr. VIP

    Joined:
    Nov 27, 2011
    Messages:
    2,219
    Likes Received:
    332
  3. Emp1!

    Emp1! Junior Member

    Joined:
    Dec 10, 2012
    Messages:
    147
    Likes Received:
    167
    Proxygo seems to have one awesome proxy list for sale even if he does not accept "new" profile such as myself...
    He seems to provide 3k+ Google passed proxies which is awesome for SB :)

    EDIT : I do not know if he authorizes scraping other thing than Search Engine. Ask him first :)
     
    • Thanks Thanks x 1
    Last edited: Aug 7, 2013
  4. cryptons

    cryptons Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 12, 2013
    Messages:
    1,229
    Likes Received:
    447
    Occupation:
    Developer,social marketer
    there are many list search in the forum!