1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

A good URL Harvester/Scraper

Discussion in 'Black Hat SEO Tools' started by retry, Oct 4, 2009.

  1. retry

    retry Junior Member

    Joined:
    Oct 1, 2008
    Messages:
    102
    Likes Received:
    48
    I'm in need of a good URL Harvester/Scraper. It dosn't matter if it's free or not.

    It will need proxy support.

    Can you suggest something?

    Regards

    Retry
     
  2. Serpico

    Serpico Junior Member

    Joined:
    Sep 6, 2009
    Messages:
    118
    Likes Received:
    5
    Location:
    UK
    A really good one comes with xrumer, hrefer. You can easily get custom 200k urls a day on a normal broadband home connection. Than pr sort them with another tool in hrefer.
     
  3. nipester

    nipester Regular Member

    Joined:
    Feb 1, 2009
    Messages:
    256
    Likes Received:
    28
    I don't get this part, where does your scraper run exactly?
     
  4. boffmaster

    boffmaster Junior Member

    Joined:
    Jun 1, 2009
    Messages:
    144
    Likes Received:
    32
    The one at urlscraper.com works well for me.
    Also has a free trial.
     
    • Thanks Thanks x 1
  5. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,038
    Location:
    ScrapeBox v2.0
    Home Page:
    ScrapeBox will do this with public/private proxies and check Pagerank (shameless plug) plus Neta1o has one that's fairly decent a totally free one and a paid one. Pretty sure his free one will take one proxy, plus will scrape content and not just URL's.
     
  6. dre2027

    dre2027 Newbie

    Joined:
    Jul 21, 2008
    Messages:
    13
    Likes Received:
    0
    I'm also in need of a good scraper.

    I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites.

    So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required].

    thx
     
  7. kendra

    kendra Power Member

    Joined:
    Aug 12, 2009
    Messages:
    538
    Likes Received:
    334
    ScrapeBox is a great scraper. Well worth the money, take a look at Sweetfunny's sig link.
     
    • Thanks Thanks x 1
  8. TheEdge

    TheEdge Newbie

    Joined:
    Sep 19, 2009
    Messages:
    6
    Likes Received:
    2
    I use
    Code:
    www.clextractor.com
    , can download for free. Extract, bulk mail for free, up to 15 emails.
     
    • Thanks Thanks x 1
  9. JBatman

    JBatman Registered Member

    Joined:
    Mar 31, 2009
    Messages:
    64
    Likes Received:
    17
    Location:
    Indiana
    Home Page:
    I've got a free one around here somewhere, gimme a little bit and I'll PM you a link
     
  10. retry

    retry Junior Member

    Joined:
    Oct 1, 2008
    Messages:
    102
    Likes Received:
    48
    I did buy ScrapeBox for a couple of weeks ago, and I'm very happy with the product:)
     
    • Thanks Thanks x 1
  11. RiTu

    RiTu Regular Member

    Joined:
    Oct 28, 2007
    Messages:
    403
    Likes Received:
    158
    Location:
    shiver down your spine
    Hrefer for SERP scraping, Visual Web Spider for custom sites scraping.
     
  12. SpamHat

    SpamHat Junior Member Premium Member

    Joined:
    Apr 27, 2009
    Messages:
    151
    Likes Received:
    67
    Location:
    UK
    http://gscrape.com/

    It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls.

    Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
     
    • Thanks Thanks x 2
  13. YSL

    YSL Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 27, 2007
    Messages:
    379
    Likes Received:
    1,132
    Anyone having issue from harvesting url from google and yahoo recently?
     
  14. SpamHat

    SpamHat Junior Member Premium Member

    Joined:
    Apr 27, 2009
    Messages:
    151
    Likes Received:
    67
    Location:
    UK
    Nope - going fine for me.

    What sort of problems are you having - I'll help if I can.
     
  15. affportal

    affportal Newbie

    Joined:
    Oct 22, 2009
    Messages:
    15
    Likes Received:
    1
    Occupation:
    AffPortal.com programmer/owner
    Location:
    Lancaster Pa
    Home Page:
    I'm not having any issues from google or yahoo.
     
  16. GuerillaDreamer

    GuerillaDreamer Registered Member

    Joined:
    Jun 1, 2009
    Messages:
    91
    Likes Received:
    63
    Location:
    Inside the enemy lines
    It depends on what footprints you use when searching. So most if not all, if they can do one, they can do the other.

    As stated above an excellent one is Hrefer that comes with Xrumer. Second to Hrefer i would say Scrapebox which for it's price does a kickass job.
     
  17. youngguy

    youngguy Senior Member

    Joined:
    Apr 11, 2009
    Messages:
    1,053
    Likes Received:
    1,560
    Location:
    Hell
    @SpamHat: Nice code! Thanks to Firefox to allow post to cross domain through ajax.