1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Good Web Crawler better than Scrapebox?

Discussion in 'Black Hat SEO' started by JiveTurkeySammich, Aug 23, 2016.

  1. JiveTurkeySammich

    JiveTurkeySammich Regular Member

    Joined:
    Apr 6, 2014
    Messages:
    291
    Likes Received:
    46
    Gender:
    Male
    Looking for a good webcrawler other than Scrapebox. Need it to extract all the available links in the website.

    Needs to be multi-threaded with proxy support.

    Scrapebox keeps crashing and I need something with more filters if its available.

    I've searched and tried a few out today, but they didnt multi-thread or had proxy support.
     
  2. Mr.Charles

    Mr.Charles Jr. VIP Jr. VIP

    Joined:
    Apr 6, 2012
    Messages:
    373
    Likes Received:
    30
    Gender:
    Male
    Occupation:
    Web Developer
    Location:
    Google
    Home Page:
    Have you tried gscraper? Or you can also get all internal and external links using the online tool HREF
     
  3. JiveTurkeySammich

    JiveTurkeySammich Regular Member

    Joined:
    Apr 6, 2014
    Messages:
    291
    Likes Received:
    46
    Gender:
    Male
    i'll give it a try. The sites Im trying to crawl have 100,000,000 links internally. Need something strong.