A good URL Harvester/Scraper
- Thread starter retry
- Start date
nipester
Regular Member
- Feb 1, 2009
- 262
- 28
I don't get this part, where does your scraper run exactly?
boffmaster
Junior Member
- Jun 1, 2009
- 144
- 32
The one at urlscraper.com works well for me.
Also has a free trial.
Also has a free trial.
- Jul 13, 2008
- 2,219
- 5,430
ScrapeBox will do this with public/private proxies and check Pagerank (shameless plug) plus Neta1o has one that's fairly decent a totally free one and a paid one. Pretty sure his free one will take one proxy, plus will scrape content and not just URL's.
dre2027
Newbie
- Jul 21, 2008
- 15
- 3
I'm also in need of a good scraper.
I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites.
So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required].
thx
I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites.
So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required].
thx
SpamHat
Junior Member
- Apr 27, 2009
- 151
- 67
http://gscrape.com/
It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls.
Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls.
Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
GuerillaDreamer
Registered Member
- Jun 1, 2009
- 91
- 64
It depends on what footprints you use when searching. So most if not all, if they can do one, they can do the other.
As stated above an excellent one is Hrefer that comes with Xrumer. Second to Hrefer i would say Scrapebox which for it's price does a kickass job.
