Tool ...

C

crisnoop

Newbie
Joined
Jul 10, 2012
Messages
8
Reaction score
1
Seeking a tool to extract all urls from a domain. I extract all urls from a forum or blog to know what is most pr!
 
Advertise on BHW
You could try Xenu Link Sleuth for the link extraction part. You can set it to dig X number of levels deep in a site to extract all urls from a site, ignore external links, and it is multithreaded. I usually run 50 threads. It's very fast and best of all free :-)

To check the PR in bulk you could use something like scrapebox or use Netpeak Checker...it's free too!
 
you can extract urls from same domain using scrapebox ,just put site: in the volumn ,select proxies and what type of blog it is and hit it.it will harvest all urls from single domain
 
Scrapebox can do it and for it you don't need proxy as well you are scarping its url from 1 domain instead that sites allow that much client request on server too fast ;)
 
I just bought a box scrape, someone could teach me to do what his friend iSynergy said:
site: in the volumn
put it where?
THX.
 
crisoop,

In the "harvester" box, just put in "site:domaintolookup.com" without the quotes. Also make sure the "custom footprint" option is selected. Then in the "Search Engines & Proxies" section put "1000" in the results box. This is the max number of urls you can grab at once.
 
If the site has a sitemap you can get more than 1000 results by using the "Scrapebox sitemap scraper" addon i think
 
I would use "site:domain" in Google (or scrapebox) no need to check a footprint, it'll just give you all the pages Google knows about. Then you can run them through PR checker, OBL, etc.
 
Thanks guys for sharing this going to have to break out my SB again.
 
Friends, thank you for help was exactly what I needed, now that SB is very powerful!
Another question: - If you get the self, posting comments on large lists you run the risk of being banned by google?
How does this story have banned the domain?
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top