Discussion in 'Black Hat SEO' started by WTennisPL, Feb 25, 2011.
I am looking for a tool/software can get all link from a website. Anybody know ?
win web crawler , web data extractor , visual web spider , last one is the best one , web data extractor 8.1 you can find here on BHW
one question i want to know is, i have a site full of .html files and .php files too. Does any of the above tool just extract the .html files only?
Just want to know so i could download the one which does what i need?
All of them has exclusion extensions there , so if you are trying to avoid to extract .php files , you can try to add it to other extensions like .jpg , .swf and the others , I have never try it but I think it should avoid visiting it (all of them has this option I think , Visual Web Spider is best because you can also setup H1 , H2 , H3 , <title> , <body> , <p> , <br> , <html> almost all tags , so it is much more powerful than web data extractor or win web crawler , I like win web crawler because it is quick and for link extraction is the best , I wanted to upload it here too , but when I wanted to test it for viruses , I have found 13 - , so I didn't post it here , I am using it for 2 years , never had a problem but the program is not clean .
visual web spider is the best software ? The soft can index all link from a website as well ?
yes , it can , all of them support external links extraction, you can also setup following those external links to another sites , so it can grab the links infinitely
Are you owner of that soft ?
I just used trial version but it seem not work well.
I got error MySQL Error.... in Page Tittle and Page Text
If you own scrapebox you can use addon Link Extractor, if you don't own scrapebox you should buy it. If you don't want spending money you can do it with SeoQuake toolbar it's not perfect but it's free, also you can analyze bunch of other things.
Well you could always use a simple pc of java script code that you just simply create a bookmark in firefox and Name the bookmark something descriptive such as "Scrape All Links" then for the location copy & paste the below pc of java script.
Every-time you hit that bookmark it'll scrape all links on the page you have open and it'll open a new page with all the links it scraped so you can easily save it, copy & paste it somewhere, etc.
I used to use this alot and it's quite handy even though I haven't used it in awhile 8)
I have scrape as well but the soft do only link added, not do all website
I am not an owner of the soft - Visual web spider was best so I 've purchased it , Web data extractor I have uploaded here one month ago (search for it ), it works , win web crawler I can upload here too - works really very smoothly and quickly , but as I said , I use it 2 years but when I was willing to upload it here too, I made a virustotal which reported it is 13/43 virus positive .
I never got any error with visual web spider ... strange u did.
Ok, please send me one via pm
Win Web Crawler is the best.
Scrapebox can do it.
Just Scrape all of the sites URLs, or use its XML/Sitemap crawler functionality.
Then load those into the link extractor plugin, opt in for either internal or external, or both and let it loose.
You'll get the links you need
Download "link gopher" plugin for firefox. Simply click "extract all links" and it gives you a list of the urls
Is that for the page, or the whole site?
ScrapeBox can do this will the addon called "link extractor"
ScrapeBox has an addon like this?
yes SB does. check the addons menu for link extractor. im bummed i cant get into my computer right now to use it
scrapebox can do next to anything
Separate names with a comma.