How to Scrape ALL AA LINKS On a WEBSITE

[Ghoztseo]

Hello guys i'm using Scrapebox but i don't know how to scrape all URL's of a Website? I tried Sitemap but it doesn't work!

 

[JournoNick]

Are you looking for tier targets?

 

[Ghoztseo]

Are you looking for tier targets?

yes! Journo! Do you know how to do it bro?

 

[reapV]

This can be very easily achieved by a small python or php script. I can point you to specific tutorials if you want to throw together a small tool.

 

[Ghoztseo]

This can be very easily achieved by a small python or php script. I can point you to specific tutorials if you want to throw together a small tool.

yes Reap v if you don't mind teach me how to scrape those bunch of links?

 

[0_00_0]

You can just use the "site:" operator and scrape URLs normally with Scrapebox I believe.

 

[Ghoztseo]

You can just use the "site:" operator and scrape URLs normally with Scrapebox I believe.

yes it does but it want like this. Check this out example

Site:xxxx.com

I need links from xxx.com and it should be like this

XXX.com/xx1
xxx.com/xx2
xxx.cpm/xx3
xxx.com/xx4

like that! it's for mass spamming.. btw i have tried Sitemap scrapper. it doesn't help me to find blogs with sitemap :\

 

[divok]

first try to scrape all internal links from homepage , keep going deep according to your needs and then on the last step extract all external links from those pages .

 

[Winternacht]

use the link extractor form scrapebox and scrape the internal links several times, this should give you almost every page on the website

 

[ija61]

As mentioned you can use the Link Extractor addon

1. Take your website and harvest with the "site:" operator (list1)
2. Clean the resulted list (list1) and load it in the link extractor extract internal links (list2)
3. Clean the resulted list (list2) from the link extractor and load it again and extract internal links. (list3)
4. Remove the links from list3 that you allready have in list 1 and 2 and then the resulted list load it again in the link extractor.
......

You can do this until you will get less new links.

If you have the automator you can use it and automate the process.

This is the best way to do it with SB, a second way it will be to use a web spider but a good one will cost you money.

 

[Ghoztseo]

Yes guys! thanks for your awesome advices! so i could use web extractor to harvest internal links easily xD