How can I use scrapebox to harvest all URL's on a root domain?
- Thread starter cjshort
- Start date
- Mar 21, 2013
- 24,833
- 31,592
This is basically what SB is for....
There is an email harvester plugin for SB and tutorials specifically for this purpose.
There is an email harvester plugin for SB and tutorials specifically for this purpose.
Yes I know how to harvest the emails. I just am not sure how to harvest all the URLs of a site from its root domain so I can then search all them URLs for emails. Any ideas?
- Mar 10, 2010
- 42,695
- 107,115
Screaming Frog can harvest all the urls from a domain. I'm sure that a small script can harvest the urls automatically for you so you do not have to reenter domains time and time again and then you can use SB.
Sephirot_90
Junior Member
- May 20, 2011
- 100
- 39
You can do this in 3 ways: 1. just add site: before every url you need to crawl and put it into a harvester. This will find all the urls that are indexed in the search engine.
2. Use the link extractor, set it to scrape internal links. Insert the 600+ urls and it will find an x number of urls of these sites. Now insert the x number of urls you got and repeat until you get all the internal urls.
3. using the sitemap scraper which is also a plugin in scrapebox
With this three ways you can find all the links you need. happy scraping
2. Use the link extractor, set it to scrape internal links. Insert the 600+ urls and it will find an x number of urls of these sites. Now insert the x number of urls you got and repeat until you get all the internal urls.
3. using the sitemap scraper which is also a plugin in scrapebox
With this three ways you can find all the links you need. happy scraping
Perfect, thanks boss.
In regards to 1, where am I putting the url with site: before? In the keywords or the footprint?
J-S-T
Supreme Member
- Jul 27, 2013
- 1,296
- 656
Enter all 600 urls in the keywords box
Then create a .txt file and input site: in it and save the file.
Now click on "M" which means Merge in scrapebox, select the .txt file you created.
Now you should have it like this
site:url1
site:url2
site:url3
Start Harvesting.
Option number 2 which is using Link extractor plugin is much better for you i guess.
Sephirot_90
Junior Member
- May 20, 2011
- 100
- 39
I agree. Its faster,you will get urls that arent indexed and you don`t need proxy`s.
