Scraping Websites

A

agag2

Supreme Member

Feb 17, 2009

1,303

260

• Aug 8, 2013

• #1

Hi

If i wanted to scrape news websites and extract ALL content from the site - how would I go about retrieving the urls for all content on site in absence of an xml file or sitemap?

Thanks

 

Emp1!

Junior Member

Dec 10, 2012

150

167

• Aug 8, 2013

• #2

Like all scrapers do: by following links...
You can use: httrack. 1 minute install, 1 minute to understand how it works (at least for the basics), and then you can download the website

Have Fun.

 

B

bubbaranks

Regular Member

Jan 29, 2013

200

34

• Aug 8, 2013

• #3

Scrapebox internal link scraper or dork site:targetsite.com

 

C

CanCan87

Newbie

Jun 8, 2013

40

5

• Aug 8, 2013

• #4

The one and only beast SB(scrapebox) will help you to find the desired sites.

 

starstrafe

Registered Member

Jan 27, 2013

86

54

• Aug 8, 2013

• #5

GSA SER has an option "crawl online" which crawls as many levels as you wish.

Regards