Scrape a site for Meta Desc, H1, Title on each page?

cookie1979

Regular Member

Jul 16, 2011

432

131

• Nov 18, 2013

• #1

Anyone tell me the best way to crawl a site and then get the Meta description, H1 tag, and Title from each of the sites urls?

Much appreciated.

 

G

gavinb

Junior Member

Dec 2, 2012

129

28

• Nov 18, 2013

• #2

i don't think there is any software on BHW capable of doing that. You can maybe write one yourself with basic regex model. Since all urls are from different sites you won't need proxies. Just scrape h1, meta and title tag from the page. However not all pages will have meta desc tag and some have titles as h1 tag.

Regards,

 

acidevil

Junior Member

Jr. VIP

Jan 10, 2009

178

351

• Nov 18, 2013

• #3

cookie1979 said:

Anyone tell me the best way to crawl a site and then get the Meta description, H1 tag, and Title from each of the sites urls?

Much appreciated.

Click to expand...

there is a software called ScreamingFrog. google it

 

clyde

Elite Member

Dec 5, 2009

1,990

1,136

• Nov 18, 2013

• #4

acidevil said:

there is a software called ScreamingFrog. google it

Click to expand...

This. The software is called SEO Spider:
http://www.screamingfrog.co.uk/seo-spider/

 

R

rratovsky

Junior Member

Jul 9, 2013

128

22

• Nov 18, 2013

• #5

Thanks for answering the OP question, I wanted to ask the same question. I'm going to check this program out.

 

cookie1979

Regular Member

Jul 16, 2011

432

131

• Nov 18, 2013
• Thread Starter Thread Starter

• #6

Thanks guys, will try them out!