1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrape a site for Meta Desc, H1, Title on each page?

Discussion in 'Black Hat SEO' started by cookie1979, Nov 18, 2013.

  1. cookie1979

    cookie1979 Regular Member

    Joined:
    Jul 16, 2011
    Messages:
    265
    Likes Received:
    70
    Location:
    Melbourne
    Anyone tell me the best way to crawl a site and then get the Meta description, H1 tag, and Title from each of the sites urls?

    Much appreciated.
     
  2. gavinb

    gavinb Junior Member

    Joined:
    Dec 2, 2012
    Messages:
    129
    Likes Received:
    28
    Occupation:
    AM
    Location:
    CA
    i don't think there is any software on BHW capable of doing that. You can maybe write one yourself with basic regex model. Since all urls are from different sites you won't need proxies. Just scrape h1, meta and title tag from the page. However not all pages will have meta desc tag and some have titles as h1 tag.

    Regards,
     
  3. acidevil

    acidevil Registered Member

    Joined:
    Jan 10, 2009
    Messages:
    69
    Likes Received:
    298
    Location:
    imagine-freedom
    there is a software called ScreamingFrog. google it :)
     
  4. clyde

    clyde Elite Member

    Joined:
    Dec 5, 2009
    Messages:
    1,991
    Likes Received:
    1,131
    • Thanks Thanks x 1
  5. rratovsky

    rratovsky Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 9, 2013
    Messages:
    128
    Likes Received:
    22
    Occupation:
    Affiliate Manager
    Location:
    BHW
    Thanks for answering the OP question, I wanted to ask the same question. I'm going to check this program out.
     
  6. cookie1979

    cookie1979 Regular Member

    Joined:
    Jul 16, 2011
    Messages:
    265
    Likes Received:
    70
    Location:
    Melbourne
    Thanks guys, will try them out!