1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

is there way to scrape pages with certain elements?

Discussion in 'Black Hat SEO Tools' started by joshualo, Mar 28, 2017.

  1. joshualo

    joshualo Registered Member

    Joined:
    Nov 9, 2011
    Messages:
    67
    Likes Received:
    8
    hi guys, I want to know if there is a way to scrape all the sub pages under one domain with certain elements.
    for example, there are 10000 sub pages under blackhatworld.com.like member1.blackhatworld.com...merber10000.blackhatworld.com.
    some of them have embedded videos on it.
    now I need to find all the pages with embedded videos(different pages with different videos)
     
  2. scnonah

    scnonah Jr. VIP Jr. VIP

    Joined:
    Jan 2, 2017
    Messages:
    227
    Likes Received:
    124
    Gender:
    Male
    Hope that's an example, not sure scraping blackhatworld is allowed in TOS.
    You would need to scrape every page and check if the element exists.
    From your post, you have a lot of work to do. Learn PHP or python and you can easily write a scrape bot.