is there way to scrape pages with certain elements?

joshualo · Mar 28, 2017

hi guys, I want to know if there is a way to scrape all the sub pages under one domain with certain elements.
for example, there are 10000 sub pages under blackhatworld.com.like member1.blackhatworld.com...merber10000.blackhatworld.com.
some of them have embedded videos on it.
now I need to find all the pages with embedded videos(different pages with different videos)

scnonah · Mar 28, 2017

joshualo said:
hi guys, I want to know if there is a way to scrape all the sub pages under one domain with certain elements.
for example, there are 10000 sub pages under blackhatworld.com.like member1.blackhatworld.com...merber10000.blackhatworld.com.
some of them have embedded videos on it.
now I need to find all the pages with embedded videos(different pages with different videos)

Hope that's an example, not sure scraping blackhatworld is allowed in TOS.
You would need to scrape every page and check if the element exists.
From your post, you have a lot of work to do. Learn PHP or python and you can easily write a scrape bot.

is there way to scrape pages with certain elements?

joshualo

Registered Member

scnonah

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World