Web Scraping Advice

[LukeN153]

I'm looking to scrape websites with a certain piece of unique code. Does anyone know if there is a tool available to do this?

 

[TheToolbox]

Yes you can use Scrapebox. @loopline is expert in this tool

Or you code your scraper by you self with python requests and lxml.

 

[turelink]

Learn python

 

[TechEinstien]

You can use Python.

 

[loopline]

I'm looking to scrape websites with a certain piece of unique code. Does anyone know if there is a tool available to do this?

I need clarification to be able to more accurately help you.

You want to scrape the web looking for sites that have certain code?

Or you have a list of websites and you want to scan them and see if the contain certain code ?

or

 

[LukeN153]

I need clarification to be able to more accurately help you.

You want to scrape the web looking for sites that have certain code?

Or you have a list of websites and you want to scan them and see if the contain certain code ?

or

Hi @loopline
I'm trying to web scrape all websites using certain WordPress Themes. Do you know the best method to do this?

 

[Steptoe]

To search for a certain piece of code rather than keywords you can try https://publicwww.com/ but their database is a bit limited.
So for a particular WP theme I would search for "/wp-content/themes/themenamehere"

 

[BassTrackerBoats]

I'm trying to web scrape all websites using certain WordPress Themes. Do you know the best method to do this?

Somebody found something that is going to make them money once they find the process to gather the information.

I hope this does turn into revenue for you as you are setting up a nice base for that.

 

[PPC_BOSS]

why don't you use scrape box you might find it useful

 

[LukeN153]

Somebody found something that is going to make them money once they find the process to gather the information.

I hope this does turn into revenue for you as you are setting up a nice base for that.

You've never heard of anyone advertising this type of software? hmmm, interesting...

 

[BassTrackerBoats]

You've never heard of anyone advertising this type of software? hmmm, interesting...

I know about advertising using software such as you are looking for. I've done that in the past myself but my comment was on you finding an opportunity that could turn into money for you should what you are offering to those buy your product or service.

It reminds me of when I would sell in real life. I'd not target the businesses on the main roads but rather the ones on the off streets as I had little to no competition because almost everyone else would only target the businesses on the main roads.

 

[LukeN153]

I know about advertising using software such as you are looking for. I've done that in the past myself but my comment was on you finding an opportunity that could turn into money for you should what you are offering to those buy your product or service.

It reminds me of when I would sell in real life. I'd not target the businesses on the main roads but rather the ones on the off streets as I had little to no competition because almost everyone else would only target the businesses on the main roads.

I've created an Escrow payment plugin for WooCommerce and I'm trying to scrape some leads.

You are exactly right, trying to target some smaller businesses in some niche industries!

 

[Zazdrel]

I'm looking to scrape websites with a certain piece of unique code. Does anyone know if there is a tool available to do this?

It depends on what kind of information you need to get.
I handle this kind of task with xpath queries and screaming frog