How would you go about scraping this:

Swolsby · Jan 26, 2023

Ladies/Gentlemen,

How would you go about obtaining emails from similar https://www.imenet.com/search?search%5Bquery%5D=&search%5Bspecialty%5D=277&search%5Blocation%5D=&commit=Search? They list the phone numbers and website of the guy but not the emails, other than a contact form. My client is in the business of medical grade PPE and has asked to obtain ALL provider's from this website, and I don't see a way about other than manual work.

I am open to any ideas that can automate this procedure other than MTURK or Microworkers.

Much appreciated.

Steptoe · Jan 26, 2023

Yep, it looks like the email addresses are hidden via the form.

The way I would do it would be to scrape the url of each provider, then crawl each of those urls and try and pattern match an email address. Unless each of the providers have a very uniformly designed website, it's probably your best option.

Smackhead · Jan 26, 2023

Steptoe said:
Yep, it looks like the email addresses are hidden via the form.

The way I would do it would be to scrape the url of each provider, then crawl each of those urls and try and pattern match an email address. Unless each of the providers have a very uniformly designed website, it's probably your best option.

My thoughts exactly.

Swolsby · Jan 26, 2023

Steptoe said:
Yep, it looks like the email addresses are hidden via the form.

The way I would do it would be to scrape the url of each provider, then crawl each of those urls and try and pattern match an email address. Unless each of the providers have a very uniformly designed website, it's probably your best option.

Smackhead said:
My thoughts exactly.

Thank you both!

Unfortunately, that website is like a directory and each of those providers have their own unique email address.

Alternatively, I reckon this would be the best way given the circumstances, and would love to get your take on it: Scrape each provider's link to get their full name, contact info and website, THEN, Have that list on Microworkers (never used them in past) for those gents to google the providers name or go to their websites to retrieve their email address and paste onto a Gsheet or excel.

Thoughts on that? Also, please let me know if either one of you has experience with Microworkers.

Steptoe · Jan 26, 2023

Swolsby said:
Alternatively, I reckon this would be the best way given the circumstances, and would love to get your take on it: Scrape each provider's link to get their full name, contact info and website, THEN, Have that list on Microworkers (never used them in past) for those gents to google the providers name or go to their websites to retrieve their email address and paste onto a Gsheet or excel.

Seeing as this is the programming subforum, this is how I would script it:

1) Scrape the url of each member from the speciality areas search.
2) Scrape the name, company name, phone and website from these urls. The data follows a structure so no real effort needed.
3) Scrape the homepage of each website scraped, and try and locate an email address, through regex or other pattern matching. Each member's website will have a different design, so data may or may not be readily available. You could also scrape any urls containing 'contact' or similar just in case they also use a contact form.
4) Group all data up into a database and export as csv or xlsx.

I don't have experience with microworkers, but if you posted a thread in hire a freelancer or want to buy here, I'm sure you'll find people who can do it for you.

Swolsby · Jan 26, 2023

Steptoe said:
Seeing as this is the programming subforum, this is how I would script it:

1) Scrape the url of each member from the speciality areas search.
2) Scrape the name, company name, phone and website from these urls. The data follows a structure so no real effort needed.
3) Scrape the homepage of each website scraped, and try and locate an email address, through regex or other pattern matching. Each member's website will have a different design, so data may or may not be readily available. You could also scrape any urls containing 'contact' or similar just in case they also use a contact form.
4) Group all data up into a database and export as csv or xlsx.

I don't have experience with microworkers, but if you posted a thread in https://www.blackhatworld.com/forums/hire-a-freelancer.76/ or https://www.blackhatworld.com/forums/want-to-buy.92/ here, I'm sure you'll find people who can do it for you.

I appreciate your input @Steptoe; neatly written. Thank you again!

How would you go about scraping this:

Swolsby

Newbie

Steptoe

Elite Member

Smackhead

Regular Member

Swolsby

Newbie

Steptoe

Elite Member

Swolsby

Newbie

Main Menu

Marketplace

Making Money

BlackHat World