Scraping Question

Q

quickavenue

Newbie
Joined
May 29, 2015
Messages
7
Reaction score
0
I have a list of 1000+ domains.

I'd like to scrape for all emails associated with each domain

How would I go about doing this without scraping every page of the entire site and harvesting all the emails and then filtering by emails that contain each domain?
 
Advertise on BHW
You can run them through Srapebox's whois scraper or Link assistant..maybe these two would help you, but I only guesss
 
Seowarlords said:
You can run them through Srapebox's whois scraper or Link assistant..maybe these two would help you, but I only guesss
Click to expand...

seowarlords, yeah I'm not interested in scraping the WHOIS data, I'm looking for search operators or some other method that utilize Google to hunt out the email addresses to harvest. The other alternative is GSA Email Spider to scrape the WHOLE domain but some of these domains have hundreds of thousands of pages, it would take me ages.
 
extremeboy said:
Better to try services like Email hunter.
Click to expand...

That's what I'm thinking too. How do you think they're acquiring their data? A LOT of scraping? I can get the same results if I had 10x computers scraping the internet all day :D
 
quickavenue said:
That's what I'm thinking too. How do you think they're acquiring their data? A LOT of scraping? I can get the same results if I had 10x computers scraping the internet all day :D
Click to expand...

Well, are we dealing with hypotheticals, or do you really want the data? Obviously you don't have 10x computers scraping. For something like this ... perhaps you can scrape enough of the data in the short-term with short-term solutions to generate enough leads converting to cash so you can either buy more expensive tools, or the other equipment you want to mine this data. Of course, it's always an option to simply get whatever you can from the information brokers.
 
You could filter the pages down to ones that might be mentioning emails, footprints such as "@domain.com" "@gmail.com" "@outlook.com".
 
I dont know of any tools that can do that custom function you are looking for. If you have some budget on this, you can easily get a tool done which can scrape the domains single level deep, and scrape all emails and filter if it matches to the respective domain.

Well, if you want to go multiple levels down, the time you would need will go up exponentially.
 
Well you mentioned search operators and you can try these, they will likely get you most of the information but it will not guarantee you 100% complete data set. Also you mentioned all of the emails associated with the domain and this will only get you contact emails I suppose. Sometimes sites also use a contact form instead of an email.

That being said try these to start:

Code: 
site:domain.com inurl:contact

Code: 
site:domain.com inurl:about

You can easily do this in Scrapebox and then harvest the emails from Scrapebox as well.
 
quickavenue said:
That's what I'm thinking too. How do you think they're acquiring their data? A LOT of scraping? I can get the same results if I had 10x computers scraping the internet all day :D
Click to expand...
Agreed with above mate @davers instead of setup your own system use already built system to get your works done.

if you are going to need this type of solution for very long term and know you will have ROI on it so go a head for it else not worth it.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top