Tool for email scraping ? - Must be able to get past login

[MinistryOfThought]

Hi BHW,

I am looking for a tool that will allow me to scrape emails from sites like forums, linkedin, twitter, ect..
I found a free tool call EmailscaperWizard which is free and good but it cannot get pass logins

I want to scrape a forum that requires you to login to see posts and member info (I have a login)

Can anyone recommend a good software ?
I am not sure if scrapebox can do this. Paid is fine.

Thanks !

 

[goodperson]

Try Email Extractor, 1 addon of firefox. It's ok

 

[MinistryOfThought]

I am looking for something that will also be able to crawl the site. There are thousands of users and posts.

 

[ads9oa7]

I'd also be interested in such a tool. Especially the full website scraping part...

There is a python framework optimised for scraping, it's called scrapy and there are some nice tutorials on YouTube. Looks like a bit more work than I was willing to invest...

 

[MinistryOfThought]

I'd also be interested in such a tool. Especially the full website scraping part...

There is a python framework optimised for scraping, it's called scrapy and there are some nice tutorials on YouTube. Looks like a bit more work than I was willing to invest...

perhaps hiring a VA do the leg work ?
or pooling resources together ? what use are you trying to get out of yours ?

 

[skytek666]

Im also interested in what your after as i could be interested?

 

[MinistryOfThought]

Im also interested in what your after as i could be interested?

I am trying to get a bot that will scrape the emails from users profiles. The site that I want to run it on has thousands of users however, it is brick-walled behind a login.
I have a pro account and thus access to the users via member search and forum posts, I want a bot to crawl the forums and go to member's pages to get emails.
Should be simple, I tried a few bots but they dont work since they have no login functionality.

 

[MinistryOfThought]

Try Email Extractor, 1 addon of firefox. It's ok

Do you know of anyway to integrate this with a crawler ?

 

[apexbeat]

You want to scrap a website after your account logs in right? You can do this with Linux wget command for free (if website has cloud flair that might be an issue)
wget --recursive --post-data "email=$EMAIL&password=$PASSWRD" read the wget manual - its a long read but should will work (maybe not if they have cloud flair)

https://www.gnu.org/software/wget/manual/wget.html

If u need help and i got the time PM me. because "email=$EMAIL&password=$PASSWRD" might not be the proper format expected by the site

 

[SEO Sicko]

You can try The Best Email Scraper.
It is easy tool you can just place the URL's you want to scrap.
After the process the results will be downloaded automatically.

 

[MinistryOfThought]

You want to scrap a website after your account logs in right? You can do this with Linux wget command for free (if website has cloud flair that might be an issue)
wget --recursive --post-data "email=$EMAIL&password=$PASSWRD" read the wget manual - its a long read but should will work (maybe not if they have cloud flair)


If u need help and i got the time PM me. because "email=$EMAIL&password=$PASSWRD" might not be the proper format expected by the site

Thanks I will take a look into this. I have entry level coding skills but hopefully I can figure it out. I can't PM you now due to post count but I will give it a try myself and let you know if I need a little help. Big thanks.

 

[MinistryOfThought]

You can try The Best Email Scraper.
It is easy tool you can just place the URL's you want to scrap.
After the process the results will be downloaded automatically.

I am not sure if I can find the tool you are referring to since a google search for that term brings many results and I didnt see any software with that specific link. Maybe you could link directly ?

Are you suggesting that I scrape URLs with another tool such as scrapebox and then run the URLs through "the best email scrapper?" I am looking to crawl thousands of pages so manual is not really an option.
The solution above could work though since although user profile data is blocked out I can browse the forum and get to member's pages.
I could then log in and scrape using the best email scraper on my browser I assume

 

[kimberlarck]

maybe u can try scrapebox....

 

[fastlinks]

I can make custom bot for this task, add me

skype: ipowerhost2