[GET] Twitter email scraper (realtime)

[xpwizard]

I've decided to share some more code with the forum


Twitter Email Scraper


Features:
- Scrapes Twitter search api for emails in real-time
- Ensures no duplicates are saved
- Saves emails into monthly log files based on the originating domain name (under 'logs')
- Outputs log file with total emails found during each run (located in 'system/log/twitter/')

Requirements:
- Curl enabled
- Cron enabled (optional, but required to fulfill the potential of the script)
** Cron every one minute for the best results

How to use:
- Edit the first line in 'settings.txt' to your search query (I've left mine in there)
- If you save to file, define a file name
- Logs will be saved into a subfolder named "logs"

Note:
- This script was made to be run via cron. While it can run manually it will not be as efficient/less emails will be collected
- If you do not have php-cli enabled, you can wget the script as your cron call
- This script is apart of a bigger scraping project which is why there are so many folders
- If there are no emails, the log file will not output anything. As can be seen below at '6:00pm'


Samples of files:

settings.txt

"hotmail.com" OR "gmail.com" OR "yahoo.com" OR "aol.com" OR "live.com" OR "msn.com" OR "aim.com" -http

Gmail_com.txt:

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
...

09-01_log.txt:

####################### 5:55 PM 09 Jan 2012 #######################
5:55 PM  - Total = 12 Emails added
####################### 5:56 PM 09 Jan 2012 #######################
5:56 PM  - Total = 9 Emails added
####################### 5:57 PM 09 Jan 2012 #######################
5:57 PM  - Total = 18 Emails added
####################### 5:58 PM 09 Jan 2012 #######################
5:58 PM  - Total = 12 Emails added
####################### 5:59 PM 09 Jan 2012 #######################
5:59 PM  - Total = 5 Emails added
####################### 6:00 PM 09 Jan 2012 #######################
####################### 6:01 PM 09 Jan 2012 #######################
6:01 PM  - Total = 16 Emails added
...

Download:

http://www.mediafire.com/?lvo1pyy4yk5asoa

Virustotal:

http://www.virustotal.com/file-scan/report.html?id=ecd813d906f143bbea1f8d01a147b7ca37a05df6003ffc9448494e0f4c1bd886-1326160952

Result: 0/ 43 (0.0%)


**EDIT**

It seems a few members are unsure what this does...

This is a script that simply searches for the last 100 tweets (or the # since your last search) containing your search query.
From there it will look for any emails within those tweets and scrape them.

There is one flaw in this script that I forgot to mention. When you change the search query (in settings.txt), you have to delete the second line of text if it exists...
So you will only be left with the first line (no second line at all, no spaces, or anything).

 

[christianbed]

Thanks for the share xpwizard...will definitely try this one out.

 

[xpwizard]

Just to clarify what this does:

If setup via cron for every 1minute (recommended), then it will scrape the latest tweets since your last search.
In other words you get fresh emails every 1minute (or whenever you set the cron for).

The emails are not targeted. But you could easily change the search query to only include certain geo locations.

I've run this script for quite a long time, and I've found that as long as you stick to running the script every 1minute, then Twitter will not block/rate limit your IP.

 

[kiwilx]

and what do you recommend to send those emails?

 

[Humble]

and what do you recommend to send those emails?

Spam. Or you can scrape more targeted audience and send a relevant link.

Great tool OP! Thanks for the release

 

[xpwizard]

and what do you recommend to send those emails?

As humble said, you can either use it for bulk spam, but there are so many other uses:
- Check to see if the emails are associated with a FB account.
- Target keywords and send offers (iPad, iPhone, etc)
- Target emails from a certain country/city, and then send regional offers

Targeting keywords alone can make good money on an auto-pilot system.

Great tool OP! Thanks for the release

Thanks

 

[Baadier]

hmm looks great and a really useful addition to any seo arsenal even without being able to make it highly targeted though with location and quite possibly being able to add some bio into the query we might be able to make it more targeted.

 

[Xictus]

i dont understand.... this scrape emails from what? the tweets on the search query? or it scrape the registered email of the twitter accounts?

 

[xpwizard]

hmm looks great and a really useful addition to any seo arsenal even without being able to make it highly targeted though with location and quite possibly being able to add some bio into the query we might be able to make it more targeted.

Yep, if you change the search query in the settings file. Make sure to delete the second line (if it exists).
But if you're wanting to delve deeper into the options, simply do an advanced search on Twitter, and copy the parameters to the settings file,.. But it's highly configurable.

i dont understand.... this scrape emails from what? the tweets on the search query? or it scrape the registered email of the twitter accounts?

This grabs the latest 100 tweets containing your search parameter.
So lets say I search "ipad -http". That will search for any iPad tweets that do not contain http links, and will return the last 100 tweets (or the # since your last search).

This is the same as going to Twitter and searching for tweets. All this does is it will scrape any emails it finds within those tweets.

 

[Xictus]

Mmmm ok, that's nice, thanks....

Yep, if you change the search query in the settings file. Make sure to delete the second line (if it exists).
But if you're wanting to delve deeper into the options, simply do an advanced search on Twitter, and copy the parameters to the settings file,.. But it's highly configurable.



This grabs the latest 100 tweets containing your search parameter.
So lets say I search "ipad -http". That will search for any iPad tweets that do not contain http links, and will return the last 100 tweets (or the # since your last search).

This is the same as going to Twitter and searching for tweets. All this does is it will scrape any emails it finds within those tweets.

 

[giveitlegs]

Very cool little tool. Thanks. Will try it out tomorrow.

Cheers

 

[sfidirectory]

Looks like an awesome script . I tried to run it on my portable server but couldn't get the cron jobs to work so just uploaded it to one of my free hosting providers that allow cron jobs and it works pretty good .

 

[sfidirectory]

Hey everyone,

Have updated my twitter settings .txt file and wondering what you think of this search string?

keyword1 OR keyword2 OR keyword3 +"hotmail.com" OR "gmail.com" OR "yahoo.com" OR "aol.com" OR "live.com" OR "msn.com" OR "aim.com" -http
?since_id=159194453679157248&q=%22hotmail.com%22%20OR%20%22gmail.com%22%20OR%20%22yahoo.com%22%20OR%20%22aol.com%22%20OR%20%22live.com%22%20OR%20%22msn.com%22%20OR%20%22aim.com%22%20-http&lang=en

I am running a cron now but would like peoples opinions on it just in case it looks like it could be tweaked to get more targeted resources.

 

[Malazath]

Great share! I really appreciate it, I have been using it as targeted email marketing for my two main sites. I have been using it for about 5 days now and have found a 125% increase in traffic so far.

This is a great little script and I am saddened that I didn't think of this earlier as I am a PHP Developer as my main job.

Again I really appreciate it!

 

[MAC-11]

Thanks for the share, will post my results with it !

 

[monkid]

Hi, can anyone here tell me how to add cron job? Or I only just put my url to the command box? Thanks.

 

[extremehacker]

Did this stop working for anyone else? Is there an update?

 

[Backlink Database]

Thanks for the script, and the spam in my email.

 

[extremehacker]

No update on this??? For some reason mine stopped working.... Dont know if anyone elses has too???