1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[GET] Twitter email scraper (realtime)

Discussion in 'PHP & Perl' started by xpwizard, Jan 10, 2012.

  1. xpwizard

    xpwizard Junior Member

    Joined:
    Nov 6, 2010
    Messages:
    198
    Likes Received:
    122
    I've decided to share some more code with the forum :)


    Twitter Email Scraper


    Features:
    - Scrapes Twitter search api for emails in real-time
    - Ensures no duplicates are saved
    - Saves emails into monthly log files based on the originating domain name (under 'logs')
    - Outputs log file with total emails found during each run (located in 'system/log/twitter/')

    Requirements:
    - Curl enabled
    - Cron enabled (optional, but required to fulfill the potential of the script)
    ** Cron every one minute for the best results

    How to use:
    - Edit the first line in 'settings.txt' to your search query (I've left mine in there)
    - If you save to file, define a file name
    - Logs will be saved into a subfolder named "logs"

    Note:
    - This script was made to be run via cron. While it can run manually it will not be as efficient/less emails will be collected
    - If you do not have php-cli enabled, you can wget the script as your cron call
    - This script is apart of a bigger scraping project which is why there are so many folders
    - If there are no emails, the log file will not output anything. As can be seen below at '6:00pm'


    Samples of files:

    settings.txt
    Code:
    "hotmail.com" OR "gmail.com" OR "yahoo.com" OR "aol.com" OR "live.com" OR "msn.com" OR "aim.com" -http
    Gmail_com.txt:
    Code:
    checkboyzmusik@gmail.com
    twin2010@gmail.com
    desanam28@gmail.com
    dogeatdogtruck@gmail.com
    bossmac003@gmail.com
    flygoonrecords@gmail.com
    ohfancybaby@gmail.com
    fixyafaceentertainment@gmail.com
    luchiesantiago@gmail.com
    djhieyeqmp3@gmail.com
    finance4businesses@gmail.com
    generalphlint@gmail.com
    beautifuldreamcd@gmail.com
    ...
    09-01_log.txt:
    Code:
    ####################### 5:55 PM 09 Jan 2012 #######################
    5:55 PM  - Total = 12 Emails added
    ####################### 5:56 PM 09 Jan 2012 #######################
    5:56 PM  - Total = 9 Emails added
    ####################### 5:57 PM 09 Jan 2012 #######################
    5:57 PM  - Total = 18 Emails added
    ####################### 5:58 PM 09 Jan 2012 #######################
    5:58 PM  - Total = 12 Emails added
    ####################### 5:59 PM 09 Jan 2012 #######################
    5:59 PM  - Total = 5 Emails added
    ####################### 6:00 PM 09 Jan 2012 #######################
    ####################### 6:01 PM 09 Jan 2012 #######################
    6:01 PM  - Total = 16 Emails added
    ...

    Download:
    Code:
    http://www.mediafire.com/?lvo1pyy4yk5asoa
    Virustotal:
    Code:
    http://www.virustotal.com/file-scan/report.html?id=ecd813d906f143bbea1f8d01a147b7ca37a05df6003ffc9448494e0f4c1bd886-1326160952
    Result: 0/ 43 (0.0%)


    **EDIT**

    It seems a few members are unsure what this does...

    This is a script that simply searches for the last 100 tweets (or the # since your last search) containing your search query.
    From there it will look for any emails within those tweets and scrape them.

    There is one flaw in this script that I forgot to mention. When you change the search query (in settings.txt), you have to delete the second line of text if it exists...
    So you will only be left with the first line (no second line at all, no spaces, or anything).
     
    • Thanks Thanks x 10
    Last edited: Jan 10, 2012
  2. christianbed

    christianbed Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 17, 2011
    Messages:
    1,341
    Likes Received:
    879
    Location:
    alert("Make Money")
    Home Page:
    Thanks for the share xpwizard...will definitely try this one out.
     
  3. xpwizard

    xpwizard Junior Member

    Joined:
    Nov 6, 2010
    Messages:
    198
    Likes Received:
    122
    Just to clarify what this does:

    If setup via cron for every 1minute (recommended), then it will scrape the latest tweets since your last search.
    In other words you get fresh emails every 1minute (or whenever you set the cron for).

    The emails are not targeted. But you could easily change the search query to only include certain geo locations.

    I've run this script for quite a long time, and I've found that as long as you stick to running the script every 1minute, then Twitter will not block/rate limit your IP.
     
    • Thanks Thanks x 1
  4. kiwilx

    kiwilx Junior Member

    Joined:
    Oct 3, 2011
    Messages:
    186
    Likes Received:
    12
    and what do you recommend to send those emails?
     
  5. Humble

    Humble Registered Member

    Joined:
    Jul 17, 2010
    Messages:
    81
    Likes Received:
    51
    Occupation:
    Human
    Location:
    North American
    Spam. Or you can scrape more targeted audience and send a relevant link.

    Great tool OP! Thanks for the release
     
  6. xpwizard

    xpwizard Junior Member

    Joined:
    Nov 6, 2010
    Messages:
    198
    Likes Received:
    122
    As humble said, you can either use it for bulk spam, but there are so many other uses:
    - Check to see if the emails are associated with a FB account.
    - Target keywords and send offers (iPad, iPhone, etc)
    - Target emails from a certain country/city, and then send regional offers

    Targeting keywords alone can make good money on an auto-pilot system.

    Thanks :)
     
  7. Baadier

    Baadier Regular Member

    Joined:
    Nov 25, 2011
    Messages:
    214
    Likes Received:
    39
    Location:
    Cape Town, South Africa
    hmm looks great and a really useful addition to any seo arsenal even without being able to make it highly targeted though with location and quite possibly being able to add some bio into the query we might be able to make it more targeted.
     
  8. Xictus

    Xictus Regular Member

    Joined:
    May 16, 2011
    Messages:
    390
    Likes Received:
    185
    i dont understand.... this scrape emails from what? the tweets on the search query? or it scrape the registered email of the twitter accounts?
     
  9. xpwizard

    xpwizard Junior Member

    Joined:
    Nov 6, 2010
    Messages:
    198
    Likes Received:
    122
    Yep, if you change the search query in the settings file. Make sure to delete the second line (if it exists).
    But if you're wanting to delve deeper into the options, simply do an advanced search on Twitter, and copy the parameters to the settings file,.. But it's highly configurable.

    This grabs the latest 100 tweets containing your search parameter.
    So lets say I search "ipad -http". That will search for any iPad tweets that do not contain http links, and will return the last 100 tweets (or the # since your last search).

    This is the same as going to Twitter and searching for tweets. All this does is it will scrape any emails it finds within those tweets.
     
  10. Xictus

    Xictus Regular Member

    Joined:
    May 16, 2011
    Messages:
    390
    Likes Received:
    185
    Mmmm ok, that's nice, thanks....

     
  11. giveitlegs

    giveitlegs Regular Member

    Joined:
    Nov 26, 2008
    Messages:
    418
    Likes Received:
    195
    Occupation:
    Dir. of IT for a really big hotel company
    Location:
    Orlando / New York
    Very cool little tool. Thanks. Will try it out tomorrow.

    Cheers
     
  12. sfidirectory

    sfidirectory Senior Member

    Joined:
    Mar 29, 2010
    Messages:
    899
    Likes Received:
    483
    Occupation:
    Web developer/BTC enthusiast
    Location:
    php artisan make:migration
    Home Page:
    Looks like an awesome script :). I tried to run it on my portable server but couldn't get the cron jobs to work so just uploaded it to one of my free hosting providers that allow cron jobs and it works pretty good :).
     
  13. sfidirectory

    sfidirectory Senior Member

    Joined:
    Mar 29, 2010
    Messages:
    899
    Likes Received:
    483
    Occupation:
    Web developer/BTC enthusiast
    Location:
    php artisan make:migration
    Home Page:
    Hey everyone,

    Have updated my twitter settings .txt file and wondering what you think of this search string?

    Code:
    keyword1 OR keyword2 OR keyword3 +"hotmail.com" OR "gmail.com" OR "yahoo.com" OR "aol.com" OR "live.com" OR "msn.com" OR "aim.com" -http
    ?since_id=159194453679157248&q=%22hotmail.com%22%20OR%20%22gmail.com%22%20OR%20%22yahoo.com%22%20OR%20%22aol.com%22%20OR%20%22live.com%22%20OR%20%22msn.com%22%20OR%20%22aim.com%22%20-http&lang=en
    I am running a cron now but would like peoples opinions on it just in case it looks like it could be tweaked to get more targeted resources.
     
  14. Malazath

    Malazath Newbie

    Joined:
    Jan 25, 2012
    Messages:
    28
    Likes Received:
    4
    Great share! I really appreciate it, I have been using it as targeted email marketing for my two main sites. I have been using it for about 5 days now and have found a 125% increase in traffic so far.

    This is a great little script and I am saddened that I didn't think of this earlier as I am a PHP Developer as my main job.

    Again I really appreciate it! :)
     
    • Thanks Thanks x 1
  15. MAC-11

    MAC-11 Junior Member

    Joined:
    Feb 3, 2012
    Messages:
    168
    Likes Received:
    96
    Occupation:
    marketing/ sales
    Location:
    Canada
    Thanks for the share, will post my results with it !
     
  16. monkid

    monkid Power Member

    Joined:
    Jan 2, 2010
    Messages:
    680
    Likes Received:
    310
    Hi, can anyone here tell me how to add cron job? Or I only just put my url to the command box? Thanks.
     
  17. extremehacker

    extremehacker Junior Member

    Joined:
    Apr 17, 2010
    Messages:
    102
    Likes Received:
    288
    Home Page:
    Did this stop working for anyone else? Is there an update?
     
  18. Kries

    Kries Junior Member

    Joined:
    Aug 13, 2013
    Messages:
    180
    Likes Received:
    27
    Thanks for the script, and the spam in my email.
     
  19. extremehacker

    extremehacker Junior Member

    Joined:
    Apr 17, 2010
    Messages:
    102
    Likes Received:
    288
    Home Page:
    No update on this??? For some reason mine stopped working.... Dont know if anyone elses has too???