1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Interest Check] Google Insights Keyword Scraper

Discussion in 'White Hat SEO' started by BoredCoder, Sep 16, 2008.

  1. BoredCoder

    BoredCoder Newbie

    Joined:
    Sep 16, 2008
    Messages:
    37
    Likes Received:
    8
    I read the tutorial floating around about using Google Insights to make money advertising and I thought it was a pretty neat idea.

    However, I got sick and tired of searching for the keywords myself, so I decided to automate the process.

    I have designed a small Console Application in C# that allows you to give the program a keyword list, time frame, and 2 threshold values. Inputting the correct combination of keywords and thresholds, you hit enter, and let the program run for a short while (or CTRL+C if you don't want to wait forever) and it will create a CSV file loaded with all of the breakout search terms from all of your searches.

    * The keyword list acts as a file which you will scrape all up-and-coming keywords on that page

    * The first threshold represents the search volume you'd like to scrape from your keyword list. Example: You search for "chicken recipes", and in the list of related search terms you see the following:

    1. chicken recipe
    2. recipe
    3. recipes for chicken
    4. recipes with chicken
    5. easy chicken recipes
    6. chicken breast recipes
    7. soup recipes

    This threshold (0-100) decides which of these keywords you would also like to scrape from the page.

    * The second threshold (0-5000) determines which keywords you feel are valuable in the list of Rising searches. The program will automatically add all breakout terms to your output, but you also have the option of choosing stuff like keywords that are 1250% on the rise (just input 1250) The program automatically adds the valuable keywords to your scrape list so you can exploit any further breakout terms related to the breakout term ;)

    * The time frame is the time in months which you would like to perform your searches on. I've only really used it for 1, 3, and 9 month time-frames, but I could alter the code if you're looking for specific date ranges.



    My questions:

    1) Is there any interest if I decided to release this product publicly?

    2) How much would you guys be willing to pay for it? ( Please, no crap answers, this is a powerful utility which makes advertising keyword investigations a very very trivial experience, plus I get $40/hour at my day job, please don't BS me :) )

    3) What other features would you guys like to see in such an application? I have decided to remove a lot of the data that is inherent with the searches, but I could just as easily put the stuff back in.


    Here's a sample of what the program looks like when you run if from a command console:
    Code:
    >GoogleInsightsScraper.exe /?
    Usage: GoogleInsightScraper <keywordFile> <timeFrame> <threshold_iterations> <threshold_breakout>
    
    Keywords:
    keywordFile  -  The list of keywords you want to search against
    timeframe  -  An integer value representing how many months from today you would like results for
    threshold_iterations  -  The threshold for iterations of specific keywords
    threshold_breakout  -  The threshold for values to add to the list of up-and-coming keywords
    Code:
    >GoogleInsightsScraper.exe keywords.txt 3 95 175
    Searching [0]
    Adding search term [2]: chicken recipe
    Adding result [1]: chicken piccata recipes (Breakout)
    Adding search term [3]: chicken piccata recipes
    Searching [1]
    Adding result [2]: chicken recipes.com (Breakout)
    Adding search term [4]: chicken recipes.com
    Adding result [3]: chicken parmigiana recipe (Breakout)
    Adding search term [5]: chicken parmigiana recipe
    Searching [2]
    Searching [3]
    Searching [4]
    >
    
    As you can see as shown below, the power is in the thresholds you set. (I broke out early because it would never end if I didn't)
    Code:
    >GoogleInsightsScraper.exe keywords.txt 3 25 100
    Searching [0]
    Adding search term [2]: chicken recipe
    Adding search term [3]: recipes for chicken
    Adding search term [4]: recipes with chicken
    Adding search term [5]: easy chicken recipes
    Adding search term [6]: chicken breast recipes
    Adding search term [7]: salad recipes
    Adding search term [8]: chicken salad recipes
    Adding search term [9]: chicken salad
    Adding search term [10]: baked chicken recipes
    Adding search term [11]: baked chicken
    Adding search term [12]: fried chicken
    Adding result [1]: chicken piccata recipes (Breakout)
    Adding search term [13]: chicken piccata recipes
    Searching [1]
    Adding search term [14]: recipes
    Adding search term [15]: chicken salad recipe
    Adding search term [16]: fried chicken recipe
    Adding search term [17]: chicken soup recipe
    Adding search term [18]: easy chicken recipe
    Adding search term [19]: chicken breast recipe
    Adding search term [20]: baked chicken recipe
    Adding result [2]: chicken recipes.com (Breakout)
    Adding search term [21]: chicken recipes.com
    Adding result [3]: chicken parmigiana recipe (Breakout)
    Adding search term [22]: chicken parmigiana recipe
    Searching [2]
    Adding search term [23]: recipes for dinner
    Adding search term [24]: fried chicken recipes
    Adding search term [25]: dinner recipes
    Adding search term [26]: grilled chicken recipes
    Adding search term [27]: healthy recipes
    Adding search term [28]: crock pot recipes
    Adding result [4]: slow cooker recipes (Breakout)
    Adding search term [29]: slow cooker recipes
    Searching [3]
    Adding search term [30]: pasta recipes
    Searching [4]
    Adding search term [31]: quick easy recipes
    Adding search term [32]: easy dinner recipes
    Adding search term [33]: healthy chicken recipes
    Searching [5]
    Adding search term [34]: easy recipes
    Adding search term [35]: baked chicken breast
    Adding search term [36]: grilled chicken breast
    Adding search term [37]: stuffed chicken breast
    Searching [6]
    Adding search term [38]: salad recipe
    Adding search term [39]: pasta salad recipes
    Adding search term [40]: potato salad recipes
    Adding search term [41]: salad dressing recipes
    Adding result [5]: zucchini recipes (Breakout)
    Adding search term [42]: zucchini recipes
    Adding result [6]: quinoa salad recipes (Breakout)
    Adding search term [43]: quinoa salad recipes
    Adding result [7]: picnic salad recipes (Breakout)
    Adding search term [44]: picnic salad recipes
    Adding result [8]: orzo salad recipes (Breakout)
    Adding search term [45]: orzo salad recipes
    Adding result [9]: lettuce salad recipes (Breakout)
    Adding search term [46]: lettuce salad recipes
    Adding result [10]: beet salad recipes (Breakout)
    Adding search term [47]: beet salad recipes
    Searching [7]
    Adding search term [48]: pasta salad
    Adding result [11]: grilled chicken recipes (Breakout)
    Adding search term [49]: grilled chicken recipes
    Searching [8]
    Adding search term [50]: recipe chicken salad
    Adding search term [51]: calories chicken salad
    Adding result [12]: tarragon chicken salad (Breakout)
    Adding search term [52]: tarragon chicken salad
    Adding result [13]: mango chicken salad (Breakout)
    Adding search term [53]: mango chicken salad
    Adding result [14]: cranberry chicken salad (Breakout)
    Adding search term [54]: cranberry chicken salad
    Adding result [15]: chicken salad recipie (Breakout)
    Adding search term [55]: chicken salad recipie
    Searching [9]
    Adding search term [56]: oven baked chicken
    Searching [10]
    Adding search term [57]: chicken breast baked
    Adding result [16]: bbq chicken recipe (Breakout)
    Adding search term [58]: bbq chicken recipe
    Adding result [17]: baked chicken drumsticks (Breakout)
    Adding search term [59]: baked chicken drumsticks
    Searching [11]
    Adding search term [60]: kentucky fried chicken
    Adding search term [61]: kentucky
    Adding search term [62]: recipe fried chicken
    Adding result [18]: zac brown band (Breakout)
    Adding search term [63]: zac brown band
    Adding result [19]: spicy fried chicken (Breakout)
    Adding search term [64]: spicy fried chicken
    Adding result [20]: nas fried chicken (Breakout)
    Adding search term [65]: nas fried chicken
    Adding result [21]: fried chicken song (Breakout)
    Adding search term [66]: fried chicken song
    Adding result [22]: fried chicken receipe (Breakout)
    Adding search term [67]: fried chicken receipe
    Adding result [23]: chicken fried lyrics (+680% )
    Adding search term [68]: chicken fried lyrics
    Searching [12]
    Searching [13]
    Adding search term [69]: food recipes
    Adding search term [70]: cake recipes
    Adding search term [71]: all recipes
    Adding search term [72]: bread recipes
    Adding result [24]: canning recipes (+520% )
    Adding search term [73]: canning recipes
    Adding result [25]: fig recipes (+490% )
    Adding search term [74]: fig recipes
    Adding result [26]: peach recipes (+270% )
    Adding search term [75]: peach recipes
    Adding result [27]: blueberry recipes (+230% )
    Adding search term [76]: blueberry recipes
    Adding result [28]: cucumber recipes (+210% )
    Adding search term [77]: cucumber recipes
    Adding result [29]: squash recipes (+190% )
    Adding search term [78]: squash recipes
    Adding result [30]: tomato recipes (+120% )
    Adding search term [79]: tomato recipes
    Adding result [31]: eggplant recipes (+110% )
    Adding search term [80]: eggplant recipes
    Searching [14]
    Adding search term [81]: chicken salad sandwich
    Adding search term [82]: pasta salad recipe
    Adding search term [83]: chicken pasta salad
    Adding search term [84]: chinese chicken salad
    Adding result [32]: healthy chicken salad (Breakout)
    Adding search term [85]: healthy chicken salad
    Searching [15]
    Adding search term [86]: fried rice
    Adding search term [87]: fried rice recipe
    Adding search term [88]: chicken fried rice
    Adding search term [89]: chicken fried steak
    Adding search term [90]: oven fried chicken
    Adding search term [91]: southern fried chicken
    Searching [16]
    Adding search term [92]: chicken noodle soup
    Adding search term [93]: soup recipes
    Adding search term [94]: chicken soup recipes
    Adding search term [95]: chicken tortilla soup
    Adding search term [96]: tortilla soup recipe
    Searching [17]
    Searching [18]
    Adding search term [97]: grilled chicken recipe
    Searching [19]
    Adding result [33]: fried chicken recipe (Breakout)
    Adding search term [98]: fried chicken recipe
    Searching [20]
    Searching [21]
    Searching [22]
    Adding search term [99]: dinner ideas
    Adding search term [100]: dinner for two
    Adding search term [101]: healthy dinner recipes
    Adding search term [102]: quick dinner recipes
    Searching [23]
    Adding search term [103]: fried rice recipes
    Searching [24]
    ^C
    >
    
    The associated CSV files:

    and
    If you open the files in Excel its basically 2 columns showing the Keyword/Ratio and looks nice. Feel free to advertise these keywords as they've become popular over the last 3 months :)

    Thank you for your time.
     
  2. BoredCoder

    BoredCoder Newbie

    Joined:
    Sep 16, 2008
    Messages:
    37
    Likes Received:
    8
    One thing to note is that my only keyword inside of keywords.txt for this example was chicken recipes. No other keywords were used to optimize my search. Powerful stuff :) I ran the tool for about 25 seconds to achieve those results.
     
  3. karum

    karum Jr. VIP Jr. VIP Premium Member

    Joined:
    Feb 9, 2008
    Messages:
    214
    Likes Received:
    45
    Are you selling this software or giving it for free?
     
  4. thedorf

    thedorf Senior Member

    Joined:
    Oct 1, 2008
    Messages:
    1,193
    Likes Received:
    700
    Occupation:
    what? I gotta have a job?
    Location:
    BHW - Where else?
    I use Google Insights and am always watching the breakout terms so I agree with you this has some potential.

    If you want to sell it as a product put up a never ending scrolling page pitching the benefits of saving time and giving that always sought after "first to market with the latest rising searches" with testimonials and seven+ "buy it now" buttons. Throw in a couple freebie ebooks... Start the price at $97 and see if it converts.

    Are you running the software under a command prompt in Windows or on Unix?
     
  5. BoredCoder

    BoredCoder Newbie

    Joined:
    Sep 16, 2008
    Messages:
    37
    Likes Received:
    8
    Reviving this thread.

    It runs in a command prompt in Windows.
     
  6. newmanhia

    newmanhia Junior Member

    Joined:
    Aug 1, 2008
    Messages:
    124
    Likes Received:
    185
    So basically what is done is scrapes the top 10 for the keyword entered, then it clicks on those top 10 and goes even deeper collecting more keywords through more layers.

    This is brilliant. A tool like that would be very useful, must better than google keyword estimator which is more useful for adwords publishers.

    I hadn't thought about google insights, but using some search terms it seems like really good stuff for either a short term or long term site.

    As long as it has a decent user interface, you should do exactly what thedorf says.
     
  7. foreclosureking

    foreclosureking Regular Member

    Joined:
    Sep 30, 2008
    Messages:
    413
    Likes Received:
    116
    So, any idea when you'll have something like this together?
     
  8. bigbag

    bigbag Newbie

    Joined:
    Apr 30, 2008
    Messages:
    47
    Likes Received:
    50
    interest check-in... will be nice to have this released :)
     
  9. SirCaptain

    SirCaptain Newbie

    Joined:
    Jan 31, 2010
    Messages:
    32
    Likes Received:
    9
    Home Page:
    Old thread, but I have also got a scraper that I'm looking to put out as a paid service, whereby you submit keywords and get reports back.

    Still any interest anywhere?