My Journey To Create a Useful Keyword Suggestion App

Hi! Before I describe the motivations and goals of my journey, let me tell you a little about myself.


My Background


I 'm a 40-something YOUNG dad, tinkerer, and all around fun-seeker. I enjoy making things that people like to use. I have 16 years of experience in creating and designing software, and have been an active internet marketer on and off since 2005. I mostly make money from the bots and other personal creations I build. I treat it as a fun hobby that makes me very happy, and the money I make off of it is a bonus.


My Inspiration/Motivation


I've been reading journeys here, have gotten inspired many times over to ACT. You guys are fricken awesome. Watching others here take action, fall on their faces, pick themselves up to achieve great success has been an inspiration to me and I've picked up many techniques and ideas from reading this forum. I want to give something back.


I'd also like to spark an interest in developing apps, and in particular get the word out for a style of software development that I find fun and fulfilling: Functional Programming.


Most of the "free" keyword suggestion apps I've used are janky, poorly designed, require adobe flash, or limit you to only a few results. I'd like to use some of the best ideas from the other apps and wrap it in a pretty, easy to use interface. Note that this app will only do one thing and do it well. Suggest Keywords. It's not going to analyze competition. That's useful, but outside of the scope of what I want to achieve in this journey.

Bottom line is that I'm motivated to give something back, in even a small way and to have fun doing it.


My Goals


My goal is to build a keyword tool (obviously). What I want this tool to be is:

• A pleasure to use
• Free
• Responsive (Works on a desktop or phone)

I want to eventually find a way to monetize it while keeping it's basic functionality free and uncrippled. The ideas I have for this are:

• Ads.
• Pay features that are super useful but will not take away from the happiness of those using the free features.
• Voodoo (or whatever else I think of along the way)

I'd love to invite all of you to watch my progress as the app comes together, and to spark an interest in app development in general. Above all my #1 goal for this app is to have a ton of fun making it.


Future Plans


Next I'm going to write about the technology I'm going to be using to create the app.

Further into the future I'll post screenshots as I go along, and links to a beta version until I finally unveil the finished product. Feel free to suggest features and watch as I implement them.

Thank you all for reading this, and feel free to give your input!

Sounds like you've got some good intentions. Good luck!

** Update!**

Design

Today I'm going to think about the overall design of the new free keyword app. I want this to be fun for me to make and I want to learn a thing or two. I'm going to split up the design into two parts -
Front end and back end. I'll explain the tech that I'm going to use and why I'm using it. And what it means for the overall experience of the app. This part is critical. The choices I make here will make or break whether features are easy to add to the app, or if they're so painful to make that I avoid it.

Front End

The front end of the application is going to be a single page javascript app. Now I know that you may be thinking "single page apps suck for SEO". But I don't really care. This is not a content site, this is an app. I want people to use it and notice how smooth it is and how much nicer it is to use than the competitor's apps. If it turns out awesome, then people will use it. If necessary I can think of ways to produce an SEO friendly output for crawlers.

The tech that I'm using for this single page javascript app is called clojurescript. It's a functional language that compiles down to javascript. That's a mouthful, but in essence it helps me to write correct code in a productive way that I find fun. The fun factor is important because if I'm not having fun then what's the point?

You may have heard of react.js if you're a techie. If not, you've certainly used it. It runs facebook, instagram, and a growing number of web applications that you're familiar with.
I'll be using an excellent wrapper around react for clojurescript called reagent. This makes it easy for me to create an app-like experience in a web page.

There are many ways to structure a webapp with react, clojurescript, and reagent, but I'll be using a structure called reframe. It's going to help me stay organized and more productive, and will lead to fewer bugs in my code, and therefore a better user experience.

Why does all of this matter?
It's fun for me. Burnout is caused by boredom and tedium, so I'm creating this app in a way that's the most fun for me and it'll give me a chance to learn a new technology as I go.
When I start making money on this, it's like getting paid to learn, and getting paid to have fun. That's the kind of shit that makes a guy happy!

For now, I'm going to be using bootstrap for the css because it's simple to get started with, clean, and unobtrusive. I may end up restyling the app completely, or I may end up applying a bootstrap theme instead. I'll see how the finished product looks once it's functional then add some polish to it.

Back End

The back end will be written in clojure. Clojure is the parent to clojurescript (it's the language that clojurescript was based on). It's a functional programming language that runs on the java virtual machine. This means that I can write code on my machine and it will run the same as it will in production on a web host. The back end of the application will be responsible for querying the data sources for the keyword suggestions and sending them to the front end.

API

In order to get data from the back end to the front end I'm making an API. Probably JSON. I've still not decided whether I'll use regular ajax or a websocket for communication between the front and back end. Websockets would be fricken awesome and allow real time communication but I've basically no experience in creating an application that uses websockets. If I do use websockets, this will be interesting. I may stumble and fall and you guys will get to watch.


Thank you

Thanks for bearing with me everyone, I know that there are many of you who aren't techies but I know that all of you use apps. After reading this you'll understand what sort of thinking goes into making your favorite apps, and how mistakes are made then corrected. My hope is that you guys suggest features as you watch the app come together, then watch as the features come to life.

Screenshots of initial work coming soon...

** Update **

Progress I've made

I've gotten around to working on the user interface for the app. I've spent a lot of time thinking about how to structure
the data that the app is built around and how events such as mouse clicks and key presses will fit in to the overall flow of the app.


Here's a screenshot of what I have so far, and should give you an idea of what features I have planned for the beta. Some of the interactions are not implemented yet but I'm almost done on the front end side, and my goal for this phase of the front end part of the app is to get a minimum viable product that I'll add bells and whistles to later (like adwords CSV download).




After working on a few more interactions I'll then work on the API part (the glue between the front end and the server).
Then after that comes the really hard part of scraping my data sources without being banned.


Thanks for following my journey, and even though sometimes it feels like I'm talking into an echo chamber here, this keeps me motivated.

I don't have too much to update on today, as I've been really busy with work and family. I have been polishing up the user interface, and have also been diving deep into research about websockets. I'm going to go for it... The keywords are going to be streamed to the client via websocket. It'll take longer to make the app this way but it'll be a good learning experience for me, and should lead to a smoother experience overall.

I've come up with a name for the app, but won't reveal it right away until I have a beta version out, then you'll see why I named the app the way I did

Good luck to your journey!

** Update **

Thanks everyone for your support!

Had a wonderful holiday weekend with my family. The last before I send my kids off to school again. Instead of coding I spent a lot of time at the beach and hanging out with the kids. I'm a big believer in work/life balance, and we should all have a reason why we're doing what we're doing rather than to just work and make money. These kids are my reason.


Other than coding, I've been thinking about how I want to design the server portion of my app a little bit more. I sketched out how I envision it working on a piece of paper and laid down for a bit to ponder my design. Stopping for a while to think about how things will work is an important part of my workflow. Before I even start writing code, I'll spend a few hours thinking about whether my design is solid, if it can easily be changed, if I'll run into any limitations or even if I'm overengineering things. Yeah, I tend to do that sometimes.


Design is of utmost importance. Whether you're designing software, a PPC campaign, an idea for a niche site. Come up with an idea. Lay down for a bit to think about it. I'm not saying "don't work". No. I'm saying "ponder your work, and refine it".


Today I'm going to work on coding some of my design up. I'm excited to at least have a working beta out sometime this month. It's going to be awesome!

Are you using Google's AutoComplete API?

If yes then you should read this:
http://searchengineland.com/google-to-shut-down-autocomplete-api-on-august-10th-226131

weshare said:

Are you using Google's AutoComplete API?

If yes then you should read this:
http://searchengineland.com/google-to-shut-down-autocomplete-api-on-august-10th-226131

Click to expand...

I read that a while back, and I have a way to work around that. Not exactly ideal for Google, or the people were using their API, but the show must go on, right?

** Update **

It's been a busy month for me so far with my regular 50 hour a week career, other projects that are making me money, and now this.
I've got 3 kids in sports, too, so work/life balance is a must. That being said, I've had some time to think about this project and
get a good amount of coding finished.


Progress

Still working on the Server portion of the app. It's massively multithreaded which should give me really good performance and should scale up to a good number of concurrent clients using the app. I still have a few important things to consider when it comes to the overall design of the app. I can see this becoming a popular tool, and when it takes off, I have to make sure that I can handle all of the searches that I'll have to make. Right now there's a single point of failure in the app. It's usually the case with any kind of app like this but I see it, and
I still don't like it. I'll solve that problem when it actually becomes a problem though.


So far I have google, bing, youtube, amazon, and yahoo search suggestions integrated. I'm thinking about removing yahoo because some of it's suggestions... Well... fucking suck. I don't know where it gets some of them. I may try to tweak it, or just remove it entirely, since I can pull a lot of suggestions from other sources.


As weshare mentioned above about google's suggest api: Google suggest api is still operational, but I have an alternative in place for when or if they decide to shut the api down.


Still to do:


Add more web suggestions sources (I have 5 or so more that I'd love to integrate)


Add social media suggestion sources. This will probably not make it into the beta, since I will probably have to deal with account creation, and being banned from whatever social media
platform that I'm scraping


Caching. This is important so that common terms can be pulled from a cache rather than having to search my sources over again. I'll expire the cache after a few hours.


Send data back and forth to and from client to server and vice versa.


Most of the hard work is finished, so I expect a beta within a couple of weeks, as I have time considering some of my other projects, work, and family life.

** Update **


It's Alive!


I've implemented web, shopping and video keyword sources and it actually works now, passing data back and forth between the web app and the server. Awesome! Most things are working but I have some performance issues to work out. When keyword lists get into the thousands, the app bogs down. I have a few ideas on how to solve it, and will start experimenting with them as soon as the app
is feature complete.

It's finished other than a few fine details and the ability to download the generated keyword list. Also missing are the "social" keyword sources which I hope to incorporate soon into the app shortly after the beta.

Plan to release

* Secure my domain name
* Implement a couple of missing features + File download of keywords
* Host my beta version on redhat openshift free tier.
* Get your feedback
* Work out some quirks.
* Release Live version


Beta version will be released early next week. Stay tuned.

** Update **

It's finally here! The beta version is up.


Come check it out if you're feeling brave:


http://kwstreamer.herokuapp.com/


I tried to deploy to redhat openshift, but for hybrid http/websocket apps it sucks! So I decided to go with heroku, but with heroku there's a mandatory period of 6 hours where your app has to sleep. I hope this doesn't make anyone's experimenting with the app go awry.


I still have to implement a few things (in order of importance):


* Keyword download.
* The "Social" keyword channel isn't working. I have to implement this yet.
* Host it somewhere permanently.
* A prettier layout. Right now it's just twitter bootstrap.
* A nicer mobile layout
* Testing with IE/older browsers


Once I put the finishing touches on the app. I'll post the domain name where it can be accessed and call this journey a success. Then on to my next one. Thanks everyone for following along!

** Update **

I'm going to be posting updates as I tweak some things in the app. This is getting exciting now!

Today I've implemented the download feature and added instructions on how to use the app on the main screen.

I've tested a bit with IE, and it seems to work ok with IE11 (anyone try with 9 or 10?) I know it won't work with anything <9, but that's ok with me. I'm not interested in supporting old tech.

Now All I've gotta do is tweak the layout and add social channels support and I'll call this a beta (and host it on my own server)

Hey man this thread looks REALLY interesting! Right up my street. Went to test it but the "Start streaming" button is disabled! Down for maintenance?

Also wanna ask, are you querying the keyword suggestions via official APIs? Or being a good blackhat and scraping them yourself?

Cool project!

Nice tool, thank you!

JohnThe said:

Hey man this thread looks REALLY interesting! Right up my street. Went to test it but the "Start streaming" button is disabled! Down for maintenance?

Also wanna ask, are you querying the keyword suggestions via official APIs? Or being a good blackhat and scraping them yourself?

Cool project!

Click to expand...

Which browser are you using and what version? This works over websockets and some older browsers may not work with it. If the button is disabled, it means that the websocket connection can't be established. Perhaps try a refresh as well. Heroku has not been all that reliable for me, so it'll be better too, when I host it on my own VPS.


I do a combination of scraping and using the official API's. Google will be shutting down their suggest API soon, and I'm prepared for that. Also, I grab data from more than just google. I've got 6 sources that I'm scraping/querying right now and plan on adding more.

great plan mate good luck

Ah yes, arrived home and it's working now, this is pretty cool. I love the interactivity of it. I've learnt PHP this year, previously developing .NET applications and with PHP have loved being able to run scripts remotely on a linux VPS and letting them run 24/7 with low overhead (great for bots, mining, etc).

I do want to get into building interactive web applications at some point, like yours, perhaps even giving a frontend to some of the CLI scripts I've been making (mainly bespoke bots). Node.JS and other js libraries seem to be the way to go from what I've seen!

Best of luck to you on this journey. Tagging along.