How to program a keyword density checker - need advice...

Discussion in 'Other Languages' started by savvypro, Nov 22, 2010.

  1. savvypro

    savvypro Regular Member

    Jan 14, 2010
    Likes Received:
    How to program a keyword density checker - need advice...

    I'm currently programming a custom CRM, one of the modules that I have lined up, is a keyword density checker.

    Determining single word keywords will be easy, what I'm not sure on (more like haven't got a clue on how to proceed) is how to figure out the multi word key phrases.

    The way the module will work is as follows:
    - User inputs a keyword
    - Keyword gets put into a queue, where it waits for a worker node to process the request.
    - Worker node does a google search for the keyword, scrapes the links in the SERP's, then downloads the linked webpages and processes them for their keywords.
    - The queue is updated with the results and returned to the user. ​

    The reason I mention queues is that the CRM will be event based, built around message queues. Which will give me the ability to distribute the CRM, and take advantage of cheap cloud computing.

    As I said above, I'm not sure on how to determine the multi word key phrases from just the content of the linked webpages.

    The language I'm using to build the CRM is python, but because it will be queue based. I can have different jobs performed by different languages if need be.

    Anyone have any advice or even an existing script (source code ideally) they can point me to would be useful.