1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Python script to check rankings on Google

Discussion in 'Black Hat SEO Tools' started by anhkiet, Sep 24, 2012.

  1. anhkiet

    anhkiet Newbie

    Joined:
    Aug 18, 2012
    Messages:
    17
    Likes Received:
    11
    Home Page:
    I couldn't find any available scripts to do this so I wrote something for myself. Similar to my earlier post about a modification to Piwik, I am releasing this to the public. My hope is to, eventually, publish a set of tools available for people to use for free. I will probably give up before then, but it's worth trying no? The sad part is that the tools I am releasing will be source code, which means you will need to be a bit technical to learn how to run it. I feel that this is more flexible and allows for further innovations/additions in the future. Anyway, enough with the mumbo jumbo:

    How to run:
    1. Modify campaigns.ini with the URLs and keywords you want to check
    2. python rankchecker.py

    Since this was written for personal use, it contains very limited functionality and I don?t add features until I really need it. Here are some of the limitations (there are lots more):

    • No proxy support
    • Only checks google.com
    • No recovery. If something breaks, you have to rerun it from the beginning
    • No flexible way to change different campaigns

    Due to the fact that I only need to check for my ranks once every few days, I set the script at a high delay rate (config.json) to prevent being blocked by Google and run it over the course of the night/day. If this is a limitation, you are welcome to modify. The best place would probably be in xgoogle (the Google scraping module I'm using). To find more information about this, you can visit my homepage (from profile, sorry I can't link) and my github repo (linked from the post for this script).
     

    Attached Files:

    • Thanks Thanks x 3
  2. anhkiet

    anhkiet Newbie

    Joined:
    Aug 18, 2012
    Messages:
    17
    Likes Received:
    11
    Home Page:
    To clarify the last paragraph since it won't let me edit it:

    The lack of proxy support is due to the fact that I don't check my rankings that often. If you find that to be a limitation, the best place to modify is probably xgoogle.
     
  3. anhkiet

    anhkiet Newbie

    Joined:
    Aug 18, 2012
    Messages:
    17
    Likes Received:
    11
    Home Page:
    Naturally, the moment I posted my script, I got blocked by Google :). I am in the process of testing out the new proxy support and should post that soon!

    Not to mean to bump this thread in any way, but a member asked me "Could you direct me to some tutorials that will get me started with python programming?" Due to the fact that I cannot PM until I have 15 posts, my last resort is to respond to this thread. Firstly, I hope it gets to the right person, and secondly, I also hope that it's useful for others!

    Q: Could you direct me to some tutorials that will get me started with python programming?
    A: Sure! I'm not sure what your background in programming is, but Codecademy is a great place and they just recently added a Python section (tracks/python). My suggestion is to start out easy, but do something practical. Let's say you've done a few of the basic tutorials (syntax, flow control, lists and dictionaries), don't keep do more tutorials just for the sake of doing them. Go out and start writing a script that actually does something! For example, if I want to write a Google scraper, this is how you would want to think about the problem:

    1) How do I get the HTML from a search result?
    - What is a URL I can use to directly get a SERP
    2) How do I retrieve the HTML using Python?
    - Google something like "Python retrieve url", you'll likely see somebody who's done it before
    - Once you've retrieved the URL, know what the code gives you to work with. In this case, it's a Python string, with its content being HTML. What can you do with this?
    3) How do I get the information I want out of the HTML string? This requires more experience to know, but here's where you can email/PM people who are more experienced. Or Google like hell!
    - String split on patterns (easiest)
    - HTML parsing (harder)
    - Regular expression (hard for beginners, and it never works 100% anyway)

    Maybe you don't have to do something as complicated as that, but find something really simple that you need to automate and write it! It will be one of the hardest things you'll have ever done, but once you're done, it's VERY satisfying! If you need any help, I'm more than willing to give additional pointers. Naturally, this tip also goes to other languages like PHP and JavaScript.

    -ak
     
    • Thanks Thanks x 1
  4. anhkiet

    anhkiet Newbie

    Joined:
    Aug 18, 2012
    Messages:
    17
    Likes Received:
    11
    Home Page:
    I am back with another installment of the script! As promised in my earlier post, this version now has support for proxies. You will need to modify the config.json file and add in the proxies that you want. Remember that the last line in the proxy list DOES NOT have a comma.

    New features
    • Proxy support with configurable maximum consecutive failure retries
    • Pass in alternate configs from command line
    • Pass in alternate campaign information from the command line

    For the configs and campaign information, you can combine these options. Say client1 pays you a lot of money, you can configure to have have more proxies with lower delays so you get them data much faster:

    Code:
    python rankchecker.py --config config1.json --campaign client1.ini
    You can get more (technical) information from my blog mutedsignals (via google search or from my profile).
     

    Attached Files:

    • Thanks Thanks x 2
  5. Hinkys

    Hinkys Jr. VIP Jr. VIP

    Joined:
    Mar 3, 2012
    Messages:
    674
    Likes Received:
    547
    Location:
    Croatia
    Thanks I've been looking for this kind of script with the source code since I wanted to delve in to bot coding but having limited coding knowledge, I didn't know where to start. And what better way to learn something then to stare at it and try to figure out how it works. ;)

    Cheers