Python script to check rankings on Google

[anhkiet]

I couldn't find any available scripts to do this so I wrote something for myself. Similar to my earlier post about a modification to Piwik, I am releasing this to the public. My hope is to, eventually, publish a set of tools available for people to use for free. I will probably give up before then, but it's worth trying no? The sad part is that the tools I am releasing will be source code, which means you will need to be a bit technical to learn how to run it. I feel that this is more flexible and allows for further innovations/additions in the future. Anyway, enough with the mumbo jumbo:

How to run:

1. Modify campaigns.ini with the URLs and keywords you want to check
2. python rankchecker.py

Since this was written for personal use, it contains very limited functionality and I don?t add features until I really need it. Here are some of the limitations (there are lots more):

• No proxy support
• Only checks google.com
• No recovery. If something breaks, you have to rerun it from the beginning
• No flexible way to change different campaigns

Due to the fact that I only need to check for my ranks once every few days, I set the script at a high delay rate (config.json) to prevent being blocked by Google and run it over the course of the night/day. If this is a limitation, you are welcome to modify. The best place would probably be in xgoogle (the Google scraping module I'm using). To find more information about this, you can visit my homepage (from profile, sorry I can't link) and my github repo (linked from the post for this script).

 

[anhkiet]

To clarify the last paragraph since it won't let me edit it:

The lack of proxy support is due to the fact that I don't check my rankings that often. If you find that to be a limitation, the best place to modify is probably xgoogle.

 

[anhkiet]

Naturally, the moment I posted my script, I got blocked by Google . I am in the process of testing out the new proxy support and should post that soon!

Not to mean to bump this thread in any way, but a member asked me "Could you direct me to some tutorials that will get me started with python programming?" Due to the fact that I cannot PM until I have 15 posts, my last resort is to respond to this thread. Firstly, I hope it gets to the right person, and secondly, I also hope that it's useful for others!

Q: Could you direct me to some tutorials that will get me started with python programming?
A: Sure! I'm not sure what your background in programming is, but Codecademy is a great place and they just recently added a Python section (tracks/python). My suggestion is to start out easy, but do something practical. Let's say you've done a few of the basic tutorials (syntax, flow control, lists and dictionaries), don't keep do more tutorials just for the sake of doing them. Go out and start writing a script that actually does something! For example, if I want to write a Google scraper, this is how you would want to think about the problem:

1) How do I get the HTML from a search result?
- What is a URL I can use to directly get a SERP
2) How do I retrieve the HTML using Python?
- Google something like "Python retrieve url", you'll likely see somebody who's done it before
- Once you've retrieved the URL, know what the code gives you to work with. In this case, it's a Python string, with its content being HTML. What can you do with this?
3) How do I get the information I want out of the HTML string? This requires more experience to know, but here's where you can email/PM people who are more experienced. Or Google like hell!
- String split on patterns (easiest)
- HTML parsing (harder)
- Regular expression (hard for beginners, and it never works 100% anyway)

Maybe you don't have to do something as complicated as that, but find something really simple that you need to automate and write it! It will be one of the hardest things you'll have ever done, but once you're done, it's VERY satisfying! If you need any help, I'm more than willing to give additional pointers. Naturally, this tip also goes to other languages like PHP and JavaScript.

-ak

 

[anhkiet]

I am back with another installment of the script! As promised in my earlier post, this version now has support for proxies. You will need to modify the config.json file and add in the proxies that you want. Remember that the last line in the proxy list DOES NOT have a comma.

New features

• Proxy support with configurable maximum consecutive failure retries
• Pass in alternate configs from command line
• Pass in alternate campaign information from the command line

For the configs and campaign information, you can combine these options. Say client1 pays you a lot of money, you can configure to have have more proxies with lower delays so you get them data much faster:

python rankchecker.py --config config1.json --campaign client1.ini

You can get more (technical) information from my blog mutedsignals (via google search or from my profile).

 

[Hinkys]

Thanks I've been looking for this kind of script with the source code since I wanted to delve in to bot coding but having limited coding knowledge, I didn't know where to start. And what better way to learn something then to stare at it and try to figure out how it works.

Cheers

 

[TuanNguyenYK]

Thanks you, that's exactly what I'm looking for. But now (2019), can this script work? I tried to write a program to check keywords ranking in google, but I'm afraid of that annoying captcha =_=

 

[Hash4Q3R]

Thanks you, that's exactly what I'm looking for. But now (2019), can this script work? I tried to write a program to check keywords ranking in google, but I'm afraid of that annoying captcha =_=

Naturally, the moment I posted my script, I got blocked by Google . I am in the process of testing out the new proxy support and should post that soon!

Not to mean to bump this thread in any way, but a member asked me "Could you direct me to some tutorials that will get me started with python programming?" Due to the fact that I cannot PM until I have 15 posts, my last resort is to respond to this thread. Firstly, I hope it gets to the right person, and secondly, I also hope that it's useful for others!

Q: Could you direct me to some tutorials that will get me started with python programming?
A: Sure! I'm not sure what your background in programming is, but Codecademy is a great place and they just recently added a Python section (tracks/python). My suggestion is to start out easy, but do something practical. Let's say you've done a few of the basic tutorials (syntax, flow control, lists and dictionaries), don't keep do more tutorials just for the sake of doing them. Go out and start writing a script that actually does something! For example, if I want to write a Google scraper, this is how you would want to think about the problem:

1) How do I get the HTML from a search result?
- What is a URL I can use to directly get a SERP
2) How do I retrieve the HTML using Python?
- Google something like "Python retrieve url", you'll likely see somebody who's done it before
- Once you've retrieved the URL, know what the code gives you to work with. In this case, it's a Python string, with its content being HTML. What can you do with this?
3) How do I get the information I want out of the HTML string? This requires more experience to know, but here's where you can email/PM people who are more experienced. Or Google like hell!
- String split on patterns (easiest)
- HTML parsing (harder)
- Regular expression (hard for beginners, and it never works 100% anyway)

Maybe you don't have to do something as complicated as that, but find something really simple that you need to automate and write it! It will be one of the hardest things you'll have ever done, but once you're done, it's VERY satisfying! If you need any help, I'm more than willing to give additional pointers. Naturally, this tip also goes to other languages like PHP and JavaScript.

-ak

Thanks a lot for these advises <3