Can Python Be used for creating sophiticated SEO toolss??

To the smart as who neg repped me for my post, great job. feeling better now you noob. left me message "uninformed opinions". Really. Let me slap you a bit with my uninformed opinions:


grouplens*org/node/244
blip*tv/pycon-us-videos-2009-2010-2011/pycon-2010-understanding-the-python-gil-82-3273690
blip*tv/carlfk/mindblowing-python-gil-2243379


Python version with micro-threads i was talking about is called Python Stackless - google it.

Now, about WebDriver and bindings. You can use Python with Selenium. You can use other bindings. That is not the same as you do in C# and WebDriver component. Personally I did not found them as elegant and well integrated as C#'s way but that would be a very subjective thing. Since both are from Microsoft they are integrated more elegantly.

You know, you could have just pointed out what you don't agree with instead of neg reping me. But then again, you probably can't so just did what you could. If you're smart or know something help everybody by offering information don't act smartas by droping a neg rep. That doesn't help anybody nor does it add to the discussion.

GTK: http://code.google.com/p/pywebkitgtk/ - QT: http://qt-project.org/wiki/PySide - GECKO: https://developer.mozilla.org/en-US/docs/PyXPCOM - ActiveX: http://sourceforge.net/projects/pywin32/ - About the GIL (issue that'll never make a SINGLE difference to make a sophisticated SEO tool for you or anybody else for that matter..) here's the answer at design time for scaling: http://www.parallelpython.com/ Still wanting to use threads the "low level" way? Swig and a small wrapper where you do your own locking mechanism without the one-size-fits-all in mind. Again not your nor anybody else will EVER need it for SEO tools no matter how sophisticated they are.

PappySmurph said:

AFAIK C# would be better for a few reasons. Python is good too. I just don't think it has an equivalent of .NET's WebDriver/MSHTML component so to handle JavaScript/AJAX pages you'd need to find some solution of integrating it with an actual web browser.

Also, Python threads are pretty bad. Very poor performance. There was a Python variant that had better threads but can't remember it's name. something that had what they called micro-threads.

Click to expand...

Again, IronPython is the solution, gives you access to all of the .Net libraries plus most of the standard python ones. There's also no GIL as you are using .Net thread classes. In the end it's like you are coding in C# but with Python syntax minus a few C# specific features and a minor difference in performance.

The real challenge of programming for websites resides in understanding web protocols, mimicking headers, using proxies, implementing APIs of the captcha breaking companies, etc. Most programming languages can do it, however some languages are more documented than others as far as tutorials.

Rapid application development should be your primary concern, which means program in whatever language you know the best. If you're programming for yourself, consider pre-made programs like zenno, ubot, imacros, etc. If you want to resell it, use whatever programming language you enjoy the most.

My preference is Microsoft Visual Studio 2010 express. It is entirely free and not gimped in any way. Just start a new project, throw a webbrowser control on it and voila, you've got a fully functional microsoft internet explorer browser embedded inside your program. You can programmatically navigate it, interact with the results, parse them, etc. You can even scrape ajax content because you have full access to all the browser content, even if its rendered by javascript/ajax. You can then build executables and resell them if you wish.

I believe the reason for using C# is webbrowser control.

Z0mbie said:

I believe the reason for using C# is webbrowser control.

Click to expand...

Nope. The reason for using C# or VB.net is multithreading and web requests.

DarkPixel said:

Nope. The reason for using C# or VB.net is multithreading and web requests.

Click to expand...

Other languages don't have multithrearing or web requests?

Web browser control makes it really easy to manipulate forms, buttons, or rather webpages in general, which is essential for link building softwares

I use perl to make bot and work great

Z0mbie said:

Other languages don't have multithrearing or web requests?

Web browser control makes it really easy to manipulate forms, buttons, or rather webpages in general, which is essential for link building softwares

Click to expand...

I didn't say other languages don't have threads or web requests. I personally use Delphi with threads and sockets. My points is, that .NET languages have huge communities, great support, and are generally easier to use. Also even with coding android apps, web forms can be manipulated, pretty easily, using the browser view.

DarkPixel said:

My points is, that .NET languages have huge communities, great support, and are generally easier to use. Also even with coding android apps, web forms can be manipulated, pretty easily, using the browser view.

Click to expand...

That, I totally agree with. Even though I hate .net

I've been developing a lot of my own SEO tools and bots written in Java.

Rationale behind this decision was:
- It had to be cross-platform, I develop on a Mac and all my servers are Linux
- All the tools had to be command-line interface because they're being run remotely on cheap Linux VPS's
- Plenty tools/libraries available to speed up development, Apache HttpClient, Jsoup, Selenium .. basically everything I needed
- Last but not least, I'm not selling these tools to newbies (so rule out Windows, and GUI)

All the code I write gets compiled/packaged into a JAR file containing all its dependencies. I can literally spin-up a VPS and script the installation of Java and run my code, when the 'job' or 'task' is complete I can shut that VPS down, at the moment I'm paying $0.003 per hour of compute time I actually use (Amazon EC2 Spot Instances).

A couple of weeks back I started to clean-up and open-source my code, if you are familiar with Java its easy to tinker with it: http://ikstob.com

I use Python to write different kinds of scrapers and simple bots.

I've never tried to make a python executable but my understanding is that in most of these cases the binary format is mostly just a binary version of the abstract syntax tree of the script with all variable and function names in-tact, and that this is trivial to decompile back into the almost original python source, so those methods are not likely to keep the source-code very secure. You can use jython to make a java jar file, or iron python to make a .net assembly. It's been a long time since I looked into either of these, but years ago anyway, hardly any of the popular python libraries were supported by jython or Iron Python. IMHO, one of the best reasons to use python is for the rich selection of open-source libraries that are out there, which it seemed like you had to give up to use Jython or Iron Python.