Zennoposter Blogs - Group pay programmer

@jazzc

I looked into Gearman. I even built my custom process management and did it in a multi-process way. The problem in this case is you end up using massive amounts of RAM. For example I built this multiprocess crawler and i spawned 150 processes. That eat up about 2GB RAM considering it was a very slim script. CPU load was initially the bottleneck, but after I optimized my language detection function I went from a load of 2 to 0.2 on a 2-core. However I was still using 2GB RAM. In order to max out usage on a 100mbps dedicated line I would need to spawn about 3 times as much, increasing server cost. Personally I prefer to optimize my scripts than pay more for hardware. Over time it proves to be a better design decision than upgrading hardware.

If the particular script would be more CPU or I/O intensive, then I wouldn't bother trying to over optimize RAM consumption. However, think about this situation. Let's say I want to do a proxy tester. I have let's say 500k proxies in the table and I want to test them at a rate of 400 in parallel. The bandwidth used is quite slim since I am only returning about 1KB of data per request. However, since I would be running 400 processes it would eat up a lot of RAM.

Certainly there are trade-offs. But it depends on what you are building in the sense of how much control you need over the parallelization process and how much you want to scale out.

Indeed, implementing a proxy checker with Gearman can be inefficient. But here 's an idea that I think will be very efficient

• Create a small ram filesystem and mount it
• Create minimalistic workers using the C API that simply launch the curl binary with the appropriate parameters and have it save the output to the above filesystem
• Process the output in the files from PHP

It 's not a purist approach, but it 's simple and effective.

I'm not saying the point-n-click part of ZP is buggy but the PHP API interfacing (the code generation or whatever is called)

Click to expand...

I had the chance to work with ZP's CodeCreator for a while. I was writing C#, not PHP, though. Didn't encounter many bugs, although I found some of their macros a bit slow/unreliable. I also had to write a custom wait function because their WaitDownloading() doesn't seem to work with ajax. Other than that, it was pretty fun and easy to work with. Zenno's browser component can perform just about anything you can think of.

jazzc said:

Certainly there are trade-offs. But it depends on what you are building in the sense of how much control you need over the parallelization process and how much you want to scale out.


Indeed, implementing a proxy checker with Gearman can be inefficient. But here 's an idea that I think will be very efficient

• Create a small ram filesystem and mount it
• Create minimalistic workers using the C API that simply launch the curl binary with the appropriate parameters and have it save the output to the above filesystem
• Process the output in the files from PHP

It 's not a purist approach, but it 's simple and effective.

Click to expand...

Nice idea jazzc, but i dont think PHP is good enough for a good proxy checker. I made a proxy checker myself a while back and got good speeds with the .Net IAsyncResult (a polling implementation) and the beautiful new .Net4 Concurrent classes. It was nice and fast.

However, for a really fast proxy checker, you would have to create an NDIS driver (sorry no linux here). This would kill even nmap for speed.

ANYWAY people on another note, here is a similar community 'automation' project that I stumbled accross the other day. http://splatforms.com

I'm interested in this, if this actually is going to happen can the op pm me please

count me in

I'm in if it's still available.

~divinci said:

ANYWAY people on another note, here is a similar community 'automation' project that I stumbled accross the other day. http://splatforms.com

Click to expand...

Thank you, I wasn't aware of this one

madoctopus said:

Here's a good start with programming in Python

Click to expand...

unreal . . i just read this entire thread and thought to myself, how uncanny, i got fed up a couple weeks ago and decided to get more proactive in learning programming, and i so i went on over to MIT world to look for some structured learning on cs, and found a gem. I was going to post it here, but you beat me to the punch, however. .you should post the entire course. Its completely free. . . and funny enough (while your not really learning a language per se, you are getting a bit of a start with python). For anyone who wants to get a real education on the basics of programming and thinking like a programmer:

http://ocw.mit.edu/courses/electric...o-computer-science-and-programming-fall-2008/

It everything full video lectures, syllabus, readings, etc. . . all 100% free from the best learning institution in these areas in the US.

I would like to contribute to the group buy thanks

This is a great thread! Was a group ever formed?

Running still? I saw some posts that this was closed, but OP you're up still?

albaniax - you missed one thing. Templates need to be updated weekly because Blogs sites changes often so group have to hire coder for that project. 1000-1200$ monthly and you can hire good coder from midde europe that will work 8 hours a day for such project.

50 guys in project - 1200/50 = 24$ monthly
100 guys in project - 12$ monthly.

! Forget about one time payment ! or it will be useless.

wordpress.com - hard to realize - $23.

Click to expand...

Are you kidding? Wordpress com is piece of cake. If somone is telling that template for wordpress com is difficult he must be total nob. Anyway what tempate will you get for 23$? I made myself few templates but my templates have integration with other tools which automaticly make backlinks for blogs.

In result my zenno template make linkwhell, second tier, and then blast 1 and 2 tier with backlinks. You will not get such templates for 23$ per site - impossible.

P.S. Dont pm me with questions how much for my template - i dont want to sell it.