you'd be looking at a $20-30k tool.

more like $5000

Most coders with degrees in Comp Sci, couldn't do it. It's beyond their abilities (maybe not 10-15 years into their careers though?)

has nothing to do with experience measured in time but with experience in the particular field of scraping/submitting. is not particularly hard but you need to know specifically how to code scrapers/posters and have good software architecture knowledge (including design patterns).

I'd say roughly 90-95% of "programmers" only program in Java

not true

Java is next to useless if speed is the goal

certainly not true. Java is very fast, almost as fast as C++ & C#. In certain cases is faster. Only thing is slower at is rendering GUI and that is not noticeable. Only thing where Java would be a bit too slow is game development. Even more, the shittiest Java app will still be faster than what you can do on a 1Gbps connection.

Python is a high-level language like Java

No is not! Python is interpreted and can be precompiled (just like PHP). Also it has dynamic typing. Java is a compiled language (JIT/Java VM not machine/assembler code like C/C++) and with strict types. Java speed vs Python speed is faster and a lot faster than vs PHP speed.

Though I think, in general, you could use any high-level language for this task like Perl?

Yes but you need a browser emulation library that can execute JavaScript in order to be able to trully emulate user actions for any platform. I don't think Perl has such a thing unless it can work with Selenium or something similar. It has WWW::Mechanize but from what I know that doesn't execute JavaScript is just a regular HTML parser.

You can basically use any language but since some don't support browser emulators you either have to hook into a real browser's API (like SENuke and others does, which poses security problems) or "hack" your way around the limitations. Hence, from what I know so far, Python is the best solution (from my point of view).

Update: It seems Selenium works with many languages including PHP Didn't realized it before as all examples i was looking over were in Python. I should go and play with it in PHP now...

Originally Posted by madoctopus

No is not! Python is interpreted and can be precompiled (just like PHP). Also it has dynamic typing. Java is a compiled language (JIT/Java VM not machine/assembler code like C/C++) and with strict types. Java speed vs Python speed is faster and a lot faster than vs PHP speed.

Interesting... I appreciate your explanation.
I guess Python would be excellent for this task.

Are you programming into a single interface? or are you creating a ton of executables with Python?

Likewise, I graduated with a MS in Nuclear engineering and a BS in Physics. Most programmers out of my school were almost entirely programming in Java, and apparently, they were only required to take a single C course during their tenure in college.
I suppose I was referencing the US educational system here, where it is heavily involved in programming in that language.

Personally, I think $5000 to get a tool that works on 90-100 web 2.0s would be tremendously cheap... I could not see such a tool taking anything less than 2-3 months (at least if you wanted a GUI)
Anyway I appreciate your explanations in this thread. It has seriously spurred my interests and I will be picking up this language as soon as I can.

Man this is a great convo, sadly I know nothing about what you guys speak of except only in the most basic sense. However my wifes brother is visiting this month and he has 2 IT based companies and is a graduate from MIT, perhaps I can talk him into allowing some resources into developing something along the lines of what you guys are talking about. If the ROI is high enough I am sure that he would not be opposed to at least listening. I just have to be able to speak about it semi-intelligently.

Personally, I think $5000 to get a tool that works on 90-100 web 2.0s would be tremendously cheap... I could not see such a tool taking anything less than 2-3 months (at least if you wanted a GUI)

Building software for end users is a pain in the ass because you have to cover all possible exceptions, errors, clean user input, etc. Great thing about being a coder is that you can eliminate like 70% of the development time by just doing it the "hakish" way where it works just fine only it doesn't have a GUI and is not user friendly.

Why not try with imacros and sick submitter..??

My 0.02 c

P.S. There is a guy who has done a few in iMacros.. Here is the thread where he is sharing it..

http://www.blackhatworld.com/blackha...on-script.html

iMacros is a cheap, easy way to build automation scripts.

Thanks and Regards

Well, let me tell you what I think.

Make your self well first with the things that can
be easily done (zennoposter) once you are stable making
$8k-10k a month, go ahead and outsource your tasks and
give time to learn coding. And then get back to your biz

That's me though

Originally Posted by shoaibahmad9999

Well, let me tell you what I think.

Make your self well first with the things that can
be easily done (zennoposter) once you are stable making
$8k-10k a month, go ahead and outsource your tasks and
give time to learn coding. And then get back to your biz

That's me though

Yep, you're right.

Actually, I already have 2 assistants for outsourcing - one for spinning and one for link building.

It's just that I wanted to save the time for web 2.0 blogs, to more High PR blog commenting. Further, automation is very good, when you have a lot of domains. So, each of them get's a little bit of power, without putting to much work on it day-in-and-day-out.

Nevertheless, time to learn Python, the O'Reilly book is on it's way :P

I'd be up for contributing to a zennoposter template as long as it to posts to blogs not found on senuke

Actually i did the registering the part using imacros and it worked perfectly.

Will post the ZP version once i buy ZP .


This is a great idea to get everything automated.

count me in.

Count me in. Will pay for templates for web 2.0's that are not already in SENuke.

hey buddy any update

Here's a good start with programming in Python

http://www.youtube.com/watch?v=k6U-i...4720A6F225E074

Originally Posted by madoctopus

Expert

1. Use curl_multi_* in optimal mode - constant bandwidth utilization / constant number of parallel requests. In other words, once a requests completes, start a new one immediately. There is absolutely no tutorial or documentation on how to do this. Requires days of trial and error and putting together what you read.

No documentation? Directly from the manual...
http://www.php.net/manual/en/functio...exec.php#87492

Originally Posted by jazzc

No documentation? Directly from the manual...
http://www.php.net/manual/en/functio...exec.php#87492

Well there's the aproach similar to what manixrock-~-gmail-~-com shows in the comments on that page.

However it is somewhat buggy and implementing it optimally for a dynamic system is not that easy.

Basically the idea is to have a callback/observer function that executes when a requests completes. There you do what you want with the response. After you run that, you figure out if you have more URLs/requests to make and if you do, you add the next one to the curl_multi handle. And so on...

The idea is, you do NOT know in advance the next URL you have to request and/or you have many URLs (e.g. 100mil) and can't put them all to run at once.

In practice I personally ran into some problems with how curl_multi works. Ideally I would have used threads but PHP doesn't support threads.

Originally Posted by madoctopus

Basically the idea is to have a callback/observer function that executes when a requests completes.

Your approach is more elegant.

The idea is, you do NOT know in advance the next URL you have to request and/or you have many URLs (e.g. 100mil) and can't put them all to run at once.

I supposed the callbacks from curl don't stack and cause a mess? That 's a big problem. This is why I prefer the KISS principle. His method is not elegant at all, he is brute-forcing, but is very straight forward and with no thread-safety type of issues.

Ideally I would have used threads but PHP doesn't support threads.

Gearman is our friend I don't do curl_multi anymore - much easier to scale.

@jazzc

I looked into Gearman. I even built my custom process management and did it in a multi-process way. The problem in this case is you end up using massive amounts of RAM. For example I built this multiprocess crawler and i spawned 150 processes. That eat up about 2GB RAM considering it was a very slim script. CPU load was initially the bottleneck, but after I optimized my language detection function I went from a load of 2 to 0.2 on a 2-core. However I was still using 2GB RAM. In order to max out usage on a 100mbps dedicated line I would need to spawn about 3 times as much, increasing server cost. Personally I prefer to optimize my scripts than pay more for hardware. Over time it proves to be a better design decision than upgrading hardware.

If the particular script would be more CPU or I/O intensive, then I wouldn't bother trying to over optimize RAM consumption. However, think about this situation. Let's say I want to do a proxy tester. I have let's say 500k proxies in the table and I want to test them at a rate of 400 in parallel. The bandwidth used is quite slim since I am only returning about 1KB of data per request. However, since I would be running 400 processes it would eat up a lot of RAM.

Certainly there are trade-offs. But it depends on what you are building in the sense of how much control you need over the parallelization process and how much you want to scale out.

Indeed, implementing a proxy checker with Gearman can be inefficient. But here 's an idea that I think will be very efficient

• Create a small ram filesystem and mount it
• Create minimalistic workers using the C API that simply launch the curl binary with the appropriate parameters and have it save the output to the above filesystem
• Process the output in the files from PHP

It 's not a purist approach, but it 's simple and effective.

I'm not saying the point-n-click part of ZP is buggy but the PHP API interfacing (the code generation or whatever is called)

I had the chance to work with ZP's CodeCreator for a while. I was writing C#, not PHP, though. Didn't encounter many bugs, although I found some of their macros a bit slow/unreliable. I also had to write a custom wait function because their WaitDownloading() doesn't seem to work with ajax. Other than that, it was pretty fun and easy to work with. Zenno's browser component can perform just about anything you can think of.

Originally Posted by jazzc

Certainly there are trade-offs. But it depends on what you are building in the sense of how much control you need over the parallelization process and how much you want to scale out.


Indeed, implementing a proxy checker with Gearman can be inefficient. But here 's an idea that I think will be very efficient

• Create a small ram filesystem and mount it
• Create minimalistic workers using the C API that simply launch the curl binary with the appropriate parameters and have it save the output to the above filesystem
• Process the output in the files from PHP

It 's not a purist approach, but it 's simple and effective.

Nice idea jazzc, but i dont think PHP is good enough for a good proxy checker. I made a proxy checker myself a while back and got good speeds with the .Net IAsyncResult (a polling implementation) and the beautiful new .Net4 Concurrent classes. It was nice and fast.

However, for a really fast proxy checker, you would have to create an NDIS driver (sorry no linux here). This would kill even nmap for speed.

ANYWAY people on another note, here is a similar community 'automation' project that I stumbled accross the other day. http://splatforms.com

I'm interested in this, if this actually is going to happen can the op pm me please

count me in

I'm in if it's still available.

Originally Posted by ~divinci

ANYWAY people on another note, here is a similar community 'automation' project that I stumbled accross the other day. http://splatforms.com

Thank you, I wasn't aware of this one

Originally Posted by madoctopus

Here's a good start with programming in Python

unreal . . i just read this entire thread and thought to myself, how uncanny, i got fed up a couple weeks ago and decided to get more proactive in learning programming, and i so i went on over to MIT world to look for some structured learning on cs, and found a gem. I was going to post it here, but you beat me to the punch, however. .you should post the entire course. Its completely free. . . and funny enough (while your not really learning a language per se, you are getting a bit of a start with python). For anyone who wants to get a real education on the basics of programming and thinking like a programmer:

http://ocw.mit.edu/courses/electrica...ing-fall-2008/

It everything full video lectures, syllabus, readings, etc. . . all 100% free from the best learning institution in these areas in the US.

I would like to contribute to the group buy thanks

This is a great thread! Was a group ever formed?

Running still? I saw some posts that this was closed, but OP you're up still?

albaniax - you missed one thing. Templates need to be updated weekly because Blogs sites changes often so group have to hire coder for that project. 1000-1200$ monthly and you can hire good coder from midde europe that will work 8 hours a day for such project.

50 guys in project - 1200/50 = 24$ monthly
100 guys in project - 12$ monthly.

! Forget about one time payment ! or it will be useless.

wordpress.com - hard to realize - $23.

Are you kidding? Wordpress com is piece of cake. If somone is telling that template for wordpress com is difficult he must be total nob. Anyway what tempate will you get for 23$? I made myself few templates but my templates have integration with other tools which automaticly make backlinks for blogs.

In result my zenno template make linkwhell, second tier, and then blast 1 and 2 tier with backlinks. You will not get such templates for 23$ per site - impossible.

P.S. Dont pm me with questions how much for my template - i dont want to sell it.