[Case Study] Ubot Studio !

[Gary Becks]

I don't mind strongly typed languages like C# and so on. I forget why I hated Object Pascal (was ages ago when I played around with Delphi) but I just remember I hated the syntax.

Delphi is strongly typed as well. If you are a C# guy than you most likely just hated the verboseness of it.

 

[Gary Becks]

Got the first ten sites coded and working pretty easily. Haven't added threading to it as of yet though. Will most likely save that task for last.

Question - as I said the bot is free to use/test while it is being created. I can either share it while sites are being add to it or wait until I have a full build ready - thoughts ?

 

[BlackSEOWizard]

Got the first ten sites coded and working pretty easily. Haven't added threading to it as of yet though. Will most likely save that task for last.

Question - as I said the bot is free to use/test while it is being created. I can either share it while sites are being add to it or wait until I have a full build ready - thoughts ?

Enjoy writing the threading . It is a blast!

At least it works now; somewhat. For a while it didn't. Still very piggy though and after 10-20 threads you get memory exhaustion messages even with 20gb of ram.

 

[Gary Becks]

Enjoy writing the threading . It is a blast!

At least it works now; somewhat. For a while it didn't. Still very piggy though and after 10-20 threads you get memory exhaustion messages even with 20gb of ram.

You can optimize that a bit by not using global variables where not applicable in your script.

 

[BlackSEOWizard]

You can optimize that a bit by not using global variables where not applicable in your script.

Not really the problem is the lack of sockets and the high resources of two chrome browsers (20mb+) per thread.

 

[Gary Becks]

I thought Ubot used IE not Chrome ?

 

[BlackSEOWizard]

I thought Ubot used IE not Chrome ?

4.0 uses Chrome via a third party library called Awesomium. It's basically a wrapper for Chromium. It has it's own issues and a good cause of a lot of the lockups in Ubot. But the lack of sockets (which sound like they will not be added) will always be a limiting factor and why ubot programs struggle at 5-20 threads where as other programs you can break 200 threads.

 

[Gary Becks]

4.0 uses Chrome via a third party library called Awesomium. It's basically a wrapper for Chromium. It has it's own issues and a good cause of a lot of the lockups in Ubot. But the lack of sockets (which sound like they will not be added) will always be a limiting factor and why ubot programs struggle at 5-20 threads where as other programs you can break 200 threads.

Interesting.

I am guessing that for single threaded apps chrome must be faster than IE which may be the reason they chose to switch to it.

 

[BlackSEOWizard]

Interesting.

I am guessing that for single threaded apps chrome must be faster than IE which may be the reason they chose to switch to it.

The main reason they switched is the issues with IE in general and removing all requirements of anything on the local machine. Which is a huge help, but awesomium has a lot of issues but still better IE in many cases.

 

[Gary Becks]

I really don't see how that can be considered a huge help.

Your required to have the .Net framework so expecting to have that and not IE is pretty senseless IMHO.

 

[BlackSEOWizard]

I really don't see how that can be considered a huge help.

Your required to have the .Net framework so expecting to have that and not IE is pretty senseless IMHO.

True, but not completely.
The problem is what version of IE? Each version of IE were dramatically different and each had their own problems. Packaging bots for end users, you can't guarantee what version they were running and some versions caused more problems than others. Also IE has issues with many sites and processes.

 

[Gary Becks]

That makes sense.

I guess I will make the max thread count on the bot 20 threads then.

 

[BlackSEOWizard]

That makes sense.

I guess I will make the max thread count on the bot 20 threads then.

For a while anything more than 5 threads would cause issues, but it seems to be better. I haven't tested the max and it highly depends on what each thread is doing, but you should be able to get to 20 threads, maybe 50 not quite sure as I haven't really tested it much. The problem is even with 5 threads you can get memory exhaustion errors even with 20GB of ram and a relatively small bot. Some bug that causes it I suspect.

I'd allow them to do as many threads as they want, or cap it at 50 and just put a note that more than 20 threads needs a really powerful machine/connection. I hate being told how many of anything I can do as my machine is very powerful as are my server farms. I may want to run things differently than other people. Flexibility is good.

It's worth testing though.

 

[Gary Becks]

Cool.

I think I will make the max 30 threads. There really is no use for more than 30 threads seeing as how we are only registering to 100 sites max at a time.

My machine has 12gb of ram so I'd suspect if it causes problems on my machine it will cause problems on just about all of them.

Thanks for your insight on this.

On another note, I still haven't had any problems with the size of the bot causing issues so I haven't asked about the libraries as of yet. I guess I'll just cross that bridge if/when I get to it.

 

[*zap*]

For scraping data from a page i have used upto 35 threads
For registering accounts (db size 3500) i use 20 threads and the same for posting
And i have just 4gb ram

But its only been the passed month or so that the awesomium browser has stopped crashing

Also so if you are using proxys the standed way of switching them somes to make it drop threads (well for me anyway) you have to make a work around for it

 

[Gary Becks]

For scraping data from a page i have used upto 35 threads
For registering accounts (db size 3500) i use 20 threads and the same for posting
And i have just 4gb ram

But its only been the passed month or so that the awesomium browser has stopped crashing

Also so if you are using proxys the standed way of switching them somes to make it drop threads (well for me anyway) you have to make a work around for it

Cool stuff. Sounds like 30 threads would still be optimal for only 100 sites still.

I don't think proxy support will be necessary unless requested.

 

[Gary Becks]

So I revised the Web 2.0 List by deleting all of the sucky sites last night and also adding some good ones that were missing from the list.

Looks like we will still end up with around 100 or so sites to use.

30 sites down 70 to go. !

 

[ccshack]

Hi Gary!

Thanks to kept us informed about your progress!

 

[gregstereo]

Been following this thread, very interesting indeed. +rep. I'm looking forward to reading the rest of the story as it unfolds.

 

[Gary Becks]

Thanks guys.

It's definitely proving to be pretty fun testing what the current limitations of Ubot are/are not.

So far so good though, only thing I am worried about is the bot growing to big to compile decently.

We'll see how it goes though.