Build Bots with Proxy and Multi-threading?

C

cashlo

Newbie
Joined
Jan 17, 2010
Messages
22
Reaction score
3
I have been using Selenium+Java for building my web bots, but I find it lack the support for Proxy and Multi-threading to scale up my bots. I know PHP but i don't really want to touch those HTTP header thing.

I know these have been asked many times, and I have been searching for some time now, but still can't find a solution :(
 
Advertise on BHW
VB.net / C# is your best bet i think. Hope the better bot programmers chime in
 
I thought Java supported threads and proxies in its api's? could you not make a proxy package and import it into your class?
Code: 
//Set the http proxy to webcache.mydomain.com:8080

System.setProperty("http.proxyHost", "webcache.mydomain.com");
System.setPropery("http.proxyPort", "8080");

// Next connection will be through proxy.
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();

You could maybe add support for loading for a database or parsing from a external proxy site, but this would load bad unchecked proxies!
 
C# HttpWebRequest supports multithreading + proxies but only http/https, no socks4/5...
 
OP, look into C# with WatiN. Shouldn't be that hard to grasp after Java and Selenium ;)
 
if you would use PHP just use cURL to browse the web, you can provide a proxy with every cURL request, thus also not giving any problems when going multi-thread since the settings are only valid for the current cURL object.
I use this in combination with TOR
 
java easily supports multi threading.. always has
 
Wrong section...
/EDIT: and stop spamming the whole fucking forum...

Someone earned himself a ban ;)
 
PHP doesnt support multi threading, I've been using php and curl however Im now moving away from it and looking for better solution as multi threading is needed badly.

Any suggestions for building bots on UNIX systems(linux)?
 
Saulyx said:
PHP doesnt support multi threading, I've been using php and curl however Im now moving away from it and looking for better solution as multi threading is needed badly.

Any suggestions for building bots on UNIX systems(linux)?
Click to expand...

And php compiles at runtime which makes it more slow. You can use perl for unix.
 
haridada said:
VB.net / C# is your best bet i think. Hope the better bot programmers chime in
Click to expand...

I agree here.. I have created some serious shit with C#

reinrein said:
And php compiles at runtime which makes it more slow. You can use perl for unix.
Click to expand...

I have been toying with the idea of learning another language to run on Unix. Would you say Perl is the best one to learn? I have heard good things about Ruby, would anyone recommend that?
 
isnt ruby more for web projects? e.g. social networks, etc. Anyone could feedback on perl?
 
Would be interested in perl too.
Is there a class for Http Requests with Http/Socks proxy support?
 
Stalli0n said:
Would be interested in perl too.
Is there a class for Http Requests with Http/Socks proxy support?
Click to expand...

No need for a class. Just use the HttpWebRequest.Proxy property.

Set it like this:
Code: 
httpRequest.Proxy = new WebProxy("255.255.255.255", 3123); //WebProxy(string Host, int Port)
 
sweepypanda said:
No need for a class. Just use the HttpWebRequest.Proxy property.

Set it like this:
Code: 
httpRequest.Proxy = new WebProxy("255.255.255.255", 3123); //WebProxy(string Host, int Port)
Click to expand...

It only supports http proxies though, no socks proxy. + lots of bugs when you do https requests(port 443) with httpwebrequest using proxy.
 
java:

proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("host", port));
conn = (HttpURLConnection) url.openConnection(proxy);

for socks use Proxy.Type.SOCKS
 
saxgod said:
if you would use PHP just use cURL to browse the web, you can provide a proxy with every cURL request, thus also not giving any problems when going multi-thread since the settings are only valid for the current cURL object.
I use this in combination with TOR
Click to expand...

Could you please post how to manage to get these things working with Tor! that would be great!
 
Java is like second best multithreading langauge dude.
I think delphi is first.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top