Build Bots with Proxy and Multi-threading?

C

cashlo

Newbie

Jan 17, 2010

22

3

• Jan 8, 2011

• #1

I have been using Selenium+Java for building my web bots, but I find it lack the support for Proxy and Multi-threading to scale up my bots. I know PHP but i don't really want to touch those HTTP header thing.

I know these have been asked many times, and I have been searching for some time now, but still can't find a solution

 

haridada

Senior Member

Oct 9, 2008

1,000

385

• Jan 8, 2011

• #2

VB.net / C# is your best bet i think. Hope the better bot programmers chime in

 

D

doyley

Registered Member

May 3, 2010

72

24

• Jan 8, 2011

• #3

I thought Java supported threads and proxies in its api's? could you not make a proxy package and import it into your class?

Code:

//Set the http proxy to webcache.mydomain.com:8080

System.setProperty("http.proxyHost", "webcache.mydomain.com");
System.setPropery("http.proxyPort", "8080");

// Next connection will be through proxy.
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();

You could maybe add support for loading for a database or parsing from a external proxy site, but this would load bad unchecked proxies!

 

Stalli0n

Junior Member

Nov 17, 2010

147

100

• Jan 8, 2011

• #4

C# HttpWebRequest supports multithreading + proxies but only http/https, no socks4/5...

 

N

necr0n0mic0n

Newbie

Jun 27, 2009

33

9

• Jan 10, 2011

• #5

use delphi, for winsock comms you can use indy for the start

 

B

blakamia

Junior Member

Jan 25, 2010

162

353

• Jan 10, 2011

• #6

OP, look into C# with WatiN. Shouldn't be that hard to grasp after Java and Selenium

 

ThePacketeer

Regular Member

Sep 19, 2010

373

354

• Jan 10, 2011

• #7

if you would use PHP just use cURL to browse the web, you can provide a proxy with every cURL request, thus also not giving any problems when going multi-thread since the settings are only valid for the current cURL object.
I use this in combination with TOR

 

J

jason-l

Regular Member

Feb 21, 2011

281

83

• Apr 1, 2011

• #8

java easily supports multi threading.. always has

 

Stalli0n

Junior Member

Nov 17, 2010

147

100

• Apr 1, 2011

• #9

Wrong section...
/EDIT: and stop spamming the whole fucking forum...

Someone earned himself a ban

 

S

Saulyx

Junior Member

Jan 10, 2010

107

5

• Apr 6, 2011

• #10

PHP doesnt support multi threading, I've been using php and curl however Im now moving away from it and looking for better solution as multi threading is needed badly.

Any suggestions for building bots on UNIX systems(linux)?

 

reinrein

Regular Member

Feb 8, 2008

442

346

• Apr 6, 2011

• #11

Saulyx said:

PHP doesnt support multi threading, I've been using php and curl however Im now moving away from it and looking for better solution as multi threading is needed badly.

Any suggestions for building bots on UNIX systems(linux)?

Click to expand...

And php compiles at runtime which makes it more slow. You can use perl for unix.

 

G

gnote

Registered Member

Mar 10, 2009

80

6

• Apr 6, 2011

• #12

haridada said:

VB.net / C# is your best bet i think. Hope the better bot programmers chime in

Click to expand...

I agree here.. I have created some serious shit with C#

reinrein said:

And php compiles at runtime which makes it more slow. You can use perl for unix.

Click to expand...

I have been toying with the idea of learning another language to run on Unix. Would you say Perl is the best one to learn? I have heard good things about Ruby, would anyone recommend that?

 

S

Saulyx

Junior Member

Jan 10, 2010

107

5

• Apr 6, 2011

• #13

isnt ruby more for web projects? e.g. social networks, etc. Anyone could feedback on perl?

 

Stalli0n

Junior Member

Nov 17, 2010

147

100

• Apr 6, 2011

• #14

Would be interested in perl too.
Is there a class for Http Requests with Http/Socks proxy support?

 

S

sweepypanda

Newbie

Jul 27, 2010

20

4

• Apr 18, 2011

• #15

Stalli0n said:

Would be interested in perl too.
Is there a class for Http Requests with Http/Socks proxy support?

Click to expand...

No need for a class. Just use the HttpWebRequest.Proxy property.

Set it like this:

Code:

httpRequest.Proxy = new WebProxy("255.255.255.255", 3123); //WebProxy(string Host, int Port)

 

reinrein

Regular Member

Feb 8, 2008

442

346

• Apr 18, 2011

• #16

sweepypanda said:

No need for a class. Just use the HttpWebRequest.Proxy property.

Set it like this:

Code:

httpRequest.Proxy = new WebProxy("255.255.255.255", 3123); //WebProxy(string Host, int Port)

Click to expand...

It only supports http proxies though, no socks proxy. + lots of bugs when you do https requests(port 443) with httpwebrequest using proxy.

 

S

supparman

Newbie

Jul 22, 2009

28

5

• Apr 18, 2011

• #17

java:

proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("host", port));
conn = (HttpURLConnection) url.openConnection(proxy);

for socks use Proxy.Type.SOCKS

 

G

goodbuyer

Junior Member

Aug 13, 2010

118

18

• Apr 20, 2011

• #18

saxgod said:

if you would use PHP just use cURL to browse the web, you can provide a proxy with every cURL request, thus also not giving any problems when going multi-thread since the settings are only valid for the current cURL object.
I use this in combination with TOR

Click to expand...

Could you please post how to manage to get these things working with Tor! that would be great!

 

W

wes_one

Junior Member

Apr 25, 2011

177

78

• Jul 23, 2011

• #19

Java is like second best multithreading langauge dude.
I think delphi is first.

 

The Doctor

Senior Member

Dec 18, 2010

1,110

474

• Jul 23, 2011

• #20

wes_one said:

Java is like second best multithreading langauge dude.
I think delphi is first.

Click to expand...

i highly doubt that delphi is the first best.. of anything.