1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Build Bots with Proxy and Multi-threading?

Discussion in 'General Programming Chat' started by cashlo, Jan 8, 2011.

  1. cashlo

    cashlo Newbie

    Joined:
    Jan 17, 2010
    Messages:
    22
    Likes Received:
    3
    I have been using Selenium+Java for building my web bots, but I find it lack the support for Proxy and Multi-threading to scale up my bots. I know PHP but i don't really want to touch those HTTP header thing.

    I know these have been asked many times, and I have been searching for some time now, but still can't find a solution :(
     
  2. haridada

    haridada Senior Member

    Joined:
    Oct 9, 2008
    Messages:
    996
    Likes Received:
    382
    Location:
    Chennai
    VB.net / C# is your best bet i think. Hope the better bot programmers chime in
     
  3. doyley

    doyley Registered Member

    Joined:
    May 3, 2010
    Messages:
    72
    Likes Received:
    24
    I thought Java supported threads and proxies in its api's? could you not make a proxy package and import it into your class?
    Code:
    //Set the http proxy to webcache.mydomain.com:8080
    
    System.setProperty("http.proxyHost", "webcache.mydomain.com");
    System.setPropery("http.proxyPort", "8080");
    
    // Next connection will be through proxy.
    URL url = new URL("http://java.sun.com/");
    InputStream in = url.openStream();
    
    You could maybe add support for loading for a database or parsing from a external proxy site, but this would load bad unchecked proxies!
     
  4. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    C# HttpWebRequest supports multithreading + proxies but only http/https, no socks4/5...
     
  5. necr0n0mic0n

    necr0n0mic0n Newbie

    Joined:
    Jun 27, 2009
    Messages:
    33
    Likes Received:
    9
    use delphi, for winsock comms you can use indy for the start
     
  6. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    OP, look into C# with WatiN. Shouldn't be that hard to grasp after Java and Selenium ;)
     
  7. saxgod

    saxgod Regular Member

    Joined:
    Sep 19, 2010
    Messages:
    351
    Likes Received:
    337
    if you would use PHP just use cURL to browse the web, you can provide a proxy with every cURL request, thus also not giving any problems when going multi-thread since the settings are only valid for the current cURL object.
    I use this in combination with TOR
     
  8. jason-l

    jason-l Regular Member

    Joined:
    Feb 21, 2011
    Messages:
    281
    Likes Received:
    83
    java easily supports multi threading.. always has
     
  9. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Wrong section...
    /EDIT: and stop spamming the whole fucking forum...

    Someone earned himself a ban ;)
     
  10. Saulyx

    Saulyx Junior Member

    Joined:
    Jan 10, 2010
    Messages:
    107
    Likes Received:
    5
    PHP doesnt support multi threading, I've been using php and curl however Im now moving away from it and looking for better solution as multi threading is needed badly.

    Any suggestions for building bots on UNIX systems(linux)?
     
  11. reinrein

    reinrein Regular Member

    Joined:
    Feb 8, 2008
    Messages:
    443
    Likes Received:
    343
    Home Page:
    And php compiles at runtime which makes it more slow. You can use perl for unix.
     
  12. gnote

    gnote Registered Member

    Joined:
    Mar 10, 2009
    Messages:
    80
    Likes Received:
    6
    Occupation:
    Programmer
    Location:
    USA
    I agree here.. I have created some serious shit with C#

    I have been toying with the idea of learning another language to run on Unix. Would you say Perl is the best one to learn? I have heard good things about Ruby, would anyone recommend that?
     
  13. Saulyx

    Saulyx Junior Member

    Joined:
    Jan 10, 2010
    Messages:
    107
    Likes Received:
    5
    isnt ruby more for web projects? e.g. social networks, etc. Anyone could feedback on perl?
     
  14. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Would be interested in perl too.
    Is there a class for Http Requests with Http/Socks proxy support?
     
  15. sweepypanda

    sweepypanda Newbie

    Joined:
    Jul 27, 2010
    Messages:
    20
    Likes Received:
    4
    No need for a class. Just use the HttpWebRequest.Proxy property.

    Set it like this:
    Code:
    httpRequest.Proxy = new WebProxy("255.255.255.255", 3123); //WebProxy(string Host, int Port)
     
  16. reinrein

    reinrein Regular Member

    Joined:
    Feb 8, 2008
    Messages:
    443
    Likes Received:
    343
    Home Page:
    It only supports http proxies though, no socks proxy. + lots of bugs when you do https requests(port 443) with httpwebrequest using proxy.
     
  17. supparman

    supparman Newbie

    Joined:
    Jul 22, 2009
    Messages:
    28
    Likes Received:
    5
    java:

    proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("host", port));
    conn = (HttpURLConnection) url.openConnection(proxy);

    for socks use Proxy.Type.SOCKS
     
  18. goodbuyer

    goodbuyer Junior Member

    Joined:
    Aug 13, 2010
    Messages:
    118
    Likes Received:
    18
    Could you please post how to manage to get these things working with Tor! that would be great!
     
  19. wes_one

    wes_one Junior Member

    Joined:
    Apr 25, 2011
    Messages:
    176
    Likes Received:
    75
    Java is like second best multithreading langauge dude.
    I think delphi is first.
     
  20. The Doctor

    The Doctor Jr. VIP Jr. VIP

    Joined:
    Dec 18, 2010
    Messages:
    612
    Likes Received:
    188
    Occupation:
    Computer Scientist, Engineer, Programmer.
    Location:
    ☆☆☆☆☆☆
    i highly doubt that delphi is the first best.. of anything.