1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

question about proxies, threads and multiple browsers using C#

Discussion in 'C, C++, C#' started by j0b0123, Jan 4, 2011.

  1. j0b0123

    j0b0123 Regular Member

    Joined:
    Oct 30, 2009
    Messages:
    262
    Likes Received:
    218
    Occupation:
    professional trader - stocks, forex, futures
    Location:
    Las Vegas, USA
    Home Page:
    Hi guys,

    I am working on an application in C#, but I need to open maybe 10-15 or even 20 browser windows at the same time (each its own thread), each using its own proxy. As far as my research goes, I cannot tell how this is possible. I am not doing the coding myself, I have some help. From what I can tell, changing the proxy happens in the registry, but you cannot have more than 1 active at any one time.

    I know that scrapebox and others open hundreds of threads, each with their own proxy. What is the best method to accomplish this type of action, or can you point me to a resource that I can learn from?



    Thanks for any help here
     
  2. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Check the classes HttpWebRequest & HttpWebResponse ;)

    P.S.: HttpWebRequest.Proxy
    altough it doesn't support SOCKS4/5 :(
     
  3. heedcase

    heedcase Registered Member

    Joined:
    Aug 31, 2009
    Messages:
    71
    Likes Received:
    5
    you cannot do this with the browser control.. as stallion says above you must use the http request classes and set the proxy on the object itself before posting / getting any request on the connection.
     
  4. abhi007

    abhi007 Jr. VIP Jr. VIP

    Joined:
    Aug 31, 2010
    Messages:
    5,300
    Likes Received:
    3,740
    Location:
    snip.li/TubH
    yes mate i agree with stallion you need to be more creative
     
  5. crepito

    crepito Junior Member

    Joined:
    Oct 5, 2008
    Messages:
    145
    Likes Received:
    28
    Location:
    Portugal
    Plus, if you render all the pages, your app will be slow as hell...

    HttpWebRequest & HttpWebResponse is the way to go if you don't want to get your hands dirty with Sockets directly :p
     
  6. smack

    smack Junior Member

    Joined:
    Feb 1, 2010
    Messages:
    182
    Likes Received:
    78
    Occupation:
    Software Engineer/Evil Genius
    Location:
    inside .NET
    every time you use a WebBrowser control, God kills a kitten.

    that's not hyperbole, that's fact.

    as previously mentioned HTTPWebRequest is a great starting point. it's a good class for those new to web interactions in the .NET framework. it exposes a good bit of functionality including proxies.

    there are some quirks to it though. it isn't always the best at handling cookies, and sometimes SSL requests take a bit of kung fu to get right.

    there is also the System.Net.Sockets namespace. that is pretty much the bees knees as far as functionality goes, but it requires a good bit of work to write your own HTTP class as a wrapper for it. very much worth the time and effort if you are planning for the future.

    you also have the option of using 3rd party libraries such as ChillKat or similar. but why buy something when you can grow you own? also i prefer not to use 3rd party tools whenever possible. 9/10 times i can create something far better, even if it takes more time, and then you have the added benefit of advancing your own knowledge which is always a good thing.

    as far as multiple threads and different proxies with the WebBrowser control, i don't think this is a possibility. perhaps multiple threads may be possible if you created the WB classes in your code as opposed to in the UI (i've never tried this to confirm, just theorizing), but it is my understanding that since the WB control relies on the same proxy functionality as IE does (registry key) it is not really a possibility to use a different proxy for each instance. perhaps if you have another product that managed your requests, i think there used to be something called Multi-Proxy that operated similar to this, but it's not really a very practical method of deployment.

    my advice is to start with HTTPWebRequest, knowing that it has limitations as well albeit far fewer than the WB control, and then as you progress dedicate yourself to learning the System.Net.Sockets namespace. that knowledge will pay dividends for a long time to come with its power and flexibility.
     
    • Thanks Thanks x 1
  7. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    Smack I see you mentioned buggy cookie functionality in HttpWebrequest. You find any fix for this? Even applying all the hacks I found on the internet, I cannot log into a wordpress site. In the end I had to write my own cookie parser. Looking for something easier than that... lol.

    Thanks,

    -blaka
     
  8. smack

    smack Junior Member

    Joined:
    Feb 1, 2010
    Messages:
    182
    Likes Received:
    78
    Occupation:
    Software Engineer/Evil Genius
    Location:
    inside .NET
    from what i remember, and it's been a while since i have used httpwebrequest, the issues i noticed were mostly related to domain/subdomain/path parsing. so study packet logs of your requests very closely and look for which cookies are not being sent back with the request.

    you may have to explicitly add them to the request, or alter their domain accordingly.
     
  9. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    Yep, I figured that issue out. However, there's a couple of other issues, including one where HttpWebrequest is incorrectly parsing the cookies in the response. So you have to do it yourself. Very ugly =/

    The entire class has so many bugs I wonder if they even tested it before release.
     
  10. smack

    smack Junior Member

    Joined:
    Feb 1, 2010
    Messages:
    182
    Likes Received:
    78
    Occupation:
    Software Engineer/Evil Genius
    Location:
    inside .NET
    that's a lot worse than i remember it (granted it's been a while).

    it's one of those things where it gives you a lot more than a WB control, but has it's own issues. the only ultimate solution is System.Net.Sockets

    one thing you can try is making a wrapper class for your HTTPWebRequest. since you're manually parsing the cookies anyways if you abstract that out a little, and use a CookieCollection instead of CookieContainer, you might be able to put together a good working stop gap solution.

    it has always baffled me that the Sockets namespace is so good, but HTTPWebRequest is so bad (and WB control even worse).

    what kinds of parsing issues are you having specifically?
     
  11. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    When I try to log into a wordpress blog, HttpWebRequest parses the cookies completely wrong, so you have to do it on your own. And yes, I've implemented a hack to fix the subdomain issue and tested to make sure it works, so I'm pretty sure that's not it.

    -blaka
     
  12. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,059
    Thats because you need to use the CookieContainer object along with your requests.
     
  13. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    Reread my posts. I am using CookieContainer object. CookieContainer object is bugged. It doesn't handle subdomains correctly and it doesn't parse cookies correctly on various webpages. Don't believe me? Google 'CookieContainer bug'.
     
  14. kaidoristm

    kaidoristm Power Member

    Joined:
    Feb 13, 2009
    Messages:
    561
    Likes Received:
    726
    Occupation:
    Freelancer
    Location:
    Estonia
    Home Page:
    Well HttpWebRequest isnt best for bots. And why because it doesn't support javascript.
    For example if your trying to grab recaptcha image with it your stuck.
    Which is easy job with webbrowser and dom.
    Well the best solution which i can think of is pac files. Never tested but it can be only solution for webbrowser. And yes you can multithread webbrowser so that's not problem.
     
    • Thanks Thanks x 1
  15. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Well i managed to get a recaptcha image and solve it via HttpWebRequest.
     
  16. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    Same. Is it supposed to be hard :confused:
     
  17. smack

    smack Junior Member

    Joined:
    Feb 1, 2010
    Messages:
    182
    Likes Received:
    78
    Occupation:
    Software Engineer/Evil Genius
    Location:
    inside .NET
    if you know what you're doing, it's not hard at all.

    the best solution is making a real HTTP class from System.Net.Sockets, and not having to deal with the crap WebBrowser/HTTPWebRequest
     
  18. J0kerz

    J0kerz Supreme Member

    Joined:
    Nov 2, 2009
    Messages:
    1,413
    Likes Received:
    435
    Occupation:
    IM
    Location:
    There
    Just use the Curl library, it will work just fine.
     
  19. Monrox

    Monrox Power Member

    Joined:
    Apr 9, 2010
    Messages:
    615
    Likes Received:
    579
    That still won't solve (to my knowledge) the javascript requirements if some are set to combat spam e.g. a script must be executed for the comment not to be automatically discarded. That's why slow poster is so 'slow' - it loads the page.

    Such websites are valuable though since they won't be spammed by normal bots.

    It's about time MS implements an IE wrapper capable of independent proxifying.
     
  20. kaidoristm

    kaidoristm Power Member

    Joined:
    Feb 13, 2009
    Messages:
    561
    Likes Received:
    726
    Occupation:
    Freelancer
    Location:
    Estonia
    Home Page:
    Yes you can if that specific site still uses noscript option otherwise your fucked.

    Your absolutely right + for sockets from me as well :)