1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[C# Discussion] WebRequest or WebBrowser for web automation?

Discussion in 'C, C++, C#' started by vebxperts, Oct 17, 2012.

  1. vebxperts

    vebxperts Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 15, 2008
    Messages:
    1,657
    Likes Received:
    339
    Question: What is recommended? And why?
    System.Net (HttpWebRequest/WebRequest/WebClient)
    OR
    WebBrowser control

    I know System.Net is much faster than WebBrowser, then what is the purpose of WebBrowser for web automation? Like filling and submitting a form online, simulating some mouse clicking, because sometime it happens that you can't proceed without 'a mouse click' on a field because, form generates some kind of random variable or whatever from that, which is than used in that specific session.

    In such cases, is there any possible solution in System.Net?

    Then, sometime it happens that some websites generate random ID/Name/Value etc for form fields every time when form is refreshed, can we trigger such fields in System.Net?

    Once again! This is C# discussion please.
     
  2. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    When it comes to javascript protected forms (ajax) it can be tricky to make submissions with webrequests and for that browser is a lot easier. however everything is possible to emulate with webrequests aswell but you have to anaylze the code on the page to simulate all requests to the server. There is no "simple" solution to this when it comes to webrequests.
     
  3. adh0c

    adh0c Newbie

    Joined:
    Apr 20, 2012
    Messages:
    11
    Likes Received:
    7
    HttpWebRequest inherits from WebRequest and was designed for Http requests. WebRequest is pretty much the base class for all web requests.. WebClient is basically a 'streamlined' version of HttpWebRequest to make certain things easier. HttpWebRequest is going to give you more flexibility though and typically what you want to use.

    WebBrowser is no good but it's main use is emulation/parsing the Java. You can try Awesomium, it's a headless browser that you can render but not forced too. It can emulate hardware, parse Java, and is much faster than the WebBrowser control.

    Ideally I would attempt to process in HttpWebRequest and if it fails, or recognizes it needs to parse Java to succeed, it then fires up the Browser and gives it another go with emulations.
     
  4. vebxperts

    vebxperts Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 15, 2008
    Messages:
    1,657
    Likes Received:
    339
    @HatIsBlack:
    So, you mean to say that even (ajax) can be handled with webrequest. Can you guide what is the best software for tracking/sniffing requests/resposes to/from server? (The one which has complete tutorial as well) As I know of Wireshark, Fiddler but if there is anything better than these two?

    @adh0c:
    Awesomium is free? Can it be embedded just like webbrowser in .Net C# program in VS 2010?
    According to you, java is difficult to handle in httpwebrequest and when it can't be handled then you'll go for browser solution and for browser solution, you prefer awesomium? Correct?

    Thanks
     
    Last edited: Oct 17, 2012
  5. DarkPixel

    DarkPixel Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 4, 2011
    Messages:
    1,328
    Likes Received:
    1,239
    Location:
    ↓↓↓↓
    Home Page:
    Webrequest can do anything, I don't think there is something webrequest can't accomplish. And I agree awesomium is perfect, but uses a lot of ram and cpu, so if it's not necessary to use awesomium, go with the native IE browser.
     
  6. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274
    use HttpWebRequest if you want your software to fly and WebBrowser if you want it to walk like a 95 yo granny ;) As HatIsBlack mentioned, WebBrowser is much easier in some cases - I used it when I had to login into a few hundred AOL accounts and change their settings; figuring out how to do just that with HttpWebRequest could be quite unnerving and take you hours
     
  7. vebxperts

    vebxperts Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 15, 2008
    Messages:
    1,657
    Likes Received:
    339
    @DarkPixel:
    Any recommended book/tutorials/software for programing in C# with webrequest for automation stuff.

    @hpv222:
    Same thing, what do you suggest. As everyone around is saying webrequest (http) is better than webbrowser, I know and agree with you guys. But, how to?

    Like I am getting error when I try to just go to youtube,com usign httpwebrequest with a private proxy.
    Have searched a lot on google and forums but no solution. (As I have not done much work in System.net that's why don't know a lot)
    error: "The server committed a protocol violation, Section=ResponseStatusLine"
     
  8. qrazy

    qrazy Senior Member

    Joined:
    Mar 19, 2012
    Messages:
    1,111
    Likes Received:
    1,712
    Location:
    Banana Republic
    It depends on the site you target. If the site has lot of javascript or jquery, then it's easy to do with webbrowser. Though you can do it with webrequest it'll be tricky in case of jquery.
     
  9. adh0c

    adh0c Newbie

    Joined:
    Apr 20, 2012
    Messages:
    11
    Likes Received:
    7
    Yeah, Awesomium is free and preferred IMO. It gives you much more control. Works very similar to the web browser but doesn't have as much overhead as the browser control if used properly. IMO, it's much better for threading the "browser". If your not worried about threading, then the browser control works fine.

    For most websites though, HttpWebRequest will work fine. Just write a class that will parse the Form tags in the HTML and make your GET/POST requests accordingly.

    FIY,
    Regardless of the type of host; ajax, php, aspx, asmx, svc, html.. the HttpWebRequest can handle it. It simply breaks down to requesting another URL and parsing out variables 99% of the time. The only problems usually come about when the website uses javascript to generate code on DOM-events to identify "bots" Vs. more-than-likely humans. Most websites don't do this though, usually just really popular sites that get hit hard with spam/bots.

    Oh, and Fiddler2 is THE application for anything dealing with sniffing web requests.
     
  10. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,148
    As far as I know, Awesomium isn't thread-safe ;) You can have multiple "tabs" but they share the same cookies etc. Are you sure you 've been using it correctly?
     
  11. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,059
    Why do people feel the need to re-create the same threads over and over?

    In programming there is a rule called DRY - Don't Repeat Yourself, maybe this rule should be applied to posts in this forum.
     
  12. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,148
    I 'd have to delete a lot of threads then :)

    Apart from the joke, I think there 's a chance for new information to pop up every time a thread comes in. Not all people have seen the old threads and it 's a chance for them to contribute if they have something to share.
     
    • Thanks Thanks x 2
  13. xxf8xx

    xxf8xx Supreme Member

    Joined:
    Nov 30, 2009
    Messages:
    1,321
    Likes Received:
    596
    Occupation:
    IM
    I use WebRequests for all my botting purposes. All I do is sniff the traffic using Fiddler and copy/paste the headers it sent into a comment in my code and duplicate it. Don't forget to use automatic decompression if the page is encrypted. Really just make sure you are sending all of the same requests your browser does. You can see some examples if you go through a couple of my threads. All the bots I have released here are completely open source for reasons like this.

    Good luck!
     
    • Thanks Thanks x 2
  14. vebxperts

    vebxperts Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 15, 2008
    Messages:
    1,657
    Likes Received:
    339
    Inspirational post! Can you kindly add me on your Skype: vebxperts or if you can PM me yours. I think you won't mind helping me in your free time, regarding programing?
     
  15. Gary Becks

    Gary Becks Power Member

    Joined:
    Apr 11, 2010
    Messages:
    675
    Likes Received:
    282
    Location:
    Atl
    Home Page:
  16. vebxperts

    vebxperts Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 15, 2008
    Messages:
    1,657
    Likes Received:
    339
    Thank Gary, Got it and I am amazed to see that they have not talked anywhere in 630 pages about "how to use private password protected proxies, even simply how to use proxies".

    I don't think it's a complete book. Specially when you are making a Bot, proxy is the key feature that has to be discussed.
     
  17. funkseo

    funkseo Newbie

    Joined:
    Sep 27, 2012
    Messages:
    20
    Likes Received:
    1
    how can you post data with WebRequests?
    For example for blog commenting, how can you fill name,email,website,comment controls and post data?
     
  18. ikstob

    ikstob Junior Member

    Joined:
    Nov 12, 2012
    Messages:
    147
    Likes Received:
    129
    Location:
    ikstob.com
    Home Page:
    In short,

    WebRequest - is great for things you can easily automate, like RESTful API's etc. Where it starts to get more complicated is when websites use techniques like "cross-site request forgery" (CSRF) protection or fat JavaScript/AJAX applications. Pinterest use CSRF protection in all their web-forms.

    WebBrowser - basically this is a cut-down version of Internet Explorer, because its running a full browser stack it can be a lot slower as its loading Images, CSS, JavaScript and rendering the pages. Use it sparingly, where you cant just go direct with WebRequests.
     
  19. Wolfpack

    Wolfpack Junior Member

    Joined:
    Jul 13, 2011
    Messages:
    166
    Likes Received:
    346
    Don't use Web Browser for form automation! I've tried it a few times, and whilst you might think it's a headache using Web Request or HTTP Web Request, the performance difference makes it worth learning about these, especially if you want to automate it with multi-threading, which isn't so easy (not sure if it's possible) with web browsers. Use Web Browsers when there's a UI requirement for one, ie an application intended for public use including browser-based interaction.

    Get Fiddler2, learn how to use it and using Web Request it should be a piece of cake 90-99% of the time.
     
  20. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    343
    ^You can multithread WebBrowsers :) In some cases, it's better off to use WebBrowser if you're dealing with a lot of Javascript stuff. Otherwise, stick to HttpWebRequests.