[C# Discussion] WebRequest or WebBrowser for web automation?

BP247

BP247

Elite Member
Joined
Nov 15, 2008
Messages
2,234
Reaction score
470
Question: What is recommended? And why?
System.Net (HttpWebRequest/WebRequest/WebClient)
OR
WebBrowser control

I know System.Net is much faster than WebBrowser, then what is the purpose of WebBrowser for web automation? Like filling and submitting a form online, simulating some mouse clicking, because sometime it happens that you can't proceed without 'a mouse click' on a field because, form generates some kind of random variable or whatever from that, which is than used in that specific session.

In such cases, is there any possible solution in System.Net?

Then, sometime it happens that some websites generate random ID/Name/Value etc for form fields every time when form is refreshed, can we trigger such fields in System.Net?

Once again! This is C# discussion please.
 
Advertise on BHW
When it comes to javascript protected forms (ajax) it can be tricky to make submissions with webrequests and for that browser is a lot easier. however everything is possible to emulate with webrequests aswell but you have to anaylze the code on the page to simulate all requests to the server. There is no "simple" solution to this when it comes to webrequests.
 
HttpWebRequest inherits from WebRequest and was designed for Http requests. WebRequest is pretty much the base class for all web requests.. WebClient is basically a 'streamlined' version of HttpWebRequest to make certain things easier. HttpWebRequest is going to give you more flexibility though and typically what you want to use.

WebBrowser is no good but it's main use is emulation/parsing the Java. You can try Awesomium, it's a headless browser that you can render but not forced too. It can emulate hardware, parse Java, and is much faster than the WebBrowser control.

Ideally I would attempt to process in HttpWebRequest and if it fails, or recognizes it needs to parse Java to succeed, it then fires up the Browser and gives it another go with emulations.
 
@HatIsBlack:
So, you mean to say that even (ajax) can be handled with webrequest. Can you guide what is the best software for tracking/sniffing requests/resposes to/from server? (The one which has complete tutorial as well) As I know of Wireshark, Fiddler but if there is anything better than these two?

@adh0c:
Awesomium is free? Can it be embedded just like webbrowser in .Net C# program in VS 2010?
According to you, java is difficult to handle in httpwebrequest and when it can't be handled then you'll go for browser solution and for browser solution, you prefer awesomium? Correct?

Thanks
 
Last edited:
Webrequest can do anything, I don't think there is something webrequest can't accomplish. And I agree awesomium is perfect, but uses a lot of ram and cpu, so if it's not necessary to use awesomium, go with the native IE browser.
 
use HttpWebRequest if you want your software to fly and WebBrowser if you want it to walk like a 95 yo granny ;) As HatIsBlack mentioned, WebBrowser is much easier in some cases - I used it when I had to login into a few hundred AOL accounts and change their settings; figuring out how to do just that with HttpWebRequest could be quite unnerving and take you hours
 
@DarkPixel:
Any recommended book/tutorials/software for programing in C# with webrequest for automation stuff.

@hpv222:
Same thing, what do you suggest. As everyone around is saying webrequest (http) is better than webbrowser, I know and agree with you guys. But, how to?

Like I am getting error when I try to just go to youtube,com usign httpwebrequest with a private proxy.
Have searched a lot on google and forums but no solution. (As I have not done much work in System.net that's why don't know a lot)
error: "The server committed a protocol violation, Section=ResponseStatusLine"
 
It depends on the site you target. If the site has lot of javascript or jquery, then it's easy to do with webbrowser. Though you can do it with webrequest it'll be tricky in case of jquery.
 
Yeah, Awesomium is free and preferred IMO. It gives you much more control. Works very similar to the web browser but doesn't have as much overhead as the browser control if used properly. IMO, it's much better for threading the "browser". If your not worried about threading, then the browser control works fine.

For most websites though, HttpWebRequest will work fine. Just write a class that will parse the Form tags in the HTML and make your GET/POST requests accordingly.

FIY,
Regardless of the type of host; ajax, php, aspx, asmx, svc, html.. the HttpWebRequest can handle it. It simply breaks down to requesting another URL and parsing out variables 99% of the time. The only problems usually come about when the website uses javascript to generate code on DOM-events to identify "bots" Vs. more-than-likely humans. Most websites don't do this though, usually just really popular sites that get hit hard with spam/bots.

Oh, and Fiddler2 is THE application for anything dealing with sniffing web requests.
 
adh0c said:
Yeah, Awesomium is free and preferred IMO. It gives you much more control. Works very similar to the web browser but doesn't have as much overhead as the browser control if used properly. IMO, it's much better for threading the "browser". If your not worried about threading, then the browser control works fine.
Click to expand...

As far as I know, Awesomium isn't thread-safe ;) You can have multiple "tabs" but they share the same cookies etc. Are you sure you 've been using it correctly?
 
Why do people feel the need to re-create the same threads over and over?

In programming there is a rule called DRY - Don't Repeat Yourself, maybe this rule should be applied to posts in this forum.
 
Chris22 said:
Why do people feel the need to re-create the same threads over and over?

In programming there is a rule called DRY - Don't Repeat Yourself, maybe this rule should be applied to posts in this forum.
Click to expand...

I 'd have to delete a lot of threads then :)

Apart from the joke, I think there 's a chance for new information to pop up every time a thread comes in. Not all people have seen the old threads and it 's a chance for them to contribute if they have something to share.
 
I use WebRequests for all my botting purposes. All I do is sniff the traffic using Fiddler and copy/paste the headers it sent into a comment in my code and duplicate it. Don't forget to use automatic decompression if the page is encrypted. Really just make sure you are sending all of the same requests your browser does. You can see some examples if you go through a couple of my threads. All the bots I have released here are completely open source for reasons like this.

Good luck!
 
xxf8xx said:
I use WebRequests for all my botting purposes. All I do is sniff the traffic using Fiddler and copy/paste the headers it sent into a comment in my code and duplicate it. Don't forget to use automatic decompression if the page is encrypted. Really just make sure you are sending all of the same requests your browser does. You can see some examples if you go through a couple of my threads. All the bots I have released here are completely open source for reasons like this.

Good luck!
Click to expand...
Inspirational post! Can you kindly add me on your Skype: vebxperts or if you can PM me yours. I think you won't mind helping me in your free time, regarding programing?
 
Thank Gary, Got it and I am amazed to see that they have not talked anywhere in 630 pages about "how to use private password protected proxies, even simply how to use proxies".

I don't think it's a complete book. Specially when you are making a Bot, proxy is the key feature that has to be discussed.
 
how can you post data with WebRequests?
For example for blog commenting, how can you fill name,email,website,comment controls and post data?
 
In short,

WebRequest - is great for things you can easily automate, like RESTful API's etc. Where it starts to get more complicated is when websites use techniques like "cross-site request forgery" (CSRF) protection or fat JavaScript/AJAX applications. Pinterest use CSRF protection in all their web-forms.

WebBrowser - basically this is a cut-down version of Internet Explorer, because its running a full browser stack it can be a lot slower as its loading Images, CSS, JavaScript and rendering the pages. Use it sparingly, where you cant just go direct with WebRequests.
 
Don't use Web Browser for form automation! I've tried it a few times, and whilst you might think it's a headache using Web Request or HTTP Web Request, the performance difference makes it worth learning about these, especially if you want to automate it with multi-threading, which isn't so easy (not sure if it's possible) with web browsers. Use Web Browsers when there's a UI requirement for one, ie an application intended for public use including browser-based interaction.

Get Fiddler2, learn how to use it and using Web Request it should be a piece of cake 90-99% of the time.
 
^You can multithread WebBrowsers :) In some cases, it's better off to use WebBrowser if you're dealing with a lot of Javascript stuff. Otherwise, stick to HttpWebRequests.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top