For Bot Development Which One Your Prefer HTTPWebRequest Class Or HTMLAligityPack And Why
- Thread starter KirtiD
- Start date
Chris22
Regular Member
- Sep 29, 2010
- 400
- 1,069
HttpWebRequest and HtmlAgilityPack are two completely different things that serve different purposes.
HttpWebRequest is an object used to send and receive data over http.
HtmlAgilityPack is a library that lets you traverse html easily.
I think HtmlAgilityPack does retrieve html content over http, but this is bad because it puts the handling of the requests out of your control. You wouldn't be able to change useragent/proxy/etc very easily this way.
They should be used in conjunction with each other, not exclusively.
HttpWebRequest is an object used to send and receive data over http.
HtmlAgilityPack is a library that lets you traverse html easily.
I think HtmlAgilityPack does retrieve html content over http, but this is bad because it puts the handling of the requests out of your control. You wouldn't be able to change useragent/proxy/etc very easily this way.
They should be used in conjunction with each other, not exclusively.
Subsonic
Regular Member
- Mar 17, 2011
- 367
- 334
Like healzer already said Html Agility Pack most likely uses HttpWebRequest in the background. Anyway, you can't really compare those two because they're meant for totally different purposes. HttpWebRequest is used to get the data from the server and Html Agility Pack is used for parsing that data, although you only need the latter since it does the downloading of data by itself. On the other hand you are talking about bot development so parsing data might not be your main priority, then you want to use some other tool but still load the data by using HttpWebRequest.
If you want to build a bot for a website that uses lots of AJAX calls to update data, you might want to look at something like WatiN, which is for website testing but can be used for bots too. It makes life easier with AJAX, popup dialogs etc.
But anyway, I recommend that you use Html Agility Pack since it gives you a good interface for using XPath queries to get desired nodes from the web pages you are processing.
Edit: Looks like Chris22 said some similar stuff just before me. He also has a good additional point. HttpWebRequest is fine for basic bot development but when the stuff gets serious you might to move on to some other solutions that allow you to control the whole process of sending requests and receiving data. Take a look at sockets to get started.
If you want to build a bot for a website that uses lots of AJAX calls to update data, you might want to look at something like WatiN, which is for website testing but can be used for bots too. It makes life easier with AJAX, popup dialogs etc.
But anyway, I recommend that you use Html Agility Pack since it gives you a good interface for using XPath queries to get desired nodes from the web pages you are processing.
Edit: Looks like Chris22 said some similar stuff just before me. He also has a good additional point. HttpWebRequest is fine for basic bot development but when the stuff gets serious you might to move on to some other solutions that allow you to control the whole process of sending requests and receiving data. Take a look at sockets to get started.
Last edited: