1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

For Bot Development Which One Your Prefer HTTPWebRequest Class Or HTMLAligityPack And Why

Discussion in 'Visual Basic .NET' started by KirtiD, Aug 2, 2013.

  1. KirtiD

    KirtiD Newbie

    Joined:
    Jun 13, 2013
    Messages:
    49
    Likes Received:
    15
    Location:
    Beed
    HIi

    I need your expert suggestion for Bot Development Which One Your Prefer HTTPWebRequest Class Or HTMLAligityPack and Why
     
  2. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,363
    Likes Received:
    1,966
    Gender:
    Male
    Occupation:
    Marketing automation tools
    Location:
    Somewhere in Europe
    Home Page:
    I never used HTML AgPack
    Always used HHTPWebReq. it is the default way to make requests to a server.
    And the AgPack probably uses Httpwebreq. on the backend I think...
     
  3. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,059
    HttpWebRequest and HtmlAgilityPack are two completely different things that serve different purposes.

    HttpWebRequest is an object used to send and receive data over http.
    HtmlAgilityPack is a library that lets you traverse html easily.

    I think HtmlAgilityPack does retrieve html content over http, but this is bad because it puts the handling of the requests out of your control. You wouldn't be able to change useragent/proxy/etc very easily this way.

    They should be used in conjunction with each other, not exclusively.
     
  4. Subsonic

    Subsonic Regular Member

    Joined:
    Mar 17, 2011
    Messages:
    367
    Likes Received:
    333
    Location:
    DNS root zone database
    Like healzer already said Html Agility Pack most likely uses HttpWebRequest in the background. Anyway, you can't really compare those two because they're meant for totally different purposes. HttpWebRequest is used to get the data from the server and Html Agility Pack is used for parsing that data, although you only need the latter since it does the downloading of data by itself. On the other hand you are talking about bot development so parsing data might not be your main priority, then you want to use some other tool but still load the data by using HttpWebRequest.

    If you want to build a bot for a website that uses lots of AJAX calls to update data, you might want to look at something like WatiN, which is for website testing but can be used for bots too. It makes life easier with AJAX, popup dialogs etc.

    But anyway, I recommend that you use Html Agility Pack since it gives you a good interface for using XPath queries to get desired nodes from the web pages you are processing.


    Edit: Looks like Chris22 said some similar stuff just before me. He also has a good additional point. HttpWebRequest is fine for basic bot development but when the stuff gets serious you might to move on to some other solutions that allow you to control the whole process of sending requests and receiving data. Take a look at sockets to get started.
     
    Last edited: Aug 2, 2013
  5. KirtiD

    KirtiD Newbie

    Joined:
    Jun 13, 2013
    Messages:
    49
    Likes Received:
    15
    Location:
    Beed
    Thank you all of you for g8 support & suggestion