1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

C# .net HttpWebRequest vs 3rd party libraries?

Discussion in 'C, C++, C#' started by Ghoro, Feb 3, 2014.

  1. Ghoro

    Ghoro Registered Member

    Joined:
    Jun 16, 2012
    Messages:
    85
    Likes Received:
    3
    So basically I'm looking to jump in to making bots for personal use at first, maybe later on try to sell some on the side, but after reading up on it for a few hours I'm unsure if I should go with what comes with .net or get some 3rd party libraries. For now I'm mainly looking to make a bot that would be like most keyword checkers and check for difficulty of a list of keywords, but with a few changes than what is currently out there which is why I don't just use those.

    So is there any huge advantages to using any 3rd party libraries and if so which ones? Or is what comes with the latest version of C# .net all I need to get the job done just as easy?

    Thanks
     
    Last edited: Feb 3, 2014
  2. Macthetrix

    Macthetrix Regular Member

    Joined:
    Feb 21, 2013
    Messages:
    379
    Likes Received:
    390
    Occupation:
    The Boss
    Location:
    Silicon Valley
    The standard libraries that come with C# are good enough when it comes to HttpWebrequests and building bots. In few cases you might consider to use a 3rd party library but that depends on the functionality you need. E.g NewtonSoft.json.dll to read and format json responses etc
     
    • Thanks Thanks x 1
  3. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,441
    Likes Received:
    1,005
    Occupation:
    SEO
    Location:
    IM Wonderland
    If at all possible, always write your own libs instead of using 3rd party ones.
     
    • Thanks Thanks x 2
  4. severenity

    severenity Newbie

    Joined:
    Feb 3, 2014
    Messages:
    15
    Likes Received:
    6
    Occupation:
    don't really know lolz
    Location:
    wonderland
    Home Page:
    Httpwebrequest will be too slow for chuncked response, you need to create your own httpclient

    find some in stackoverflow / codeproject, they have very good reference and tutorial
     
    • Thanks Thanks x 1
  5. Ghoro

    Ghoro Registered Member

    Joined:
    Jun 16, 2012
    Messages:
    85
    Likes Received:
    3
    Thanks for the replies. And I was thinking that I wanted to start with the standard libraries if for no other reason, but to learn more about it. Although it does make sense that the only thing I may still want to get a 3rd party library for is json responses, but even then I will see what it would be like to do it on my own.

    And thanks for the heads up that Httpwebrequest on its own maybe a bit slow, even more so for chunked responses. When I get back home I will be able to search more about it, but of course if you or anyone knows any good examples on stackoverflow, codeproject or elsewhere that can help lead me in the right direction to making my own httpclient/library to deal with chunked responses and just speed in general that would be greatly appreciated :)

    Thanks
     
  6. severenity

    severenity Newbie

    Joined:
    Feb 3, 2014
    Messages:
    15
    Likes Received:
    6
    Occupation:
    don't really know lolz
    Location:
    wonderland
    Home Page:
    jsonresponse ? try simplejson

    for the tutorials, you need to googling it, all only for reference, the best is create from your own thought
     
    • Thanks Thanks x 1
  7. Ghoro

    Ghoro Registered Member

    Joined:
    Jun 16, 2012
    Messages:
    85
    Likes Received:
    3
    Well part of the problem was that I wasn't really sure what I was looking or needed to search for. But after just reading up a bit on networking with C# in general it seems like what I am looking to use to build my own httpclient/library is TcpClient, UdpClient, TcpListener, and Socket classes. Is that the general direction I need to be looking in to or not?

    Thanks
     
  8. severenity

    severenity Newbie

    Joined:
    Feb 3, 2014
    Messages:
    15
    Likes Received:
    6
    Occupation:
    don't really know lolz
    Location:
    wonderland
    Home Page:
    yes, that's correct direction. i would like to give u some code, but i cannot post anything with code mark :D
     
    • Thanks Thanks x 1
    Last edited: Feb 7, 2014
  9. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    Disagree completely. So far as to say that is terrible advice.

    Care to elaborate why the HttpWebRequest would be too slow and why he needs that speed. And why a new programmer can do better than the devs are Microsoft?

    Don't get me wrong, going a step lower is a good learning project, gives you a better understanding. But it is far from necessary.

    Why reinvent the wheel?

    Go with the HttpWebRequest class. When you know you need something faster, than drop down an abstraction level. I will put money on you won't ever need to.

    If absolute raw speed is a necessity, than you probably shouldn't be looking at C# and the .Net framework anyway.

    As the OP put
    What makes you think the HttpWebRequest is not good enough? It is exactly what he *should* be using.

    Bots do not need to run at break neck speed, if they do you can get ID'd as a bot.

    What OP needs to do, is just get started.
    Code a bot
    Complete a project
    Start another project which will be 10x better than the first
    Complete that
    etc

    Along the way he will start to realise all the code he is reusing and will build up a personal library.

    But even then, if he can find a 3rd party library why not use that? His goal is to make bots, that do something. The goal is not to understand the underyling network protocol.


    He needs to be able to
    1) Get pages
    2) decompress gzip output
    3) handle cookies
    4) post forms (multi part and url encoded)
    5) post ajax
    6) post json

    Then some HTML processing would be handy, extract forms from pages, extract inputs etc.

    If a library can handle that for him, he will have a bot ready in weeks. If he goes codes a httpWebRequest replacement like you suggest, he will never complete a bot as will give up. Fact.
     
    • Thanks Thanks x 6
  10. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    No, just no. Stop with all this sadism.

    Why bother? Why reinvent the wheel coded by (probably) better coders.

    What do you use for parsing the HTML? HtmlAgilityPack right. Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

    Want to handle JSON? Then you use Newtonsoft JSN right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

    Want to handle zips files? Use IonicZip right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

    What to verify emails via the Pop protocol? Use OpenPop right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?


    If you recode everything, you will get nothing done. Nothing wrong recoding it as a learning excerise as the end point itself. But saying NOTto use libs when the end point is get a program completed / released / sold is just bad advice
     
    • Thanks Thanks x 1
  11. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,441
    Likes Received:
    1,005
    Occupation:
    SEO
    Location:
    IM Wonderland
    For everyday stuff or bots you need for personal use - ofc, you're right.

    But when it comes to commercial software, if you are writing a web submitter for example, you can use any of the 3rd party libs out there with all the functs anyone would need included OR you can write your own which would be memory efficient, faster and clean.
     
    • Thanks Thanks x 2
  12. severenity

    severenity Newbie

    Joined:
    Feb 3, 2014
    Messages:
    15
    Likes Received:
    6
    Occupation:
    don't really know lolz
    Location:
    wonderland
    Home Page:
    Make sure you are not "DDOS" ing the site :p

    ----That's true

    OP can i know how do you handle the response ? i don't think the problem is in using Httpwebrequest, but how to handle the response
     
    • Thanks Thanks x 2
    Last edited: Feb 7, 2014
  13. Ghoro

    Ghoro Registered Member

    Joined:
    Jun 16, 2012
    Messages:
    85
    Likes Received:
    3
    Thanks a lot rootjazz. I realize I was starting to think about it a bit backwards, that before I can start messing around with making my own HttpWebRequest/HttpClient, I need to actually use the current HttpWebRequest and 3[SUP]rd[/SUP] party libraries. Bit hard to make your own HttpWebRequest when you don?t even really know what HttpWebRequest does :p

    Like you said my first goal should be getting my first bot up and running as easy and best I can, which 3[SUP]rd[/SUP] party libraries are the best way to do that. Then after I go through that I will have a much deeper understanding of the whole process and just how everything works. After that if I want I can start looking on a deeper level on how everything works and maybe play around with making my own HttpWebRequest, not really because I think I will make something better than a team of more advanced programmers, but just to learn more about it. I?m sure in the end if I ever do build something that I would sell I would rely on 3[SUP]rd[/SUP] party libraries as much as I can anyway, not just out of laziness, but because it would be better than something just one guy on his own like me could make :p
     
  14. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    If you write your own library, what makes you think there will be no bugs? What makes it will be more memory efficient / faster / clean than a library that has been around for 10+ years written by the .Net team that has been reviewed / tested thoroughly.

    I can assure you, that commercial software that sends and receives data via the HTTP protocol uses the HttpWebrequest. I will go so far to say it is probably like 99%.

    Now perhaps you are some prodigy programmer genius, who writes perfect code first time, but I doubt it.

    Yes there are crappy libs out there, but it isn't hard to find the quality ones.

    Where do you stop, what other parts of the .net framework do you want to rip out? Perhaps string handling should go back to how it was in C because you could do it better?
     
    • Thanks Thanks x 2
  15. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    This is not true. In all the places I have worked out and from others I have asked, for HTTP work, the HttpWebRequest class is used. I am sure there "some" instances, where people have dropped down a level. But it for sure is not for something as submitting a form to a website.
     
    • Thanks Thanks x 1
  16. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:

    I use a personal library built on top of the HttpWebRequest class. Purely because when I started out there were no libraries about. Have never looked recently so no idea what is about, but there is probably something.


    For getting an introduction to c# bots and HTTP in general, there is a PDF "Bot building in c#" or something, by "Heaton research institute" or something. You should buy for it, but you can find it for free if you look. It is good for a beginner as it goes through each step towards building a library.

    Check out as well
    HtmlAgilityPack, for parsing HTML.
    Regular expressions will also come in handy as will Xpath
    Fiddler2, for checking out the headers sent from your browser, this is what you need to replicate

    GetPage
    PostData
    PostDataAjax
    PostDataJson

    Not much else you really need to do than that
     
    • Thanks Thanks x 1
  17. Quffs

    Quffs Junior Member

    Joined:
    Sep 4, 2012
    Messages:
    153
    Likes Received:
    54
    Hello Jazz, I would like to ask what's the best language to learn to program a bot? I have a solid background in programming but focuses more on web, I want to dive the C family but don't know which one to study first. Thanks!
     
  18. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,441
    Likes Received:
    1,005
    Occupation:
    SEO
    Location:
    IM Wonderland
    I agree 100% - However, most of those libs are made to suit anyone's needs in the area, and end up taking up more memory even if you don't need most of it. Take a look at the RAM usage of xrumer at 200 threads and any other software on the market.

    All I am saying is, although 3rd party libs are great and get the job done faster, having a custom solution consisting only of the stuff you need can result in cleaner/simpler solutions that are easier to debug and more efficient to run.

    BTW, I might be mistaken, but I think scrapebox is handling string manipulation (dupe removes etc) using assembler, so sometimes going back to basics might not be the worst idea for the essential parts.
     
    • Thanks Thanks x 1
  19. Ghoro

    Ghoro Registered Member

    Joined:
    Jun 16, 2012
    Messages:
    85
    Likes Received:
    3
    Thanks rootjazz,
    And I have seen a few other people recommending "HTTP Programming Recipes for C# Bots" by Heaton research, I just wasn't sure if everything in there was still 100% relevant and the best way to do things with the current versions of C#, seeing as how it was published in 2007. But I guess HttpWebRequest itself hasn't changed much over the years from then, just new classes being added like HttpClient.
     
  20. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,148
    We all know that things we create are uber-perfect and everything written by anyone else is inefficient crap, right? ;)

    It 's called the NIH (Not Invented Here) syndrome.
     
    • Thanks Thanks x 1