C# .net HttpWebRequest vs 3rd party libraries?

So basically I'm looking to jump in to making bots for personal use at first, maybe later on try to sell some on the side, but after reading up on it for a few hours I'm unsure if I should go with what comes with .net or get some 3rd party libraries. For now I'm mainly looking to make a bot that would be like most keyword checkers and check for difficulty of a list of keywords, but with a few changes than what is currently out there which is why I don't just use those.

So is there any huge advantages to using any 3rd party libraries and if so which ones? Or is what comes with the latest version of C# .net all I need to get the job done just as easy?

Thanks

The standard libraries that come with C# are good enough when it comes to HttpWebrequests and building bots. In few cases you might consider to use a 3rd party library but that depends on the functionality you need. E.g NewtonSoft.json.dll to read and format json responses etc

If at all possible, always write your own libs instead of using 3rd party ones.

Httpwebrequest will be too slow for chuncked response, you need to create your own httpclient

find some in stackoverflow / codeproject, they have very good reference and tutorial

Thanks for the replies. And I was thinking that I wanted to start with the standard libraries if for no other reason, but to learn more about it. Although it does make sense that the only thing I may still want to get a 3rd party library for is json responses, but even then I will see what it would be like to do it on my own.

And thanks for the heads up that Httpwebrequest on its own maybe a bit slow, even more so for chunked responses. When I get back home I will be able to search more about it, but of course if you or anyone knows any good examples on stackoverflow, codeproject or elsewhere that can help lead me in the right direction to making my own httpclient/library to deal with chunked responses and just speed in general that would be greatly appreciated

Thanks

jsonresponse ? try simplejson

for the tutorials, you need to googling it, all only for reference, the best is create from your own thought

Well part of the problem was that I wasn't really sure what I was looking or needed to search for. But after just reading up a bit on networking with C# in general it seems like what I am looking to use to build my own httpclient/library is TcpClient, UdpClient, TcpListener, and Socket classes. Is that the general direction I need to be looking in to or not?

Thanks

yes, that's correct direction. i would like to give u some code, but i cannot post anything with code mark

severenity said:

Httpwebrequest will be too slow for chuncked response, you need to create your own httpclient

find some in stackoverflow / codeproject, they have very good reference and tutorial

Click to expand...

Disagree completely. So far as to say that is terrible advice.

Care to elaborate why the HttpWebRequest would be too slow and why he needs that speed. And why a new programmer can do better than the devs are Microsoft?

Don't get me wrong, going a step lower is a good learning project, gives you a better understanding. But it is far from necessary.

Why reinvent the wheel?

Go with the HttpWebRequest class. When you know you need something faster, than drop down an abstraction level. I will put money on you won't ever need to.

If absolute raw speed is a necessity, than you probably shouldn't be looking at C# and the .Net framework anyway.

As the OP put

So basically I'm looking to jump in to making bots for personal use at first, maybe later on try to sell some on the side

Click to expand...

What makes you think the HttpWebRequest is not good enough? It is exactly what he *should* be using.

Bots do not need to run at break neck speed, if they do you can get ID'd as a bot.

What OP needs to do, is just get started.
Code a bot
Complete a project
Start another project which will be 10x better than the first
Complete that
etc

Along the way he will start to realise all the code he is reusing and will build up a personal library.

But even then, if he can find a 3rd party library why not use that? His goal is to make bots, that do something. The goal is not to understand the underyling network protocol.


He needs to be able to
1) Get pages
2) decompress gzip output
3) handle cookies
4) post forms (multi part and url encoded)
5) post ajax
6) post json

Then some HTML processing would be handy, extract forms from pages, extract inputs etc.

If a library can handle that for him, he will have a bot ready in weeks. If he goes codes a httpWebRequest replacement like you suggest, he will never complete a bot as will give up. Fact.

accelerator_dd said:

If at all possible, always write your own libs instead of using 3rd party ones.

Click to expand...

No, just no. Stop with all this sadism.

Why bother? Why reinvent the wheel coded by (probably) better coders.

What do you use for parsing the HTML? HtmlAgilityPack right. Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

Want to handle JSON? Then you use Newtonsoft JSN right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

Want to handle zips files? Use IonicZip right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

What to verify emails via the Pop protocol? Use OpenPop right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?


If you recode everything, you will get nothing done. Nothing wrong recoding it as a learning excerise as the end point itself. But saying NOTto use libs when the end point is get a program completed / released / sold is just bad advice

rootjazz said:

No, just no. Stop with all this sadism.

Why bother? Why reinvent the wheel coded by (probably) better coders.

What do you use for parsing the HTML? HtmlAgilityPack right. Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

Want to handle JSON? Then you use Newtonsoft JSN right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

Want to handle zips files? Use IonicZip right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?

What to verify emails via the Pop protocol? Use OpenPop right? Those guys have done a great job and know more about it than you and I ever will. Why recode the lib?


If you recode everything, you will get nothing done. Nothing wrong recoding it as a learning excerise as the end point itself. But saying NOTto use libs when the end point is get a program completed / released / sold is just bad advice

Click to expand...

For everyday stuff or bots you need for personal use - ofc, you're right.

But when it comes to commercial software, if you are writing a web submitter for example, you can use any of the 3rd party libs out there with all the functs anyone would need included OR you can write your own which would be memory efficient, faster and clean.

Make sure you are not "DDOS" ing the site

accelerator_dd said:

But when it comes to commercial software, if you are writing a web submitter for example, you can use any of the 3rd party libs out there with all the functs anyone would need included OR you can write your own which would be memory efficient, faster and clean.

Click to expand...

----That's true

OP can i know how do you handle the response ? i don't think the problem is in using Httpwebrequest, but how to handle the response

Thanks a lot rootjazz. I realize I was starting to think about it a bit backwards, that before I can start messing around with making my own HttpWebRequest/HttpClient, I need to actually use the current HttpWebRequest and 3[SUP]rd[/SUP] party libraries. Bit hard to make your own HttpWebRequest when you don?t even really know what HttpWebRequest does

Like you said my first goal should be getting my first bot up and running as easy and best I can, which 3[SUP]rd[/SUP] party libraries are the best way to do that. Then after I go through that I will have a much deeper understanding of the whole process and just how everything works. After that if I want I can start looking on a deeper level on how everything works and maybe play around with making my own HttpWebRequest, not really because I think I will make something better than a team of more advanced programmers, but just to learn more about it. I?m sure in the end if I ever do build something that I would sell I would rely on 3[SUP]rd[/SUP] party libraries as much as I can anyway, not just out of laziness, but because it would be better than something just one guy on his own like me could make

accelerator_dd said:

For everyday stuff or bots you need for personal use - ofc, you're right.

But when it comes to commercial software, if you are writing a web submitter for example, you can use any of the 3rd party libs out there with all the functs anyone would need included OR you can write your own which would be memory efficient, faster and clean.

Click to expand...

If you write your own library, what makes you think there will be no bugs? What makes it will be more memory efficient / faster / clean than a library that has been around for 10+ years written by the .Net team that has been reviewed / tested thoroughly.

I can assure you, that commercial software that sends and receives data via the HTTP protocol uses the HttpWebrequest. I will go so far to say it is probably like 99%.

Now perhaps you are some prodigy programmer genius, who writes perfect code first time, but I doubt it.

Yes there are crappy libs out there, but it isn't hard to find the quality ones.

Where do you stop, what other parts of the .net framework do you want to rip out? Perhaps string handling should go back to how it was in C because you could do it better?

severenity said:

Make sure you are not "DDOS" ing the site

----That's true

OP can i know how do you handle the response ? i don't think the problem is in using Httpwebrequest, but how to handle the response

Click to expand...

This is not true. In all the places I have worked out and from others I have asked, for HTTP work, the HttpWebRequest class is used. I am sure there "some" instances, where people have dropped down a level. But it for sure is not for something as submitting a form to a website.

Ghoro said:

Thanks a lot rootjazz. I realize I was starting to think about it a bit backwards, that before I can start messing around with making my own HttpWebRequest/HttpClient, I need to actually use the current HttpWebRequest and 3[SUP]rd[/SUP] party libraries. Bit hard to make your own HttpWebRequest when you don't even really know what HttpWebRequest does

Like you said my first goal should be getting my first bot up and running as easy and best I can, which 3[SUP]rd[/SUP] party libraries are the best way to do that. Then after I go through that I will have a much deeper understanding of the whole process and just how everything works. After that if I want I can start looking on a deeper level on how everything works and maybe play around with making my own HttpWebRequest, not really because I think I will make something better than a team of more advanced programmers, but just to learn more about it. I'm sure in the end if I ever do build something that I would sell I would rely on 3[SUP]rd[/SUP] party libraries as much as I can anyway, not just out of laziness, but because it would be better than something just one guy on his own like me could make

Click to expand...

I use a personal library built on top of the HttpWebRequest class. Purely because when I started out there were no libraries about. Have never looked recently so no idea what is about, but there is probably something.


For getting an introduction to c# bots and HTTP in general, there is a PDF "Bot building in c#" or something, by "Heaton research institute" or something. You should buy for it, but you can find it for free if you look. It is good for a beginner as it goes through each step towards building a library.

Check out as well
HtmlAgilityPack, for parsing HTML.
Regular expressions will also come in handy as will Xpath
Fiddler2, for checking out the headers sent from your browser, this is what you need to replicate

GetPage
PostData
PostDataAjax
PostDataJson

Not much else you really need to do than that

Hello Jazz, I would like to ask what's the best language to learn to program a bot? I have a solid background in programming but focuses more on web, I want to dive the C family but don't know which one to study first. Thanks!

rootjazz said:

If you write your own library, what makes you think there will be no bugs? What makes it will be more memory efficient / faster / clean than a library that has been around for 10+ years written by the .Net team that has been reviewed / tested thoroughly.

I can assure you, that commercial software that sends and receives data via the HTTP protocol uses the HttpWebrequest. I will go so far to say it is probably like 99%.

Now perhaps you are some prodigy programmer genius, who writes perfect code first time, but I doubt it.

Yes there are crappy libs out there, but it isn't hard to find the quality ones.

Where do you stop, what other parts of the .net framework do you want to rip out? Perhaps string handling should go back to how it was in C because you could do it better?

Click to expand...

I agree 100% - However, most of those libs are made to suit anyone's needs in the area, and end up taking up more memory even if you don't need most of it. Take a look at the RAM usage of xrumer at 200 threads and any other software on the market.

All I am saying is, although 3rd party libs are great and get the job done faster, having a custom solution consisting only of the stuff you need can result in cleaner/simpler solutions that are easier to debug and more efficient to run.

BTW, I might be mistaken, but I think scrapebox is handling string manipulation (dupe removes etc) using assembler, so sometimes going back to basics might not be the worst idea for the essential parts.

rootjazz said:

I use a personal library built on top of the HttpWebRequest class. Purely because when I started out there were no libraries about. Have never looked recently so no idea what is about, but there is probably something.


For getting an introduction to c# bots and HTTP in general, there is a PDF "Bot building in c#" or something, by "Heaton research institute" or something. You should buy for it, but you can find it for free if you look. It is good for a beginner as it goes through each step towards building a library.

Check out as well
HtmlAgilityPack, for parsing HTML.
Regular expressions will also come in handy as will Xpath
Fiddler2, for checking out the headers sent from your browser, this is what you need to replicate

GetPage
PostData
PostDataAjax
PostDataJson

Not much else you really need to do than that

Click to expand...

Thanks rootjazz,
And I have seen a few other people recommending "HTTP Programming Recipes for C# Bots" by Heaton research, I just wasn't sure if everything in there was still 100% relevant and the best way to do things with the current versions of C#, seeing as how it was published in 2007. But I guess HttpWebRequest itself hasn't changed much over the years from then, just new classes being added like HttpClient.

rootjazz said:

If you write your own library, what makes you think there will be no bugs? What makes it will be more memory efficient / faster / clean than a library that has been around for 10+ years written by the .Net team that has been reviewed / tested thoroughly.

Click to expand...

We all know that things we create are uber-perfect and everything written by anyone else is inefficient crap, right?

It 's called the NIH (Not Invented Here) syndrome.