C# .net HttpWebRequest vs 3rd party libraries?

[roamer]

Why bother? Why reinvent the wheel coded by (probably) better coders.

We all know that things we create are uber-perfect and everything written by anyone else is inefficient crap, right?

It 's called the NIH (Not Invented Here) syndrome.

This. Ignoring the solutions already at hand is the best way to doom a project, by concentrating on the trivial aspects of the project (re-creating existing solutions from scratch) instead of working on the specific functionality needed (ie. what is actually required and is not pre-existant).

 

[rootjazz]

Thanks rootjazz,
And I have seen a few other people recommending "HTTP Programming Recipes for C# Bots" by Heaton research, I just wasn't sure if everything in there was still 100% relevant and the best way to do things with the current versions of C#, seeing as how it was published in 2007. But I guess HttpWebRequest itself hasn't changed much over the years from then, just new classes being added like HttpClient.

I believe they keep it updated. Had a browser through last year for old times sake and it was different to how I remember it years back.

But the HTTP protocol hasn't really changed for what we need.

GET this page

POST this data.

Get fiddler2 installed and see what actually happens when you browse a basic website (start basic, don't go for a JS / AJAX heavy site). To make it easier, install adblock edge then run fiddler.


All you will see is
GET page
POST data

That is all there is (in a very basic fashion)

 

[rootjazz]

This. Ignoring the solutions already at hand is the best way to doom a project, by concentrating on the trivial aspects of the project (re-creating existing solutions from scratch) instead of working on the specific functionality needed (ie. what is actually required and is not pre-existant).

We all know that things we create are uber-perfect and everything written by anyone else is inefficient crap, right?

It 's called the NIH (Not Invented Here) syndrome.

Finally some sense in this thread.

Do the bare minimum you can to complete the project, that is my motto. And expect your code to be bug ridden, spaghetti logic crap, then you are not disappointed.

Once the code is working, then make it pretty, refactor it. But get it complete, get it working, get it working and out there. Chances are the project will tank and no one will pay you money for it. But eventually you'll hit onto something that people want. Then improve.


Do try to write quality code from the start, keep your classes small, keep your methods small, do make attempts at a good OO design, but don't stess about it whether this pattern or that will give you 0.0001% greater efficiency, or whether it makes it easier to add on modules *if* the project sells.


Anyway, OP, this thread is quite old now, how is your progress?

 

[Ghoro]

Anyway, OP, this thread is quite old now, how is your progress?

Well the last few months I have been pretty busy so haven't gotten as far as I would like, but I have gotten it to the point where I can load up a list of keywords and proxies and pull out the top 10 results for each keyword with the help of HtmlAgilityPack. Although the results seem to very a bit from when I just do a google search in Chrome's incognito mode. Other than maybe looking into why that is, the next big part that I have not looked into how to do it is what is the best way to get information about a webpage, like PR, backlinks, age and just any other information. But like I said I'm not at that point just yet and haven't looked into it, so it maybe easier then I think.

Again thanks everyone for all your help.

 

[rootjazz]

like PR, backlinks, age and just any other information

so how do you do that know? go to a site that lists it? so automate it.

Or google c# pr checker. You can find the code to get that

Although the results seem to very a bit from when I just do a google search in Chrome's incognito mode.

So load up FIDDLER, perform the action in the browser then in your program. Analyse the differences in the headers. The truth is ALWAYS in the headers, even if you don't think it is, it is.