What do you use to simulate browser activity in C#

[bartosimpsonio]

How many people run plugins to randomise the header order?

But as I said, the above was from memory of the "type" of things you can do to detect phantomJS. There are a few more and you can with 100% accuracy detect it (according to the reports). Some are protectable with JS coding to remove, but things like header order would be recompiles of phantomjs or webkit and I think some old QT they are / were using.

The point being, if you want to be undetectable in your automations, phantom is not really the way to do. If you are running tests on your website it might make sense, but what with phantom bugs you would be better with selenium

No, not plugins TO randomize header order. But some plugins do add their signature to the headers. Alexa toolbar changes the user agent name, and others do too, just to give an example.

Again, header order is not mandatory in the HTTP specs, except for the first line (HTTP/version CODE) and the Host: line. Can some header checking be used to filter out referral spam and headless browsers? Maybe, but for specific versions. New versions of Chrome or Firefox may change the headers and you'll break the site if you filter those out.

 

[rootjazz]

No, not plugins TO randomize header order. But some plugins do add their signature to the headers. Alexa toolbar changes the user agent name, and others do too, just to give an example.

Again, header order is not mandatory in the HTTP specs, except for the first line (HTTP/version CODE) and the Host: line. Can some header checking be used to filter out referral spam and headless browsers? Maybe, but for specific versions. New versions of Chrome or Firefox may change the headers and you'll break the site if you filter those out.

Any proof new versions of FF / Chrome change the ordering? I imagine that code does not change. At the end of the day if a headless browser had identifying properties due to ordering, you can use it to ID.

If there is no legitimate reason for you to allow headless scraping of your site you can block it, with the caveat that you mention, should the main browsers change their ordering to be the same as the headless browser, they will also be blocked, but this hardly ever / never happens and if it does you can deal with it.

 

[bartosimpsonio]

Any proof new versions of FF / Chrome change the ordering? I imagine that code does not change. At the end of the day if a headless browser had identifying properties due to ordering, you can use it to ID.

If there is no legitimate reason for you to allow headless scraping of your site you can block it, with the caveat that you mention, should the main browsers change their ordering to be the same as the headless browser, they will also be blocked, but this hardly ever / never happens and if it does you can deal with it.

It's not about having proof, it's about following the HTTP standard. Header order doesn't matter so browsers are free to change it.

 

[rootjazz]

It is all about proof. The header spec doesn't mean anything *IF* it can be used to detect bots / fraud because things are standard. If FF is always in one particular order and chrome is always in another, and a popular headless browser is always in another order. Then why would you not use it? Just because the browser *may* change it but haven't for 10+ years


Just came across this link after reading about ad-fraud

http://geocar.sdf1.org/browser-verification.html

Browser Verification
There are very good reasons to not believe the browser's HTTP user agent field, however most of the techniques look like browser detection code. This page reviews some passive methods that I'm familiar with.

HTTP Header Order
Many browser simulators do not use the same header order as the genuine article. It is useful to store this.

Firefox goes: Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset, Keep-Alive, Connection, Referer

Chrome goes: Host, Connection, Accept, User-Agent, Referer, Accept-Encoding, Accept-Language, Accept-Charset, Keep-Alive

Chrome used to go: Host, Connection, User-Agent, Accept, Referrer, ...

MSIE after version 6 goes: Accept, Referer, Accept-Language, User-Agent, Accept-Encoding, Host, Connection, Keep-Alive, Accept-Charset

MSIE version 6 goes: Accept, Referer, User-Agent, Host, Accept-Encoding, Accept-Language, Accept-Charset

MSIE version 6 sometimes goes: Accept, Connection, Host, Referer, User-Agent, Accept-Encoding, Accept-Language, Accept-Charset

Sometimes headers are missing. Some proxy servers drop/insert headers.

Also a lot more detection methods on that page as well.

Is header order the main vector to use to detect bots / fraud, no. But *can* it be used, yes and by all accounts can be successful

 

[flashsites]

I use HTTP Requests after studying the call on Fiddler. Or if I have to use the document, I'll use the HTML Agility pack and invoke a call on the elements from there.
Theres also the NHtmlUnit nuget package which I've heard good things about.

 

[Rynox]

You can use HTTp request modification or try Selenium or other web browser framework. Either way is good.

 

[IotaInsanity]

Little late to the party but I mostly use web requests. Spoofing browsers is too annoying.

 

[Adslytic]

I was using HTTP Request, this way I was able to create my own environment that I was able fully control (which js run which not).

 

[dinkerdok]

You might need to use a mixture depending on what youre automating. For instance, some forms are difficult in nightmare js, or casperjs/phantomjs, but they work really well in selenium. Generally, I try to use a js library first, and if I get stuck then move to selenium (it's never failed me yet, but is a bit slower than js libraries) Not sure as well whether you can test headless and non-headless with the same browser in selenium - so when you go headless there might be differences but hopefully not too many.

 

[web2.0rishi]

What about GeckoFx?
Anybody used this?