What do you use to simulate browser activity in C#

rootjazz said:

How many people run plugins to randomise the header order?

But as I said, the above was from memory of the "type" of things you can do to detect phantomJS. There are a few more and you can with 100% accuracy detect it (according to the reports). Some are protectable with JS coding to remove, but things like header order would be recompiles of phantomjs or webkit and I think some old QT they are / were using.

The point being, if you want to be undetectable in your automations, phantom is not really the way to do. If you are running tests on your website it might make sense, but what with phantom bugs you would be better with selenium

Click to expand...

bartosimpsonio said:

No, not plugins TO randomize header order. But some plugins do add their signature to the headers. Alexa toolbar changes the user agent name, and others do too, just to give an example.

Again, header order is not mandatory in the HTTP specs, except for the first line (HTTP/version CODE) and the Host: line. Can some header checking be used to filter out referral spam and headless browsers? Maybe, but for specific versions. New versions of Chrome or Firefox may change the headers and you'll break the site if you filter those out.

Click to expand...

rootjazz said:

Any proof new versions of FF / Chrome change the ordering? I imagine that code does not change. At the end of the day if a headless browser had identifying properties due to ordering, you can use it to ID.

If there is no legitimate reason for you to allow headless scraping of your site you can block it, with the caveat that you mention, should the main browsers change their ordering to be the same as the headless browser, they will also be blocked, but this hardly ever / never happens and if it does you can deal with it.

Click to expand...

Browser Verification
There are very good reasons to not believe the browser's HTTP user agent field, however most of the techniques look like browser detection code. This page reviews some passive methods that I'm familiar with.

HTTP Header Order
Many browser simulators do not use the same header order as the genuine article. It is useful to store this.

Firefox goes: Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Accept-Charset, Keep-Alive, Connection, Referer

Chrome goes: Host, Connection, Accept, User-Agent, Referer, Accept-Encoding, Accept-Language, Accept-Charset, Keep-Alive

Chrome used to go: Host, Connection, User-Agent, Accept, Referrer, ...

MSIE after version 6 goes: Accept, Referer, Accept-Language, User-Agent, Accept-Encoding, Host, Connection, Keep-Alive, Accept-Charset

MSIE version 6 goes: Accept, Referer, User-Agent, Host, Accept-Encoding, Accept-Language, Accept-Charset

MSIE version 6 sometimes goes: Accept, Connection, Host, Referer, User-Agent, Accept-Encoding, Accept-Language, Accept-Charset

Sometimes headers are missing. Some proxy servers drop/insert headers.

Click to expand...