1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What do you use to simulate browser activity in C#

Discussion in 'C, C++, C#' started by Taegn, Nov 25, 2016.

  1. Taegn

    Taegn Junior Member

    Joined:
    Jul 22, 2016
    Messages:
    178
    Likes Received:
    32
    Selenium PhantomJS seems to be the best.
     
  2. gsy159

    gsy159 Power Member

    Joined:
    Apr 29, 2011
    Messages:
    684
    Likes Received:
    162
    Everybody has its favourite but yeah selenium works best for me too
     
  3. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    selenium.
     
  4. ZennoScripts

    ZennoScripts Junior Member

    Joined:
    Aug 21, 2015
    Messages:
    198
    Likes Received:
    70
    Location:
    USA
    Selenium is amazing but the underlying FireFox Build is garbage. Everyone and their mom knows how to detect a browser that is running off a FireFox ESR build which is how selenium even works at all.

    It works for 80% of sites but for the other 20% you'll need to use a Chrome based browser automation platform.
     
  5. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    Really? Are you not exaggerating slightly? Have read that phantom is easy to detect without question (although can be countered via JS). But not that FF is more detectable than chrome. Studies into detecting selenium I have read have detected chrome and ff equally well.

    Unless there is something new (haven't looked into it for a while) but googling found nothing and if "everyone and their mom" knows, then it would be easy to find.
     
  6. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,398
    Likes Received:
    636
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    Most automation can be detected now. Distil networks and many others detect selenium and phantom through js and header analysis. Having said that you'll probably find whatever you're planning on automating will ban you based on activity rather than browser/software.
     
  7. thetrustedzone

    thetrustedzone Jr. VIP Jr. VIP

    Joined:
    Jun 15, 2010
    Messages:
    2,500
    Likes Received:
    2,028
    Home Page:
    You could try fiddler .Net also .......
     
  8. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,652
    Likes Received:
    11,317
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Correct me if I'm wrong, but doesn't selenium use a real browser?

    Also you can set all the headers on PhantomJS, if you do it carefully you can fully mimic Firefox or Chrome. Just capture some legit FF or Chrome traffic and apply the same headers.

    The issue I see with PhantomJS is it crashes all the time.
     
  9. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,398
    Likes Received:
    636
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    Correct it does use a real browser but I believe there are various js hooks that can be enumerated
     
  10. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,652
    Likes Received:
    11,317
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    So client JS on web pages can check for selenium js hooks to see whether the browser is being run automated?
     
  11. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,398
    Likes Received:
    636
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    That's my understanding of it yes :)
     
    • Thanks Thanks x 1
  12. ZennoScripts

    ZennoScripts Junior Member

    Joined:
    Aug 21, 2015
    Messages:
    198
    Likes Received:
    70
    Location:
    USA
    You guys are close but not quite, Firefox has different versions one being ESR (Extended release) and release (mainstream firefox). They are practically the same but ESR is always one version behind and with each ESR version say the "ZennoPoster team" has to update their bot to manipulate the new Firefox ESR build.

    As Google and other website are fixing up their spam filters they are detecting and actively blocking Firefox ESR builds/other hardcoded footprints by selenium/phantomJS/Zennoposter/Ubot/etc.

    The good old fashioned cat/mouse game.

    Maybe @nuaru can explain this in simpler terms?
     
    • Thanks Thanks x 1
  13. SethTurin

    SethTurin Jr. VIP Jr. VIP

    Joined:
    Nov 19, 2011
    Messages:
    288
    Likes Received:
    152
    Occupation:
    President and CEO of Seth Turin Media, Inc. - Crea
    Home Page:
    If you have UBot Studio, there is always a group of folks active and talking on skype about using intelligent methods to avoid detection. Sometimes for UBot Studio, sometimes for something else. You will want to consider joining.

    In general, I would worry less about what will be detectable and what you will actually be able to do. Selenium seems easiest for new users (besides UBot Studio obviously =D) if you're building an automation engine from scratch. Not sure why people would do that tho...
     
  14. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    Phantom is easy to detect, there are a lot of details online of how to do it. It uses / or used a very old version of webkit and the headers were sent in a unique way.

    Also you could crash the JS and get phantomjs strings, you could detect specific JS properties.

    Firefox / chrome on the other hand use the INSTALLED browser on your machine. So headers are sent exactly as the browser. selenium firefox = normal firefox.

    As for JS detection, I am not aware of any. There are no exposable selenium JS code to detect, so I don't think it is as easy as it is made out to be above. I would guess someone is passing on 2nd hand information as opposed to personal experience.

    But as stated some companies are detecting selenium with a high probabilty. Distil being one of them. I read about this briefly and no one is really sure. They talk of taking a finger print of the browser and behaviour. Perhaps timings, perhaps JS events. When you call a mouse event in JS, did you trigger of other mouse events expected to occur before?

    If your mouse clicks a button, there should be lots of DOM enter / exit / hover events. Could be a metric. Bypassable by automating a real mouse though


    However, perhaps is possible but certainly is not simple that every one and their grandma knows. That is false - until I am shown evidence to the contary.

    For automation of a browser your best bet is still selenium with firefox / chrome.
     
  15. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    No you cannot set the header order. You can set the headers, but when they are sent, in what order you cannot.

    examples (not real but the idea of the problem)

    i.e. firefox would send:
    useragent: xxxx

    phantom sends
    Useragent:

    firefox sends

    useragent:
    cookies

    whereas phantom sends
    cookies
    useragent

    Easy to identify


    The way around this is to proxy the calls, then modify the header order in the proxy. I Cannot remember the tools you could use (was a while ago I looked) but maybe was haproxy, proxifier?? might be wrong there tho
     
  16. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    That may be how zenno works, but that isn't how selenium works. You do not have to use the ESR build. It will use the installed version of the machine.

    install firefox v45 and selenium will use v45. Install v40 and selenium will use that,

    Since v45 you would need to use marionette as FF broke the ffwebdriver. FF are pushing forward with teh wire protocol (how selenium works) and not waiting for selenium to keep up.
     
  17. BrPimp

    BrPimp Newbie

    Joined:
    May 2, 2015
    Messages:
    47
    Likes Received:
    16
    Location:
    Brazil
    As most here I use PhantomJS when I need a headless browser. There is others (not headless) like Imacros .Net Component, CefSharp and Selenium.

    PhantomJS could be easily detected, here is one test that I did using Fiddler to gather information. Only when using PhantomJS Tumblr sends me that function to verify my browser.

    [​IMG]
     
  18. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    There really are no quality headless browsers if you want to automate websites, they all have their problems.

    The best you are going to do is selenium running with XVFB which will appear headless. You can also give it it's own mouse to automate and containerise it to run multiple instances on a single machine
     
  19. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,652
    Likes Received:
    11,317
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    I see. But checking header order is a really fragile method, cuz you can install addons to Firefox that change the headers. If sites started filtering based on this, it would break a lot of browsers.
     
  20. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    685
    Likes Received:
    329
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    How many people run plugins to randomise the header order?

    But as I said, the above was from memory of the "type" of things you can do to detect phantomJS. There are a few more and you can with 100% accuracy detect it (according to the reports). Some are protectable with JS coding to remove, but things like header order would be recompiles of phantomjs or webkit and I think some old QT they are / were using.

    The point being, if you want to be undetectable in your automations, phantom is not really the way to do. If you are running tests on your website it might make sense, but what with phantom bugs you would be better with selenium