1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Top C# (Headless) Web Automation Libraries and Toolkits?

Discussion in 'C, C++, C#' started by macdonjo3, Sep 1, 2014.

  1. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,559
    Likes Received:
    4,315
    Location:
    Toronto
    Home Page:
    I'm just starting to learn C#. I'm a Python dev. Previously I'd use Selenium with PhantomJS for headless browsing.

    I understand web requests in C# are pretty easy to do, but what options are there for headless browsing for a page that needs to execute javascript?

    There is the IE WebBrowser object that you can place outside the frame dimensions. Is that really the most professional way to go about this in C#?
     
  2. sashablack

    sashablack Elite Member

    Joined:
    Jan 8, 2010
    Messages:
    3,697
    Likes Received:
    2,050
    Gender:
    Male
    lol so I see you decided to make your own?

    -Sasha
     
  3. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,559
    Likes Received:
    4,315
    Location:
    Toronto
    Home Page:
    lol no, another project. deciding if i want to try a project in C# or not.

    I ended up just hiring someone on SEO clerks to scrape a bunch of UIDs since it was a 1-time thing and it was faster to do that.
     
  4. kimmlouis

    kimmlouis Newbie

    Joined:
    Nov 5, 2011
    Messages:
    28
    Likes Received:
    0
    Just use the default, WebBrowser Component.
     
  5. DarkPixel

    DarkPixel Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 4, 2011
    Messages:
    1,328
    Likes Received:
    1,239
    Location:
    ↓↓↓↓
    Home Page:
    Awesomium, GeckoFX, CEF and other third party and open source wrappers. Most of them are webkit based (of course excluding GeckoFX). I personally like the API of Awesomium.

    So, you have a bunch of choices for regular browsers, but not sure for headless ones, since I don't use headless ones.
     
    • Thanks Thanks x 1
  6. indianbill007

    indianbill007 Jr. VIP Jr. VIP

    Joined:
    Jan 8, 2010
    Messages:
    4,813
    Likes Received:
    4,051
    Occupation:
    Making Money when the world is sleeping
    Location:
    Menlo Park - Next to Zuck
    Most of these leak memory if you are planning to do anything at webscale.

    For small bot projects they should be ok, I like Awesomium over others, its a chrome wrapper in C# basically.
     
  7. Precubed

    Precubed Newbie

    Joined:
    Sep 4, 2014
    Messages:
    11
    Likes Received:
    1
    Occupation:
    Programmer, web developer, admin
    Home Page:
    Flash leaks memory. The web browser components themselves do not leak memory.
     
    • Thanks Thanks x 1
  8. indianbill007

    indianbill007 Jr. VIP Jr. VIP

    Joined:
    Jan 8, 2010
    Messages:
    4,813
    Likes Received:
    4,051
    Occupation:
    Making Money when the world is sleeping
    Location:
    Menlo Park - Next to Zuck
    OOO that reminds me flash is the shittiest piece of crap ever coded in the history of programming man kind (second place to IE ofc) and is still used so extensively all across the web.

    Steve Jobs took a visionary move by disallowing it on iOS and fucking them over for the good of html5.
     
    • Thanks Thanks x 2
    Last edited: Sep 4, 2014
  9. Precubed

    Precubed Newbie

    Joined:
    Sep 4, 2014
    Messages:
    11
    Likes Received:
    1
    Occupation:
    Programmer, web developer, admin
    Home Page:
    So true. Flash is terrabad. So is Java. Both for so many reasons. And there was so much noise about Apple not including Flash support. The general opinion at the time was that Apple was crazy for not including flash support. That move effected the opinions and the direction of the entire industry.
     
  10. DarkPixel

    DarkPixel Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 4, 2011
    Messages:
    1,328
    Likes Received:
    1,239
    Location:
    ↓↓↓↓
    Home Page:
    Woah woah. Java isn't bad. Java applets are bad. The JVM is very powerful, and the language is great as well.
     
  11. beatngu

    beatngu Registered Member

    Joined:
    Apr 22, 2011
    Messages:
    63
    Likes Received:
    26
    Location:
    Red Light District
    With C# there are tons of options,i like PhantomJS so why did you not use it in C# as well?
    As far as Awsomnium i really hate that pice of junk,i have used the past year to migrate all my projects away from the shitty Awesomnium because some sites has found a way to block it.
    The main problem was some sites was blocking awesomnium from uploading images and i simply found no way around it,with PhantomJS i managed to do it without any problem with injecting the site with a javascript.

    Not sure if there is a fix for awesomnium now but to be honest i never looked back after i dumped awesomnium because there are many other options to use.
    One example was uploading the images to ebay they where added but they never showed and the loading icon just keept on loading.
     
  12. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    614
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    If you know selenium and phantomJS stick with those. The ghostdriver is included in selenium so controlling phantomjs is a piece of piss out the box