1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How do I get started on web automation?

Discussion in 'General Programming Chat' started by Descension, May 18, 2016.

  1. Descension

    Descension Newbie

    Joined:
    May 3, 2015
    Messages:
    3
    Likes Received:
    0
    Hello!


    I want to get into learning web automation through a Linux Server

    I would like to be able to test on my windows computer and be able to run it on my Linux server.

    When I say web automation, I mean like being able to click on things on websites, like a normal browser.


    I have been reading a bit about Selenium and that it might be useful for headless web browsing.

    I know java programming and I am willing to learn a new language.


    What is the best way to do web automation from a Linux server?

    What tools can I use?

    What programming language is required to know for the most effective web automation?


    Thank you for your time!
     
  2. tvm77

    tvm77 Jr. VIP Jr. VIP

    Joined:
    Nov 14, 2013
    Messages:
    177
    Likes Received:
    72
    Home Page:
    Hi,
    "When I say web automation, I mean like being able to click on things on websites, like a normal browser."
    a linux server don't run any kind of web browser so you can browse or click with a software, the best you can do in a linux server is send http post / get call to urls but will not be very effective
     
  3. Descension

    Descension Newbie

    Joined:
    May 3, 2015
    Messages:
    3
    Likes Received:
    0
    I was thinking about headless browsing. Is that not possible on a linux server? I know you won't be able to see the clicking but I could imagine it would still do it.

    Do you think selenium could work on linux?
     
  4. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    974
    Likes Received:
    680
    Occupation:
    Web/Bot Developer
    LMAO. It's funny how people who have absolutely no idea are always the first to give advice.

    There are many great ways to implement web automation from a Linux box and using Selenium with Java is certainly one option for both normal and headless browsers. There is also CasperJS (http://casperjs.org/) which is a navigation scripting & testing utility written in JavaScript. You can find a full list of available headless browsers along with their supported language here: https://github.com/dhamaniasad/HeadlessBrowsers.
     
    • Thanks Thanks x 4
  5. kahuna74

    kahuna74 Regular Member

    Joined:
    Aug 19, 2014
    Messages:
    270
    Likes Received:
    102
    Gender:
    Male
    Occupation:
    Software Developer
    Location:
    Grand Rapids, MI
    Like MrBlue said... Casperjs or Java/Selenium. I have had good luck with both. casper is easier but perhaps not quite as flexible as something driving selenium.
     
    • Thanks Thanks x 2
  6. starfish89

    starfish89 Newbie

    Joined:
    Sep 4, 2013
    Messages:
    5
    Likes Received:
    3
    Java + Selenium is the way to go :)
    You can also do headless automation wiht selenium on Linux.
     
  7. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Linux is the best way to bot period. So much more functionality then windows. And yeap, firefox + chrome headless sounds amazing. Can't wait to start using them. If only I could work out how to scrape pages with Selenium *shrugs*. Oh well I suppose I'll look at 500+ pages on stackoverflow and then realise I was over complicating things... xD
     
  8. Sergey Yermak

    Sergey Yermak Newbie

    Joined:
    Mar 12, 2016
    Messages:
    30
    Likes Received:
    3
    Gender:
    Male
    Occupation:
    Node.js
    I'm on linux and for now using Node.js for this. There are many different modules you can use. For maximum user imitation you can of course use selenium, or other headless like phantomJs.
    But if you want more speed and scrape large amount of data it would be too slow. Better to use raw requests then.
     
  9. PineappleCrush

    PineappleCrush Regular Member

    Joined:
    Jan 20, 2016
    Messages:
    340
    Likes Received:
    27
    Learning full requests is something I have been putting off for so long. I feel kind of ashamed being two years into botting and not know how to do it. I only did it for simlple scraping scripts.

    I just havent had the time and luxury of good income to learn it properly though.
     
  10. ThatWeb20Guy

    ThatWeb20Guy Registered Member

    Joined:
    Aug 21, 2017
    Messages:
    68
    Likes Received:
    13
    Are there any alternatives to selenium? I've come across problems with selenium such as when it's time to fill in recaptcha. Selenium will not let me fill in hidden form fields because being that it's a testing framework it does not allow anyone to do stuff a human cannot do.

    I've heard of phantomJS however that's always invisible so when you're first using it or trying to debug an action gone wrong that's a bit of a complication. Also heard of CEF but the nuget CEFsharp package just gives one an embeddable browser. I know that ubot uses CEF so I'd be very interested if anyone has any idea on how to get CEF setup for web automation either in linux or in windows.
     
  11. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    974
    Likes Received:
    680
    Occupation:
    Web/Bot Developer
    Try NightmareJS http://www.nightmarejs.org/
     
  12. ExtremeRandom

    ExtremeRandom Registered Member

    Joined:
    Jul 11, 2013
    Messages:
    79
    Likes Received:
    15
    I was actually looking for something similar and this was helpful thanks