How do I get started on web automation?

D

Descension

Newbie
Joined
May 3, 2015
Messages
3
Reaction score
0
Hello!


I want to get into learning web automation through a Linux Server

I would like to be able to test on my windows computer and be able to run it on my Linux server.

When I say web automation, I mean like being able to click on things on websites, like a normal browser.


I have been reading a bit about Selenium and that it might be useful for headless web browsing.

I know java programming and I am willing to learn a new language.


What is the best way to do web automation from a Linux server?

What tools can I use?

What programming language is required to know for the most effective web automation?


Thank you for your time!
 
Advertise on BHW
Hi,
"When I say web automation, I mean like being able to click on things on websites, like a normal browser."
a linux server don't run any kind of web browser so you can browse or click with a software, the best you can do in a linux server is send http post / get call to urls but will not be very effective
 
I was thinking about headless browsing. Is that not possible on a linux server? I know you won't be able to see the clicking but I could imagine it would still do it.

Do you think selenium could work on linux?
 
tvm77 said:
...a linux server don't run any kind of web browser so you can browse or click with a software, the best you can do in a linux server is send http post / get call to urls but will not be very effective...
Click to expand...

LMAO. It's funny how people who have absolutely no idea are always the first to give advice.

Descension said:
...
What is the best way to do web automation from a Linux server?
What tools can I use?
What programming language is required to know for the most effective web automation?
Do you think selenium could work on linux?
...
Click to expand...

There are many great ways to implement web automation from a Linux box and using Selenium with Java is certainly one option for both normal and headless browsers. There is also CasperJS (http://casperjs.org/) which is a navigation scripting & testing utility written in JavaScript. You can find a full list of available headless browsers along with their supported language here: https://github.com/dhamaniasad/HeadlessBrowsers.
 
Like MrBlue said... Casperjs or Java/Selenium. I have had good luck with both. casper is easier but perhaps not quite as flexible as something driving selenium.
 
Java + Selenium is the way to go :)
You can also do headless automation wiht selenium on Linux.
 
starfish89 said:
Java + Selenium is the way to go :)
You can also do headless automation wiht selenium on Linux.
Click to expand...
Linux is the best way to bot period. So much more functionality then windows. And yeap, firefox + chrome headless sounds amazing. Can't wait to start using them. If only I could work out how to scrape pages with Selenium *shrugs*. Oh well I suppose I'll look at 500+ pages on stackoverflow and then realise I was over complicating things... xD
 
I'm on linux and for now using Node.js for this. There are many different modules you can use. For maximum user imitation you can of course use selenium, or other headless like phantomJs.
But if you want more speed and scrape large amount of data it would be too slow. Better to use raw requests then.
 
Learning full requests is something I have been putting off for so long. I feel kind of ashamed being two years into botting and not know how to do it. I only did it for simlple scraping scripts.

I just havent had the time and luxury of good income to learn it properly though.
 
Are there any alternatives to selenium? I've come across problems with selenium such as when it's time to fill in recaptcha. Selenium will not let me fill in hidden form fields because being that it's a testing framework it does not allow anyone to do stuff a human cannot do.

I've heard of phantomJS however that's always invisible so when you're first using it or trying to debug an action gone wrong that's a bit of a complication. Also heard of CEF but the nuget CEFsharp package just gives one an embeddable browser. I know that ubot uses CEF so I'd be very interested if anyone has any idea on how to get CEF setup for web automation either in linux or in windows.
 
Having a Java base is great. I'd say just learn how HTTP requests work and do it in straight up Java.

This is always my advice though; keep using the language you know to do a new thing you want to do.

It's always easier to learn to do a new thing in a language you know. So when you run into problems, you are already an expert at debugging. Whereas if you make the bot in a new language and have a problem, you'll have to figure out if it's stemming from being bad at the language, or making the bot incorrectly.
^ That paragraph is terribly worded, I hope it makes sense lol.

Take a look at the Java documentation of HttpURLConnection. And for threading I like to use Executors.

If you're planning on selling/distributing your final product though, Java is easy to reverse engineer. But you probably already knew that.
 
Python selenium phantomjs

Put the above in google and practice the tutorials. Trust me you will be up and running in no time.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top