In which technology web bots are made today

[LCPGM]

Hello, I would like to ask you in which technology are web bots made today. I do not care about things related to UI, or web servers, but about actual technologies that scrape websites, fills it with data, etc.
Is Selenium still the leader in the market?
In which technology Jarvee or GramMultiTool works?
Thank you for your help, I am developing small projects for personal purposes, but I want something which will be the best

 

[Alex D.]

The main purpose of selenium is not scraping but testing websites.
The bots you mentioned like GMT works using HTTP requests (GET, POST...). If they would use selenium it would be unimaginably slow and would take a ton of resources.
To see how it works I recommend you to install Fidler and track the requests. You will see it is not that hard.
I recommend you to learn how to send HTTP requests in your programming language, once you do that everything will be easy.

 

[LCPGM]

The main purpose of selenium is not scraping but testing websites.
The bots you mentioned like GMT works using HTTP requests (GET, POST...). If they would use selenium it would be unimaginably slow and would take a ton of resources.
To see how it works I recommend you to install Fidler and track the requests. You will see it is not that hard.
I recommend you to learn how to send HTTP requests in your programming language, once you do that everything will be easy.

Thank you for your answer.
Okay, i got it, but what about some javascript trackers etc, which may appear on site? I heard that direct API calls are dangerous, but if I am getting it right I have to somehow fake responses for these trackers and other stuff. Am I right?

 

[Alex D.]

You will need to simulate it, don't get yourself tricked because selenium is also easily detectable since it adds webdriver="true" tag.

 

[Scrapax]

Alex D. This can easily be bypassed so it does not set navigator.webdriver to true.

 

[Scrapax]

If the webpage uses a lot of javascript then selenium is by far one of the easiest. If the website doesn't then you should use http requests, for example in c# or java you can use httpclient.

 

[litportnet]

There is a relatively new (couple years) thing on the market called headless chrome which is quite popular for those purposes.

 

[Kissmyskunk]

As others said : if the website you target generate content using javascript, you will easily do it using selenium which is a library with you can control an instance of chrome or firefox. If the website use protection against scraping, you may want to set certains navigator variables manually and the user agent mannualy. If the website is a static one (it doesnt generate html thought scripts) you can scrap it by using a simple http request. Good luck !

 

[FB-sell]

If you mean automation of routine tasks , then selenium or zennoposter. But if u mean for example search bots which uses google, then http protocol requests.
Regarding the automation of routine tasks, - selenium + python is a perfect tool and the syntax is simple. Everything is built on teams javascript. sorry for my English

 

[deadalus1212]

Yeah, I use Python and Elixir for all my scraping. Multithreading and scalability are important.

Selenium is a fine tool but headless is better for testing.

 

[ahmedmani]

Requests and puppeteer and nodejs are powerfull tools and valuable skills selenium is easily detectable and pyautogui is trash tbh i heard u can make bots with .net framework too so requests and puppeteer are the best

 

[rootjazz]

selenium is no more detectable than puppeteer with chromeheadless. Both CAN be detectable. Both can mitigate that

 

[ahmedmani]

selenium is no more detectable than puppeteer with chromeheadless. Both CAN be detectable. Both can mitigate that

Really? I read somewhere that you run puppeteer on a "headed" mode

 

[S.t.j.i.t]

Interesting topic!
Here is my go to move with Python for Web automation & Web scraping.

API = requests

Simple website = requests with beautifulsoup
JavaScript generated websites = Scrapy with scrapy-splash

Simple form submits = Scrapy
Lots of form = Selenium

I am still learning & experimenting with these tools.