In which technology web bots are made today

L

LCPGM

Newbie
Joined
Apr 14, 2017
Messages
13
Reaction score
2
Hello, I would like to ask you in which technology are web bots made today. I do not care about things related to UI, or web servers, but about actual technologies that scrape websites, fills it with data, etc.
Is Selenium still the leader in the market?
In which technology Jarvee or GramMultiTool works?
Thank you for your help, I am developing small projects for personal purposes, but I want something which will be the best ;)
 
Advertise on BHW
The main purpose of selenium is not scraping but testing websites.
The bots you mentioned like GMT works using HTTP requests (GET, POST...). If they would use selenium it would be unimaginably slow and would take a ton of resources.
To see how it works I recommend you to install Fidler and track the requests. You will see it is not that hard.
I recommend you to learn how to send HTTP requests in your programming language, once you do that everything will be easy.
 
Alex D. said:
The main purpose of selenium is not scraping but testing websites.
The bots you mentioned like GMT works using HTTP requests (GET, POST...). If they would use selenium it would be unimaginably slow and would take a ton of resources.
To see how it works I recommend you to install Fidler and track the requests. You will see it is not that hard.
I recommend you to learn how to send HTTP requests in your programming language, once you do that everything will be easy.
Click to expand...
Thank you for your answer.
Okay, i got it, but what about some javascript trackers etc, which may appear on site? I heard that direct API calls are dangerous, but if I am getting it right I have to somehow fake responses for these trackers and other stuff. Am I right?
 
You will need to simulate it, don't get yourself tricked because selenium is also easily detectable since it adds webdriver="true" tag.
 
Alex D. This can easily be bypassed so it does not set navigator.webdriver to true.
 
If the webpage uses a lot of javascript then selenium is by far one of the easiest. If the website doesn't then you should use http requests, for example in c# or java you can use httpclient.
 
There is a relatively new (couple years) thing on the market called headless chrome which is quite popular for those purposes.
 
As others said : if the website you target generate content using javascript, you will easily do it using selenium which is a library with you can control an instance of chrome or firefox. If the website use protection against scraping, you may want to set certains navigator variables manually and the user agent mannualy. If the website is a static one (it doesnt generate html thought scripts) you can scrap it by using a simple http request. Good luck !
 
If you mean automation of routine tasks , then selenium or zennoposter. But if u mean for example search bots which uses google, then http protocol requests.
Regarding the automation of routine tasks, - selenium + python is a perfect tool and the syntax is simple. Everything is built on teams javascript. sorry for my English
 
Yeah, I use Python and Elixir for all my scraping. Multithreading and scalability are important.

Selenium is a fine tool but headless is better for testing.
 
Requests and puppeteer and nodejs are powerfull tools and valuable skills selenium is easily detectable and pyautogui is trash tbh i heard u can make bots with .net framework too so requests and puppeteer are the best
 
selenium is no more detectable than puppeteer with chromeheadless. Both CAN be detectable. Both can mitigate that
 
Interesting topic!
Here is my go to move with Python for Web automation & Web scraping.

API = requests

Simple website = requests with beautifulsoup
JavaScript generated websites = Scrapy with scrapy-splash

Simple form submits = Scrapy
Lots of form = Selenium

I am still learning & experimenting with these tools.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top