Best Web scrapper and bot to program (Selenium, puppeteer, python...) ?

G

gandolfi

Newbie
Joined
Jan 9, 2010
Messages
41
Reaction score
5
Hello,

I am a computer scientist and I would like to use my own tools without having to go through subscription programs. I am want to program web scraper for my personnal use. For the moment i use google sheet with importxml but is very long and very limited.
I would like to learn about a tool or programming language that is powerful with multiple possibilities so that it can serve me for many years.

- I want to scraped website in html and/or with server side rendering js to import content on my website, csv or google sheet.
which program or programming language do you recommend?

I want also program a bot to auto create account (twitter, quora....) and post message with the created profile.
which program or programming language do you recommend?

thanks
 
Last edited:
Advertise on BHW
You have two options that will work for your use cases:

1 - Python
2 - NodeJS

Your choice really!
 
Use any form of Chrome automation: Selenium, Puppeteer etc. The trick to web scraping is for websites to get them to think you're human, and it's much easier if you're running a real browser (not headless).

Source: have several successful projects based on web scraping.
 
Thanks. For the moment i use puppeteer.
- Do you have advice to automate account creation on quora (captcha, proxy...) ?
- To create a lot of mail for account creation do you use nodejs or other program ?
 
gandolfi said:
Thanks. For the moment i use puppeteer.
- Do you have advice to automate account creation on quora (captcha, proxy...) ?
- To create a lot of mail for account creation do you use nodejs or other program ?
Click to expand...

Automate email creation part too, the part of knowing how to create accs properly needs a lot of testing with different proxies, blocking fingerprint, spoofing, etc
 
Selenium works well for Web scraping.
It supports multiple programming language also.
 
gandolfi said:
Thanks. For the moment i use puppeteer.
- Do you have advice to automate account creation on quora (captcha, proxy...) ?
- To create a lot of mail for account creation do you use nodejs or other program ?
Click to expand...

Do not attempt this as your first project. It's extremely hard to get this right and there are significant anti-bot measures.

Start with websites that aren't trying to block you actively, then grow into doing these types of projects. Then you will find the answers yourself as you gain more experience.
 
thanks for the advice.
so is it very difficult to applied this method ? https://www.blackhatworld.com/seo/method-earn-1000-daily-or-more-guarantee.1296513/unread

I'm getting back into the business and I'm looking for a way to generate 1000 dollars a month. I used to have websites for adults but for years it hasn't worked so well or it now requires a lot of money. I want to use simply method.
 
orangejuice said:
Use any form of Chrome automation: Selenium, Puppeteer etc. The trick to web scraping is for websites to get them to think you're human, and it's much easier if you're running a real browser (not headless).

Source: have several successful projects based on web scraping.
Click to expand...
Agreed, puppeteer is great. Just beware that puppeteer does generate some fingerprints of its own. Check out puppeteer stealth plugin to help alleviate some fingerprinting issues.
 
Jrim_Software said:
Agreed, puppeteer is great. Just beware that puppeteer does generate some fingerprints of its own. Check out puppeteer stealth plugin to help alleviate some fingerprinting issues.
Click to expand...

Yep, this is long cat and mouse game. Even with obvious finger prints removed, there are so many techniques to detect it: fonts installed, resolution, user-agent, rendering latency, how you handle cookies between new browser instances (i.e. do you retain them or are they always gone), plugins available (e.g. flash), operating system reported vs reality (e.g. claiming a Windows user agent but really it's running on Linux) how you respond to alerts/location requests etc etc.
 
If anyone knows i'm looking for a blackhat russian bot to program automation software it's named "balbo" or something like that. I cannot find any reference for it on google.
 
Automation isn't as easy as it used to be. You will have to get comfortable with multiple technologies depending on your specific use case.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top