Need to create a bot that scrapes websites
- Thread starter soheb123
- Start date
IYASS
Regular Member
- Jun 30, 2018
- 337
- 637
rasheemnotes2
Newbie
- Jun 16, 2020
- 45
- 4
Look at python beautiful soup
Chicilikit
Supreme Member
- Dec 21, 2010
- 1,298
- 401
I use ubot and I'm very happy with it, especially with exbrowser plugin.
greydingo
Junior Member
- Dec 31, 2014
- 152
- 73
Puppeteer is great for webscraping.
A lot of sites these days require you to have javascript enabled. So you need to automate an actual browser.
The docs are a good place to start. https://pptr.dev/
A lot of sites these days require you to have javascript enabled. So you need to automate an actual browser.
The docs are a good place to start. https://pptr.dev/
rauza
Regular Member
- Mar 13, 2014
- 409
- 588
You can utilize PHP if you want to do basic web scraping that involves grabbing HTML from web pages and analyzing them using String manipulation. You can also do this through Python using BeautifulSoup.
If you want to scrape pages that have Javascript being invoked on the page with dynamic web elements, I would recommend checking out Selenium. This can be run on many different languages but I recommend Java. It acts as a browser emulator which can interact with dynamic elements on a web page and give you more information than just basic HTML scraping.
If you want to scrape pages that have Javascript being invoked on the page with dynamic web elements, I would recommend checking out Selenium. This can be run on many different languages but I recommend Java. It acts as a browser emulator which can interact with dynamic elements on a web page and give you more information than just basic HTML scraping.
Levi Beaumont
Newbie
- Jan 29, 2020
- 21
- 6
I recommend Phantom JS
