Help with making a scraping tool

[yardyknow]

Hey yall, I wanna make a scraping tool for a website with peoples information on it, however I dont know where to start, what language do i have to learn for this?

 

[ozanthedeveloper]

The easiest way to learn scraping is by learning Python and Selenium. But if you wanna scrape static information you can just use an HTML parser library like beautifulsoup4 and a basic GET request to get page source. It depends on what kind of website you wanna scrape data from.

 

[Sickfly]

Exactly like this

The easiest way to learn scraping is by learning Python and Selenium. But if you wanna scrape static information you can just use an HTML parser library like beautifulsoup4 and a basic GET request to get page source. It depends on what kind of website you wanna scrape data from.

 

[ygg]

There are drag and drop softwares to automate browser tasks, like BAS (free)

 

[BobbyBummbleBee]

There are a lot of options and a bunch of different schools of thought. Here are a few ideas.https://www.reddit.com/r/webscraping/
(https://www.reddit.com/r/webscraping/)

Why Python?
Straightforward syntax and an a bunch of libraries tailored to and for scraping (BeautifulSoup (BS4) and Requests). BS4 and Requests can do a ton... again know your needs. BeautifulSoup simplifies HTML parsing, while Requests is key for making HTTP requests and fetching webpage data.

The Ultimate Guide To Building a Web Scraper With Pyppeteer:​

(https://scrapeops.io/python-web-scraping-playbook/python-selenium-vs-pyppeteer/)

Selenium vs Python Pyppeteer for Web Scraping:​

(https://scrapeops.io/python-web-scraping-playbook/python-selenium-vs-pyppeteer/)

Pyppeteer vs. Selenium:

Each tool has its own fans/followers for automating browser actions and scraping dynamic content.

• Pyppeteer: A Python adaptation of Puppeteer, it operates using headless Chrome and is known for its speed, especially on JavaScript-heavy sites.
• Selenium: Offers more flexibility with browser support and language options, better suited for complex browser interactions but generally slower than Pyppeteer.

Headless Browsers:
Headless browsers allow for scraping dynamic content.

Options:
puppeteer-extra-plugin-stealth (https://www.npmjs.com/package/puppeteer-extra-plugin-stealthhttps://www.npmjs.com/package/puppeteer-extra-plugin-stealth)
Pyppeteer Stealth (https://github.com/MeiK2333/pyppeteer_stealth)
Playwright for Python (https://github.com/microsoft/playwright-python) with a stealth plugin (https://pypi.org/project/playwright-stealth/)

This will get you in the right direction.

 

[JLR]

... ^^^ I have one. I love it. LOVE IT.

 

[DevWaveJourney]

Hey yall, I wanna make a scraping tool for a website with peoples information on it, however I dont know where to start, what language do i have to learn for this?

I Think HTML and Python, help you a lot.

 

[NulledCode]

Have a look at Crawlee.

 

[Bonvi]

There are a lot of options and a bunch of different schools of thought. Here are a few ideas.https://www.reddit.com/r/webscraping/
(https://www.reddit.com/r/webscraping/)

Why Python?
Straightforward syntax and an a bunch of libraries tailored to and for scraping (BeautifulSoup (BS4) and Requests). BS4 and Requests can do a ton... again know your needs. BeautifulSoup simplifies HTML parsing, while Requests is key for making HTTP requests and fetching webpage data.

The Ultimate Guide To Building a Web Scraper With Pyppeteer:​

(https://scrapeops.io/python-web-scraping-playbook/python-selenium-vs-pyppeteer/)

Selenium vs Python Pyppeteer for Web Scraping:​

(https://scrapeops.io/python-web-scraping-playbook/python-selenium-vs-pyppeteer/)

Pyppeteer vs. Selenium:

Each tool has its own fans/followers for automating browser actions and scraping dynamic content.

• Pyppeteer: A Python adaptation of Puppeteer, it operates using headless Chrome and is known for its speed, especially on JavaScript-heavy sites.
• Selenium: Offers more flexibility with browser support and language options, better suited for complex browser interactions but generally slower than Pyppeteer.

Headless Browsers:
Headless browsers allow for scraping dynamic content.

Options:
puppeteer-extra-plugin-stealth (https://www.npmjs.com/package/puppeteer-extra-plugin-stealth)
Pyppeteer Stealth (https://github.com/MeiK2333/pyppeteer_stealth)
Playwright for Python (https://github.com/microsoft/playwright-python) with a stealth plugin (https://pypi.org/project/playwright-stealth/)

This will get you in the right direction.

yoink

 

[nicogd93]

Javascript its a good option too and easy to learn imo. you can do several things

 

[spicymelon]

Hey yall, I wanna make a scraping tool for a website with peoples information on it, however I dont know where to start, what language do i have to learn for this?

If you are feeling adventurous and have time, use Golang.

 

[Nikolaos]

Python is the go to language for everything bot related.

 

[Celphtitled]

Why do people entertain low-effort posts like these that could be solved by OP with literally one Google search?
I mean, it would be entirely different if it was a very specific snippet of code etc.

 

[Bonvi]

Why do people entertain low-effort posts like these that could be solved by OP with literally one Google search?
I mean, it would be entirely different if it was a very specific snippet of code etc.

Dunno. Why not? It doesn't cost anything. And google doesn't always have the best answer on things.

 

[Arpito]

It’s not hard to build. I have my own web scrapper and tg scrapper + adder I’m more worried about in not getting banned my accounts haha