Script that will scrape other websites in specific niche and update my website with that information automatically.
- Sep 10, 2010
- 11,805
- 26,657
With a simple scraper? This can be done with node.js/python.
If you are looking for a freelancer, post a thread in hire a freelancer section.
If you are looking for a tutorial, it depends on how complicated the scraping job is. If it is not that difficult (no login, info is available in the html itself), it could probably be done with a simple fetch/axios request. For more complicated tasks, you could use puppeteer. There are lots of tutorials available for both on YT. Just search for the keywords.
If you are looking for a freelancer, post a thread in hire a freelancer section.
If you are looking for a tutorial, it depends on how complicated the scraping job is. If it is not that difficult (no login, info is available in the html itself), it could probably be done with a simple fetch/axios request. For more complicated tasks, you could use puppeteer. There are lots of tutorials available for both on YT. Just search for the keywords.
EvoCreatix
BANNED
- Apr 16, 2023
- 1,968
- 172
Hi! can you help me with some information!
d11
Regular Member
- Sep 1, 2022
- 225
- 56
yes but I want it to scrape and then fully automatically post scraped info to my website.
Also Ive tried to find some information already but unfortunately any success yet, so decided to post here.
- Sep 10, 2010
- 11,805
- 26,657
You just need a mysql query to insert the scraped data to the database.. That is it.
What language are you using to scrape the data? Google how to make a mysql insert (or better yet, a insert ignore/replace into query if the data can have duplicates) query in that language. Node has mysql2 for example. It should be pretty straightforward. Be sure to back everything up first though.
d11
Regular Member
- Sep 1, 2022
- 225
- 56
I am not using any language for scraping atm, just asking now.
Thank you for your replies btw
- Sep 22, 2013
- 1,209
- 1,670
hiscariote
Newbie
- Sep 8, 2019
- 17
- 9
Are you a developer?
Depends which website you want to scrap.
If you want to scrap wordpress, I highly recommend you use the API.
Check this URL: https://developer.wordpress.org/rest-api/reference/pages/ for scrap pages and /posts/ for getting posts.
W/ this you only need fetch(javascript) or beautifulsoup(python). now depends what you want. I recommend you after scraping, doing some post-processing of the data.
I am currently doing like this with fetch.
Depends which website you want to scrap.
If you want to scrap wordpress, I highly recommend you use the API.
Check this URL: https://developer.wordpress.org/rest-api/reference/pages/ for scrap pages and /posts/ for getting posts.
W/ this you only need fetch(javascript) or beautifulsoup(python). now depends what you want. I recommend you after scraping, doing some post-processing of the data.
I am currently doing like this with fetch.
shaymiller
Regular Member
- Dec 3, 2016
- 456
- 45
ranahashim620
Newbie
- Oct 17, 2024
- 25
- 4
You'll definitely need to look into python with libraries like beautifulsoup or scrapy to pull that data, but honestly, be super careful about copyright issues and keeping your site from looking like total spam. It’s a bit of a techy rabbit hole, so maybe start by checking out some beginner tutorials on web scraping to see if you can handle the coding side of things
- Aug 2, 2011
- 253
- 91
If you are using WordPress, there are already some plugins that can scrape some websites like Amazon, etc., and publish the content on your website. If you need some help, be specific so we can try to help. If you need to hire someone, then we have a different section in the forum.
shaymiller
Regular Member
- Dec 3, 2016
- 456
- 45
can you do this using claude or other ai tools?