I just joined premium forum for 1 month. I want to scrape their content for later because my subscription is only 1 month.the title said everything
Is there any way I can scrape all content including attachments and read offline?
Thank you.
ASK ME ANYTHING about web automation | web scraping
- Thread starter SeasonedCode
- Start date
-
- Tags
- python
Is there any way I can scrape all content including attachments and read offline?
Thank you.
- Oct 24, 2022
- 1,132
- 1,375
yeah you can build a scraper to do this
I mean, do you have any experience with this?
Is there an easy tool to scrape forum posts?
- Oct 24, 2022
- 1,132
- 1,375
I don't know any tool you need to hire a dev to build for you what you want
middeen2004
Registered Member
- May 17, 2021
- 55
- 25
It seems I didn't try use it without custom settingsdelete all this and write this
import undetected_chromedriver as uc
driver = uc.chrome()
# your code here
and that's set you are don't if you run this code a new Chrome will open and you can log in with it on Google
Thanks for you I really appreciate it
Minos9
Newbie
- Feb 9, 2023
- 17
- 17
Yea, you can send me the url and i'll take a look on it
johnpham
Regular Member
- Aug 18, 2021
- 334
- 179
Here is an example that I can quote: searchenginejournal.com
If it violates the regulations of BHW, please let the management know
Minos9
Newbie
- Feb 9, 2023
- 17
- 17
Here is 20 pages of the news sections with article's title, url, description and publish date
https://gofile.io/d/2VzRC7
https://www.virustotal.com/gui/file/45d3f039fd6f740a6924f003c6e79427901f353e64965e85bbdd40b2ca1d1d83?nocache=1
lostbadger
Registered Member
- Feb 23, 2022
- 77
- 16
Yeah, its better if you request a custom script as you won't find something pre-made for that specific forum. Be aware
of everything involved like captchas (I use Death By Captcha) and proxies (Storm, or any other with good quality).
FabioSelau
Senior Member
- Oct 19, 2018
- 820
- 452
johnpham
Regular Member
- Aug 18, 2021
- 334
- 179
Thanks, mate, but I want to ask about the method. I don't want the result.
Minos9
Newbie
- Feb 9, 2023
- 17
- 17
You need to build a scraper . The method can be implemented by using a programming language and tools that send a request , parse the data and extract them. You can look into selenium, bs4 , puppeteer
robotic301
Newbie
- Aug 26, 2021
- 23
- 10
Can you download images from this site? https://www.peakpx.com/
I have tried to make an automation tool to download images from the site, but it seems that the site using google captcha v2 to prevent me. I can download some images and after that, the tool will not work
I have tried to make an automation tool to download images from the site, but it seems that the site using google captcha v2 to prevent me. I can download some images and after that, the tool will not work
Minos9
Newbie
- Feb 9, 2023
- 17
- 17
I was able to get images no problem, get the urls first and then loop through them. What tools do you use?
robotic301
Newbie
- Aug 26, 2021
- 23
- 10
I using jsoup and selenium, and both of them are not working. My tool can only download 20-30 images (I mean original images, not thumbnail), and after that the site block my tool, and the image downloaded will be wrong.
- Jul 7, 2018
- 126
- 26
Highly recommend taking a look at the following presentation by Sanaea Daruwalla, Chief Legal Officer at Zyte:
- Oct 24, 2022
- 1,132
- 1,375
there is nothing to do about this
unless you stop using browsers and start sending requests directly to the server but you need to know how to get those private APIs, handle authentication and more...