Scraping with python issue
- Thread starter Pacific12
- Start date
-
- Tags
- scraping web scraping
Liddells Doll
Newbie
- Nov 30, 2022
- 11
- 5
Websites should have a robots.txt file that states what resources can be scraped by bots among other things.
https://www.google.com/robots.txt
https://en.wikipedia.org/robots.txt
https://www.google.com/robots.txt
https://en.wikipedia.org/robots.txt
- Oct 24, 2022
- 1,135
- 1,378
I agree with this
TheToolbox
Elite Member
- Jul 25, 2021
- 1,568
- 643
Yes in robots.txt are the permissions named
- May 2, 2010
- 9,937
- 16,034
If you use proxies, you should also write a function to take your master IP address and compare it each time with the proxy IP before doing an action. Only if the master_ip != proxy_ip, then you continue. Just to be on the safe side and allow you to sleep easy!
- Jul 7, 2018
- 126
- 26
I suggest you take a look at the presentation by Sanaea Daruwalla, Chief Legal Officer at Zyte, at the Web Data Extraction Summit of 2022.
Video:
Slides: https://info.zyte.com/hubfs/Extract Summit 2022 slides/Extract Summit 2022 - Sanaea Daruwalla.pdf
From what I recall from the presentation, it is suggested to follow the policies of websites, but if you do not explicitly agree to the policies (e.g., on sign-up), it is hard for them to be enforced. (See slide 5)
All videos of the Web Data Extraction Summit: https://www.extractsummit.io/2022-videos/
Video:
Slides: https://info.zyte.com/hubfs/Extract Summit 2022 slides/Extract Summit 2022 - Sanaea Daruwalla.pdf
From what I recall from the presentation, it is suggested to follow the policies of websites, but if you do not explicitly agree to the policies (e.g., on sign-up), it is hard for them to be enforced. (See slide 5)
All videos of the Web Data Extraction Summit: https://www.extractsummit.io/2022-videos/
