- Nov 4, 2023
- 599
- 159
I want to keep the orignal html , only remove the ad block
Is there any existing solution for this purpose?
Is there any existing solution for this purpose?
Is there any free scrape package/too etc in python which can crawle a wordpress site and skip the ad block
- Thread starter social_newbie
- Start date
Not sure of a pre built library... Newspaper is ok from Python but it hasn't been updated in years so YMMV.
Most WordPress sites don't disable access to the wp-json V2 API so if you want to get rid of a lot of website fluff be and make less requests to the server, just put
wp-json/wp/v2/posts (or pages)
At the end of the homepage URL and you'll be able to see everything. You can read more about it following the WordPress WP Json API documentation.
Hope this helps.
Most WordPress sites don't disable access to the wp-json V2 API so if you want to get rid of a lot of website fluff be and make less requests to the server, just put
wp-json/wp/v2/posts (or pages)
At the end of the homepage URL and you'll be able to see everything. You can read more about it following the WordPress WP Json API documentation.
Hope this helps.
- Nov 4, 2023
- 599
- 159
Thanks very much, will try later on
D
Deleted member 752298
Guest
Yes, there is a python scraper available with filter functions:
https://github.com/MickaelWalter/wp-json-scraper
https://github.com/MickaelWalter/wp-json-scraper
- Nov 4, 2023
- 599
- 159
Thanks, this is also useful
hostonetech
Newbie
- Aug 26, 2023
- 22
- 4
you should check scrapy module for python. just search scrapy python on Google
cheetahogreen
Regular Member
- Apr 27, 2024
- 258
- 51
You can use libraries like BeautifulSoup with requests or Scrapy for scraping WordPress sites, and remove ad blocks by targeting specific HTML elements or classes associated with ads. Combining html.parser with custom filtering logic should help you keep the original HTML structure while removing ad blocks.
- Apr 23, 2024
- 2,108
- 4,227
Will try this! Seems to be that the scraping is a good business this days