- Apr 23, 2012
- 2,850
- 5,865
Hello lovelies,
Dropping another script that can help with your workflow
You've shown a lot of love to my Keyword Clustering Script, Keyword Similarity Check, YouTube Scraper and Google SERP Scraper & Summarizer that I shared on the forum. Here's one more.
Built a barebones version of this for a member of the forum earlier today, and thought I'd modify it slightly, make it more userfriendly and share it.
The Problem
A lot of you want to scrape off WordPress sites for migration, archiving, or just as fodder for your AI to create content. Doing it by hand is slow. Paid scrapers and APIs either cost money per request or give you limited request. And a lot of sites don’t give you a clean export anyway.
The Magic of the script
This script will
1. Scrape a WordPress site for posts (entire site, or just the categories you pick)
2. Save them as HTML, Markdown, and/or JSON
3. Open a simple web UI in your browser so you can configure everything without touching code
4. Resume if you stop halfway (it remembers what’s already scraped)
5. Let you paste proxies in the UI and cycles through them ramdomly.
How to install - Simple 2 Steps
Link to Github Repository - https://github.com/iamBlogPro/WP-Scraper
Before anything else, Please make sure you have downloaded and installed Python ( https://www.python.org/downloads/ ) and are using an IDE like VSCode ( https://code.visualstudio.com/download )
Step 1 - Clone the Repository or Download the Zip
Or download the zip - https://github.com/iamBlogPro/WP-Scraper/archive/refs/heads/main.zip
Extract and Open it in VS Code or an IDE of your choice.
Step 2 - Create a virtual environment and install dependencies
Running Everything - The Fun Part
You should get a response like this
That's it - your server is up and running!
Now let's visit
You should see this page
Now here's a few things
1. Proxy - you can add a proxy and add several. It supports userass and normal IP whitelisted.
2. If your target site has custom DIV classes, you can add those too.
3. The scraper prefers the REST API (on by default on most sites since WordPress 4.7 - if that fails, it false back to HTML
4. If you click "Load Categories" - the scraper will try to fetch categories from the website. You can check one or more and only those will be downloaded.
5. You can choose to save the scraped data in Markdown, HTML or JSON (or all three)
6. It should work just fine in Blogger blogs as well, but thats not heavily tested yet.
//
Hope some of you find this useful. As always, if you get stuck anywhere do let me know. Play around with it, if there are more features you need or if there's a site that's causing issues, let me know.
The code is open source, you're free to fork it and modify and enhance it.
Have a good one guys.
B.
Dropping another script that can help with your workflow
You've shown a lot of love to my Keyword Clustering Script, Keyword Similarity Check, YouTube Scraper and Google SERP Scraper & Summarizer that I shared on the forum. Here's one more.
Built a barebones version of this for a member of the forum earlier today, and thought I'd modify it slightly, make it more userfriendly and share it.
The Problem
A lot of you want to scrape off WordPress sites for migration, archiving, or just as fodder for your AI to create content. Doing it by hand is slow. Paid scrapers and APIs either cost money per request or give you limited request. And a lot of sites don’t give you a clean export anyway.
The Magic of the script
This script will
1. Scrape a WordPress site for posts (entire site, or just the categories you pick)
2. Save them as HTML, Markdown, and/or JSON
3. Open a simple web UI in your browser so you can configure everything without touching code
4. Resume if you stop halfway (it remembers what’s already scraped)
5. Let you paste proxies in the UI and cycles through them ramdomly.
How to install - Simple 2 Steps
Link to Github Repository - https://github.com/iamBlogPro/WP-Scraper
Before anything else, Please make sure you have downloaded and installed Python ( https://www.python.org/downloads/ ) and are using an IDE like VSCode ( https://code.visualstudio.com/download )
Step 1 - Clone the Repository or Download the Zip
Bash:
git clone https://github.com/iamBlogPro/WP-Scraper.git
cd WP-ScraperOr download the zip - https://github.com/iamBlogPro/WP-Scraper/archive/refs/heads/main.zip
Extract and Open it in VS Code or an IDE of your choice.
Step 2 - Create a virtual environment and install dependencies
Bash:
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txtRunning Everything - The Fun Part
Bash:
python main.py
#Or python3 main.py depending on your python installYou should get a response like this
That's it - your server is up and running!
Now let's visit
http://127.0.0.1:8765/You should see this page
Now here's a few things
1. Proxy - you can add a proxy and add several. It supports user
ass and normal IP whitelisted. 2. If your target site has custom DIV classes, you can add those too.
3. The scraper prefers the REST API (on by default on most sites since WordPress 4.7 - if that fails, it false back to HTML
4. If you click "Load Categories" - the scraper will try to fetch categories from the website. You can check one or more and only those will be downloaded.
5. You can choose to save the scraped data in Markdown, HTML or JSON (or all three)
6. It should work just fine in Blogger blogs as well, but thats not heavily tested yet.
//
Hope some of you find this useful. As always, if you get stuck anywhere do let me know. Play around with it, if there are more features you need or if there's a site that's causing issues, let me know.
The code is open source, you're free to fork it and modify and enhance it.
Have a good one guys.
B.
