srdjan87
Regular Member
- Jan 11, 2011
- 212
- 108
Hi guys,
I want to share a free Python script that scrapes posts from any WordPress website and publishes them to your WP website. Little contribution from me, nothing huge, but still not too bad
First script
collecting post IDs
Where example.com is, just add the WordPress domain that you want to scrape. Run the code via CMD, Windows Power Shell, or Terminal (whatever you use). Remember, you will need to install requests and json Python modules.
Once the scraping is done copy and paste all the IDs into a notepad file and name it whatever you like. After that, let's go to another step!
Second Script
adding IDs and publishing the scraped content from the first script
Install all these modules in order script to work.
html
re
slugify
BeautifulSoup
time
xmlrpc
Just do pip install for all of them individually if you are a beginner. Maybe I missed something but should be all good.
Please note that I set up this script to automatically run every 7 seconds(my favourite number) you can change that in the last line.
That's should be it.
Obviously, this is pure duplicated content, but if you have some nice authority-aged domain(news approved if possible) and 200k articles generated with this script, you would be surprised how works very well and makes money. Or you can use it for fun and do some testing. It's free anyway.
Cheers guys
I want to share a free Python script that scrapes posts from any WordPress website and publishes them to your WP website. Little contribution from me, nothing huge, but still not too bad
First script
collecting post IDs
Python:
import requests
import json
headers ={
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6",
"cache-control": "max-age=0",
"sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\"",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1"
}
for i in range(1,10000):
try:
g = requests.get("https://example.com/wp-json/wp/v2/posts/?page="+str(i) , headers=headers)
js = g.json()
for j in js:
print(""+str(j['id']))
except:
NoneWhere example.com is, just add the WordPress domain that you want to scrape. Run the code via CMD, Windows Power Shell, or Terminal (whatever you use). Remember, you will need to install requests and json Python modules.
Once the scraping is done copy and paste all the IDs into a notepad file and name it whatever you like. After that, let's go to another step!
Second Script
adding IDs and publishing the scraped content from the first script
Python:
import html
import json
import requests
import re
from slugify import slugify
from bs4 import BeautifulSoup
import time
def pop(file):
with open(file, 'r+') as f: # open file in read / write mode
firstLine = f.readline() # read the first line and throw it out
data = f.read() # read the rest
f.seek(0) # set the cursor to the top of the file
f.write(data) # write the data back
f.truncate() # set the file size to the current size
return firstLine
def loadJson(url,id):
g1_post = json.loads( requests.get("https://"+str(url).strip().rstrip().lstrip()+"/wp-json/wp/v2/posts/"+str(id).strip().rstrip().lstrip(),verify=False).text )
title = re.sub('&(.*?);','',str(g1_post['title']['rendered']))
try:
soup = BeautifulSoup(str(g1_post['content']['rendered']),"html.parser")
f = soup.find('div', attrs={"id":"toc_container"})
f.decompose()
content = str(soup)
except:
content = g1_post['content']['rendered']
return {"title":html.unescape(title).replace('[','').replace(']','').replace('/','').replace('SOLVED','').replace('Solved','').replace('”','').replace('’','').replace('-','').replace(':','').replace('“','').replace('(','').replace(')','').replace('-',''),"slug":slugify(html.unescape(title).replace('[','').replace(']','').replace('/','').replace('SOLVED','').replace('Solved','').replace('”','').replace('’','').replace('-','').replace(':','').replace('“','').replace('(','').replace(')','').replace('-','')),"content":content + '<br><br>Source: ' + url}
from wordpress_xmlrpc import WordPressPost
from wordpress_xmlrpc import Client, WordPressPost
from wordpress_xmlrpc.methods import posts
from wordpress_xmlrpc.methods.posts import GetPosts, NewPost
from wordpress_xmlrpc.methods.users import GetUserInfo
from wordpress_xmlrpc import Client, WordPressPost
from wordpress_xmlrpc.compat import xmlrpc_client
from wordpress_xmlrpc.methods import media, posts
while True:
# run your script here
first = pop('path/of_your_notepad_with_IDs_in_it/example.txt')
data = loadJson("here_add_url_of_wp_website_you_scraping_the_articles",first)
client = Client('https://your_url/xmlrpc.php', 'your_wp_username', 'your_wp_password')
post = WordPressPost()
post.title = data['title']
post.content = data['content']
post.terms_names = {
}
post.id = client.call(posts.NewPost(post))
post.post_status = 'publish'
client.call(posts.EditPost(post.id, post))
time.sleep(7)Install all these modules in order script to work.
html
re
slugify
BeautifulSoup
time
xmlrpc
Just do pip install for all of them individually if you are a beginner. Maybe I missed something but should be all good.
Please note that I set up this script to automatically run every 7 seconds(my favourite number) you can change that in the last line.
That's should be it.
Obviously, this is pure duplicated content, but if you have some nice authority-aged domain(news approved if possible) and 200k articles generated with this script, you would be surprised how works very well and makes money. Or you can use it for fun and do some testing. It's free anyway.
Cheers guys
Last edited:
