wordpress plugin as standalone

shaymiller

Regular Member
Joined
Dec 3, 2016
Messages
456
Reaction score
45
I am looking for a script which will allow me to click & scrape content then save it to mysql. I only found wordpress plugins do that. they work perfectly fine. My issues is that I don't want to use wordpress. mainly because the wordpress database becomes larger, if I am scraping large amount od data. wordpress has security issues too as its open source
 
Here you go. A simple Python code, that allows you to scrape the content and save it o MySQL:

import json, requests
import pandas as pd
import MySQLdb
from pandas.io.json import json_normalize

# Set URL
url = 'https://google.com'

# For loop to
for i in range(100):
data = json.loads(requests.get(
url=url,
params={'page': i}
).text)['results']

data_norm = pd.read_json(json.dumps(data))

def dbconnect():
try:
db = MySQLdb.connect(
host='localhost',
user='root',
passwd='password',
db='nameofdb'
)
except Exception as e:
sys.exit("Can't connect to database")
return db

def insertDb():
try:
db = dbconnect()
cursor = db.cursor()
cursor.execute("""
INSERT INTO nameoftable(nameofcolumn) \
VALUES (%s) """, (data))
cursor.close()
except Exception as e:
print e
 
Back
Top