100% Automated Review Website From Scratch. My Journey Learning Python.

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
Learning some programming language was one of my choices in late 2018. I literally lost interest in coding in 2019. Good luck with achieving those goals, OP.
You should pick it up again! You can do so much with it. Thanks!
 

nikhil94

Regular Member
Joined
Apr 17, 2012
Messages
338
Reaction score
92
i have something similiar in mind doing. just the content creation will be a bit different. did you figure out how to create posts in wordpress just using the database?
every post in wordpress is basically just one database entry, but also connected to some other databases. but creating that post should just be an simple sql create in the database. but still not 100% sure how to do it, iam also super new to python. so nice journey wish you good luck and work hard
Use wp-json API. There is a wrapper available for it in Python.
 

Shaybib

Power Member
Joined
Mar 23, 2018
Messages
712
Reaction score
193
You can also try to scrape realated content from other websites to make it more unique and add value
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
Hey you better screen record of what you doing... It will be very helpful
I don't need to record it for myself. I don't plan on editing or releasing videos either. I don't mind answering any questions tho.
You can also try to scrape realated content from other websites to make it more unique and add value
I thought about this. It might be a very good idea.
 

M4DM4X

Elite Member
Joined
Jan 21, 2015
Messages
1,765
Reaction score
903
Nice project. I was doing the same thing a few years ago but for youtube. My bot generated title and description from the Amazon product page. It was cool, but now this kind of content don't rank anymore. Or maybe with a better spinning than mine :) good luck mate, following your quest.
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
Nice project. I was doing the same thing a few years ago but for youtube. My bot generated title and description from the Amazon product page. It was cool, but now this kind of content don't rank anymore. Or maybe with a better spinning than mine :) good luck mate, following your quest.
Thanks! I hope it ranks but it's not the main objective. I'll give it 100% tho. I'll make sure to put some extra effort on the spinner. I try to pass Copyscape + have a minimum of 600 words.
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
So far so good. This is more complicated than I first expected to make a really good quality output. There's also a few challenge to NOT get banned by Amazon Affiliate. I'm trying to map out what I need since the code keeps growing. I'll get a full-on update and MIA output very soon.
 

DanDD

Senior Member
Joined
Mar 10, 2014
Messages
911
Reaction score
302
why dont u make ur own cms? learning wordpress codec takes time
also theres an exact cms for sale at marketplace, you can edit that one
anyway good luck, following
 

TheNights

Junior Member
Joined
Nov 14, 2017
Messages
106
Reaction score
28
Best wishes for the journey mate. I'm really interested in the content generation part, to see where this goes. I recently applied for access to GPT-3 API, plan to use it for text generation.

Good decision to stick to Wordpress for now, you can use the API to manage content on the blog and later on build your own CMS with django when you have a better understanding of the custom stuff you need.

Feel free to DM me if you have any questions related to Python.
 

200

Senior Member
Joined
Jul 15, 2020
Messages
1,139
Reaction score
910
Cute idea you have here. Good luck in your automation.
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
why dont u make ur own cms? learning wordpress codec takes time
also theres an exact cms for sale at marketplace, you can edit that one
anyway good luck, following
I want to get my hands dirty. I want to build it from scratch so I learn how to do everything.
Following. Good luck.

What Linux distro do you use?
Ubuntu 18
Best wishes for the journey mate. I'm really interested in the content generation part, to see where this goes. I recently applied for access to GPT-3 API, plan to use it for text generation.

Good decision to stick to Wordpress for now, you can use the API to manage content on the blog and later on build your own CMS with django when you have a better understanding of the custom stuff you need.

Feel free to DM me if you have any questions related to Python.
I'll be using the API to generate the pages. Anything style-wise will be edited via Wordpress theme.

What does the GPT-3 have that GPT-2 does not have? I have the 355 model right now trying to figure out if I can use it. Training is crazy. Might not be worth it for me.

I'll posting a good update soon.
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
What up everyone. My computer crashed so I decided it was time for an update.

I learned a lot so far. Let's break it down according to my goals:

Every interaction with a website needs to look human.
I'm currently using selenium & scrapy. I'm pretty sure I can make both of them almost invisible to amazon with proper user-agent & randomized waiting time. I'll keep learning about those but it's not urgent for the moment.

Articles need to have an infographic kinda vibe. The rest will be generated or spun somehow.
I decided to use multiple short texts since it's easier to generate. I'll be using something 6 "blocks" like this:

Effectiveness
7/10
This thing does what it was meant to do. (..)

Value for Price
8/10
This thing does what it was meant to do. (..)


This will be followed by a 300-400 word review. I want to add more stuff but I will focus on this for the moment

To generate the "blocks", I scrape all 5 and 4 stars review from the product. I clean-up the data until only the meaningful words are left. MIA will then use NGRAM (a type of language analysis) to attribute each "block" a score. Each score from 6-10 represents a database for the text generator. The text generated will always represent the score.

I'm now in the clean-up part. Here's a sneak peek using all reviews for a MacBook (I only printed the first 500 words):
['Okay', 'see', 'people', 'display', 'Macbooks', 'always', 'fancy', 'own', 'one', 'premium', 'device', 'Now', 'Amazon', 'offer', 'Macbook', 'Air', 'reduce', 'price', 'get', 'tempt', 'buy', 'STOP', 'consider', 'get', 'money', 'spend', 'buy', 'think', 'add', 'premiumness', 'profile', 'device', 'next', 'line', 'support', 'withdrawn', 'Apple', 'Resist', 'temptation', 'THAT', 'Apple', 'sell', 'The', 'last', 'reason', 'buy', 'feel', 'improves', 'profile', 'Apple', 'user', 'smirk', 'see', 'obsolete', "'dinosaur", 'device', 'display', 'late', 'toys.If', 'window', 'user', 'well', 'friend', 'Macbook', 'try', 'activity', 'MAC', 'great', 'till', 'try', 'change', 'something', 'puny', 'mouse', 'cursor', 'You', 'realise', 'stuck', 'little', 'cozy', 'jail', 'Apple', 'try', 'anything', 'non-standard', 'Try', 'transfer', 'data', 'USB', 'stick', 'laptop', 'one', '....', 'need', 'training', 'USB', 'partition', 'ExFAT', 'evn', 'partition', 'MAC', 'chance', 'Windows', 'recognise', 'USB', 'stick', 'power', 'user', 'real', 'pain', 'course', 'pretend', 'like', 'many', 'thousand', 'Apple', 'user', 'matter', 'But', 'hey', 'premium', 'DELL', 'may', 'best', 'option', 'Yes', 'first', 'discourage', 'buying', 'explain', 'purchase', 'Know', 'give', 'option', 'buying', 'RAM', 'SSD', 'neighbour', 'shop', 'extend', 'device', 'life', 'You', 'wil', 'depend', 'Apple', 'Apple', 'alone', 'course', 'generic', 'spare', 'expensive', 'DELL', 'spare', 'Once', 'Apple', 'pull', 'plug', 'support', 'spare', 'may', 'get', 'even', 'expensive', 'fact', 'hardware', 'upgrade', 'beyond', 'increase', 'SSD', 'much', 'rare', 'luxury', 'Thankfully', 'kept', 'snob-race', 'Apple', 'peddles.If', 'still', 'reading', 'good', 'side', 'purchase', 'spite', 'write', 'Mac', 'Mac', 'Mac', 'one', 'The', 'hardware', 'perfectly', 'match', 'performance', 'always', 'optimal', 'This', 'laptop', 'carry', 'along', 'second', 'laptop', 'alongside', 'official', 'one', 'DELL', 'workhorse', 'MAcbook', 'Air', 'lightweight', 'run', 'operating', 'system', 'use', 'run', 'Windows', 'Linux.The', 'MAC', 'stable', 'Operating', 'system', 'Chances', 'never', 'face', 'single', 'crash', 'lifetime', 'use', 'Macbook', 'Air', 'Windows', 'fragile', 'many', 'bug', 'update', 'Linux', 'difficult', 'need', 'little', 'training', 'That', 'Macbook', 'Air', 'come', 'Handy', 'With', 'Parallels', 'desktop', 'load', 'Linux', 'Windows', 'Virtual', 'Machines', 'Mac', 'You', 'seamlessly', 'one', 'another', 'without', 'reboot', 'However', 'note', "'parallels", 'desktop', 'operating', 'Windows', 'Linux', 'little', 'cell', 'Apple', 'jail', 'And', 'prepared', 'shell', 'good', 'amount', 'buy', 'software', 'possible', 'load', 'linux', 'differently', 'work', 'outside', 'Apple', 'shell', 'Quite', 'save', 'grace', 'Using', 'BootCamp', 'dual', 'bootng', 'feature', 'take', 'advantage', 'multi-tasking', 'power', 'Windows', 'without', 'risk', 'get', 'blue', 'screen', 'hang', 'function', 'overall', 'look', 'great.Now', 'come', 'back', 'rating', '...', 'run', 'mostly', 'Windows', 'Linux', 'MAC', 'Happy', 'hardware', 'great', 'consider', 'power', 'user', 'know', 'find', 'alternative', 'Apple', 'pull', 'plug', 'support', 'available', 'least', 'another', 'year', 'certain', 'period', 'Apple', 'support', 'obsolete', 'device', 'Hence', 'durability', 'rating', 'star', 'That', 'truth', 'laid', 'bare', 'Please', 'decide', 'buy', 'risk.~~', 'Buy', 'like', 'hardware', 'know', 'DELL', 'would', 'give', 'much', 'well', 'hardware', 'price', 'upgradable', 'last', 'much', 'longer', 'Buy', 'understand', 'hardware', 'stable', 'Mac', 'protect', 'window', 'work', 'MAC', 'Shell', 'Buy', 'run', 'Linux', 'Mac', 'make', 'obsolete.~~', 'buy', 'feel', 'add', 'premium', 'value', 'dinosaur', 'soon', 'left', 'strand', 'Think', 'plight', 'iPhone', 'premium', 'month', 'ago', 'People', 'rush', 'purchase', 'cheap', 'iPhone', 'around', 'year', 'back', 'Today', 'pride', 'show', 'device', 'obsolete', 'Tomorrow', 'Apple', 'make', 'sluggish', 'force', 'upgrade.EDIT/ADD', '02/12/2019', 'plan', 'upgrade', 'SSD', '500GB', 'please', 'look', 'SSD', 'ASIN', 'B07J2WBKXF', 'adapter', 'ASIN', 'B07D7VM8GM', 'able', 'successfully', 'upgrade', 'Review', 'SSD', 'page', 'amaze', '.....', 'battery', 'last', 'around', '-12', '.....', 'best', 'laptop', 'student', 'coder', 'Finally', 'MacBook', 'possession', 'Got', '50k', 'Diwali', 'sale', 'Technically', 'good', 'laptop.Good', 'News.luxury', 'product', 'niche', 'technology', 'good', 'battery', 'life', 'reliable', 'machine', 'laptop', 'crash', 'become', 'defunct', '6-10', 'year', 'anti-virus', 'require', 'security', 'data', 'transaction', 'much', 'well', 'Terrific', 'sense', 'possession', 'pride.Bad', 'News', 'Problems', 'switch', 'window', 'base', 'system', 'less', 'space', 'hard', 'disk', 'drive', 'inability', 'transfer', 'data', 'Mac']
That's it for now. Time to jump back in.
 

beDreamful

Registered Member
Joined
Feb 15, 2018
Messages
88
Reaction score
57
Here's an update.

Mia can now analyze all the reviews and give her rating to the product.
Let's say I take this Apple MacBook. It has a 4.7 stars rating on Amazon. This is the bot output based on the reviews:

quality 8.9/10
style 8.4/10
value for price 9.6/10
price 8/10
packaging 7.4/10
ease of use 7.9/10

Now I need to generate text for all of those blocks. That's another story.

To be able to spin my main text, I automated deepl.com to translate the review in Chinese, then back in English. This sorta changes the structure of the sentences while being readable.

I'm now working on the spinning part. Changing words here and there after the translation. After that, I need to find a way to automate Grammarly to correct this mess.

I'm going back in!
 
Joined
Sep 12, 2020
Messages
14
Reaction score
4
I'm going to take a place

Do you know if it is currently expected that amazon blocks you for detecting that you are browsing products in an automated way? I mean, did you test it?
I´m actually at a similar project but I dont pretend to scrap descriptions or comments, just title, price and product image.
 
Top