100% Automated Review Website From Scratch. My Journey Learning Python.

[beDreamful]

Learning some programming language was one of my choices in late 2018. I literally lost interest in coding in 2019. Good luck with achieving those goals, OP.

You should pick it up again! You can do so much with it. Thanks!

 

[nikhil94]

i have something similiar in mind doing. just the content creation will be a bit different. did you figure out how to create posts in wordpress just using the database?
every post in wordpress is basically just one database entry, but also connected to some other databases. but creating that post should just be an simple sql create in the database. but still not 100% sure how to do it, iam also super new to python. so nice journey wish you good luck and work hard

Use wp-json API. There is a wrapper available for it in Python.

 

[googleagent59]

Hey you better screen record of what you doing... It will be very helpful

 

[Shaybib]

You can also try to scrape realated content from other websites to make it more unique and add value

 

[beDreamful]

Hey you better screen record of what you doing... It will be very helpful

I don't need to record it for myself. I don't plan on editing or releasing videos either. I don't mind answering any questions tho.

You can also try to scrape realated content from other websites to make it more unique and add value

I thought about this. It might be a very good idea.

 

[M4DM4X]

Nice project. I was doing the same thing a few years ago but for youtube. My bot generated title and description from the Amazon product page. It was cool, but now this kind of content don't rank anymore. Or maybe with a better spinning than mine good luck mate, following your quest.

 

[beDreamful]

Nice project. I was doing the same thing a few years ago but for youtube. My bot generated title and description from the Amazon product page. It was cool, but now this kind of content don't rank anymore. Or maybe with a better spinning than mine good luck mate, following your quest.

Thanks! I hope it ranks but it's not the main objective. I'll give it 100% tho. I'll make sure to put some extra effort on the spinner. I try to pass Copyscape + have a minimum of 600 words.

 

[beDreamful]

So far so good. This is more complicated than I first expected to make a really good quality output. There's also a few challenge to NOT get banned by Amazon Affiliate. I'm trying to map out what I need since the code keeps growing. I'll get a full-on update and MIA output very soon.

 

[DanDD]

why dont u make ur own cms? learning wordpress codec takes time
also theres an exact cms for sale at marketplace, you can edit that one
anyway good luck, following

 

[MrOusho]

Following. Good luck.

What Linux distro do you use?

 

[TheNights]

Best wishes for the journey mate. I'm really interested in the content generation part, to see where this goes. I recently applied for access to GPT-3 API, plan to use it for text generation.

Good decision to stick to Wordpress for now, you can use the API to manage content on the blog and later on build your own CMS with django when you have a better understanding of the custom stuff you need.

Feel free to DM me if you have any questions related to Python.

 

[nikhil94]

I recently applied for access to GPT-3 API

Please update us if you get access.

 

[200]

Cute idea you have here. Good luck in your automation.

 

[mandroid999]

Looks interesting.

 

[beDreamful]

why dont u make ur own cms? learning wordpress codec takes time
also theres an exact cms for sale at marketplace, you can edit that one
anyway good luck, following

I want to get my hands dirty. I want to build it from scratch so I learn how to do everything.

Following. Good luck.

What Linux distro do you use?

Ubuntu 18

Best wishes for the journey mate. I'm really interested in the content generation part, to see where this goes. I recently applied for access to GPT-3 API, plan to use it for text generation.

Good decision to stick to Wordpress for now, you can use the API to manage content on the blog and later on build your own CMS with django when you have a better understanding of the custom stuff you need.

Feel free to DM me if you have any questions related to Python.

I'll be using the API to generate the pages. Anything style-wise will be edited via Wordpress theme.

What does the GPT-3 have that GPT-2 does not have? I have the 355 model right now trying to figure out if I can use it. Training is crazy. Might not be worth it for me.

I'll posting a good update soon.

 

[Jammy0308]

I love these sorts of threads, always wanted to do something like this with python. Good luck!

 

[beDreamful]

What up everyone. My computer crashed so I decided it was time for an update.

I learned a lot so far. Let's break it down according to my goals:

Every interaction with a website needs to look human.
I'm currently using selenium & scrapy. I'm pretty sure I can make both of them almost invisible to amazon with proper user-agent & randomized waiting time. I'll keep learning about those but it's not urgent for the moment.

Articles need to have an infographic kinda vibe. The rest will be generated or spun somehow.
I decided to use multiple short texts since it's easier to generate. I'll be using something 6 "blocks" like this:

Effectiveness
7/10
This thing does what it was meant to do. (..)

Value for Price
8/10
This thing does what it was meant to do. (..)


This will be followed by a 300-400 word review. I want to add more stuff but I will focus on this for the moment

To generate the "blocks", I scrape all 5 and 4 stars review from the product. I clean-up the data until only the meaningful words are left. MIA will then use NGRAM (a type of language analysis) to attribute each "block" a score. Each score from 6-10 represents a database for the text generator. The text generated will always represent the score.

I'm now in the clean-up part. Here's a sneak peek using all reviews for a MacBook (I only printed the first 500 words):

['Okay', 'see', 'people', 'display', 'Macbooks', 'always', 'fancy', 'own', 'one', 'premium', 'device', 'Now', 'Amazon', 'offer', 'Macbook', 'Air', 'reduce', 'price', 'get', 'tempt', 'buy', 'STOP', 'consider', 'get', 'money', 'spend', 'buy', 'think', 'add', 'premiumness', 'profile', 'device', 'next', 'line', 'support', 'withdrawn', 'Apple', 'Resist', 'temptation', 'THAT', 'Apple', 'sell', 'The', 'last', 'reason', 'buy', 'feel', 'improves', 'profile', 'Apple', 'user', 'smirk', 'see', 'obsolete', "'dinosaur", 'device', 'display', 'late', 'toys.If', 'window', 'user', 'well', 'friend', 'Macbook', 'try', 'activity', 'MAC', 'great', 'till', 'try', 'change', 'something', 'puny', 'mouse', 'cursor', 'You', 'realise', 'stuck', 'little', 'cozy', 'jail', 'Apple', 'try', 'anything', 'non-standard', 'Try', 'transfer', 'data', 'USB', 'stick', 'laptop', 'one', '....', 'need', 'training', 'USB', 'partition', 'ExFAT', 'evn', 'partition', 'MAC', 'chance', 'Windows', 'recognise', 'USB', 'stick', 'power', 'user', 'real', 'pain', 'course', 'pretend', 'like', 'many', 'thousand', 'Apple', 'user', 'matter', 'But', 'hey', 'premium', 'DELL', 'may', 'best', 'option', 'Yes', 'first', 'discourage', 'buying', 'explain', 'purchase', 'Know', 'give', 'option', 'buying', 'RAM', 'SSD', 'neighbour', 'shop', 'extend', 'device', 'life', 'You', 'wil', 'depend', 'Apple', 'Apple', 'alone', 'course', 'generic', 'spare', 'expensive', 'DELL', 'spare', 'Once', 'Apple', 'pull', 'plug', 'support', 'spare', 'may', 'get', 'even', 'expensive', 'fact', 'hardware', 'upgrade', 'beyond', 'increase', 'SSD', 'much', 'rare', 'luxury', 'Thankfully', 'kept', 'snob-race', 'Apple', 'peddles.If', 'still', 'reading', 'good', 'side', 'purchase', 'spite', 'write', 'Mac', 'Mac', 'Mac', 'one', 'The', 'hardware', 'perfectly', 'match', 'performance', 'always', 'optimal', 'This', 'laptop', 'carry', 'along', 'second', 'laptop', 'alongside', 'official', 'one', 'DELL', 'workhorse', 'MAcbook', 'Air', 'lightweight', 'run', 'operating', 'system', 'use', 'run', 'Windows', 'Linux.The', 'MAC', 'stable', 'Operating', 'system', 'Chances', 'never', 'face', 'single', 'crash', 'lifetime', 'use', 'Macbook', 'Air', 'Windows', 'fragile', 'many', 'bug', 'update', 'Linux', 'difficult', 'need', 'little', 'training', 'That', 'Macbook', 'Air', 'come', 'Handy', 'With', 'Parallels', 'desktop', 'load', 'Linux', 'Windows', 'Virtual', 'Machines', 'Mac', 'You', 'seamlessly', 'one', 'another', 'without', 'reboot', 'However', 'note', "'parallels", 'desktop', 'operating', 'Windows', 'Linux', 'little', 'cell', 'Apple', 'jail', 'And', 'prepared', 'shell', 'good', 'amount', 'buy', 'software', 'possible', 'load', 'linux', 'differently', 'work', 'outside', 'Apple', 'shell', 'Quite', 'save', 'grace', 'Using', 'BootCamp', 'dual', 'bootng', 'feature', 'take', 'advantage', 'multi-tasking', 'power', 'Windows', 'without', 'risk', 'get', 'blue', 'screen', 'hang', 'function', 'overall', 'look', 'great.Now', 'come', 'back', 'rating', '...', 'run', 'mostly', 'Windows', 'Linux', 'MAC', 'Happy', 'hardware', 'great', 'consider', 'power', 'user', 'know', 'find', 'alternative', 'Apple', 'pull', 'plug', 'support', 'available', 'least', 'another', 'year', 'certain', 'period', 'Apple', 'support', 'obsolete', 'device', 'Hence', 'durability', 'rating', 'star', 'That', 'truth', 'laid', 'bare', 'Please', 'decide', 'buy', 'risk.~~', 'Buy', 'like', 'hardware', 'know', 'DELL', 'would', 'give', 'much', 'well', 'hardware', 'price', 'upgradable', 'last', 'much', 'longer', 'Buy', 'understand', 'hardware', 'stable', 'Mac', 'protect', 'window', 'work', 'MAC', 'Shell', 'Buy', 'run', 'Linux', 'Mac', 'make', 'obsolete.~~', 'buy', 'feel', 'add', 'premium', 'value', 'dinosaur', 'soon', 'left', 'strand', 'Think', 'plight', 'iPhone', 'premium', 'month', 'ago', 'People', 'rush', 'purchase', 'cheap', 'iPhone', 'around', 'year', 'back', 'Today', 'pride', 'show', 'device', 'obsolete', 'Tomorrow', 'Apple', 'make', 'sluggish', 'force', 'upgrade.EDIT/ADD', '02/12/2019', 'plan', 'upgrade', 'SSD', '500GB', 'please', 'look', 'SSD', 'ASIN', 'B07J2WBKXF', 'adapter', 'ASIN', 'B07D7VM8GM', 'able', 'successfully', 'upgrade', 'Review', 'SSD', 'page', 'amaze', '.....', 'battery', 'last', 'around', '-12', '.....', 'best', 'laptop', 'student', 'coder', 'Finally', 'MacBook', 'possession', 'Got', '50k', 'Diwali', 'sale', 'Technically', 'good', 'laptop.Good', 'News.luxury', 'product', 'niche', 'technology', 'good', 'battery', 'life', 'reliable', 'machine', 'laptop', 'crash', 'become', 'defunct', '6-10', 'year', 'anti-virus', 'require', 'security', 'data', 'transaction', 'much', 'well', 'Terrific', 'sense', 'possession', 'pride.Bad', 'News', 'Problems', 'switch', 'window', 'base', 'system', 'less', 'space', 'hard', 'disk', 'drive', 'inability', 'transfer', 'data', 'Mac']

That's it for now. Time to jump back in.

 

[beDreamful]

Here's an update.

Mia can now analyze all the reviews and give her rating to the product.
Let's say I take this Apple MacBook. It has a 4.7 stars rating on Amazon. This is the bot output based on the reviews:

quality 8.9/10
style 8.4/10
value for price 9.6/10
price 8/10
packaging 7.4/10
ease of use 7.9/10

Now I need to generate text for all of those blocks. That's another story.

To be able to spin my main text, I automated deepl.com to translate the review in Chinese, then back in English. This sorta changes the structure of the sentences while being readable.

I'm now working on the spinning part. Changing words here and there after the translation. After that, I need to find a way to automate Grammarly to correct this mess.

I'm going back in!

 

[B.B.K.K.B.K.K.]

I'm going to take a place

Do you know if it is currently expected that amazon blocks you for detecting that you are browsing products in an automated way? I mean, did you test it?
I´m actually at a similar project but I dont pretend to scrap descriptions or comments, just title, price and product image.