SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
I only scrape data (numbers, sizes, model...). And with that data I create an article. Check previous messages
[CASE-STUDY] Blasting scraped-AI articles on an old domain
- Thread starter SEOCodeLab
- Start date
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
So just a quick update, the webpage has not recovered its peak performance and it is still very low visitors. I have been accepted on adsense though 
.
I have used this time to make my python scripts more efficient and faster. I think I have a list of mistakes I have done here that should not happen again:
- Use better hosting
- Use different way of posting articles (maybe not blast all the articles, but keep posting every 1 min or so). 80k articles at 1 min per articles is around: 1000h to post all articles. So I think, best way will be to split these articles into different webpages. 80k articles on 10 websites = 8k per website, which could be reasonable.
- I am able to have the articles more unique, so should also do that.
- Check if I can do the same on other niches (most probably yes)
- Try to automate as much as possible with python and wordpress plugins.
- Create internal links between articles. What I was thinking on doing is in every article link back to the previous article, and so on and so forth... Can this be enough?
- Check if wordpress is actually the best way forward and not pure PHP/HTML or some other CMS?
- Use english webpage! I think english can have more potential visitors.
Then, this project becomes a data-management type and not a blogging per se.
I will keep monitoring this website and get more pointers from this.
Regards!
.I have used this time to make my python scripts more efficient and faster. I think I have a list of mistakes I have done here that should not happen again:
- Use better hosting
- Use different way of posting articles (maybe not blast all the articles, but keep posting every 1 min or so). 80k articles at 1 min per articles is around: 1000h to post all articles. So I think, best way will be to split these articles into different webpages. 80k articles on 10 websites = 8k per website, which could be reasonable.
- I am able to have the articles more unique, so should also do that.
- Check if I can do the same on other niches (most probably yes)
- Try to automate as much as possible with python and wordpress plugins.
- Create internal links between articles. What I was thinking on doing is in every article link back to the previous article, and so on and so forth... Can this be enough?
- Check if wordpress is actually the best way forward and not pure PHP/HTML or some other CMS?
- Use english webpage! I think english can have more potential visitors.
Then, this project becomes a data-management type and not a blogging per se.
I will keep monitoring this website and get more pointers from this.
Regards!
Peter Davies
Junior Member
- Jul 23, 2019
- 151
- 77
If you don't mind, what hosting are you using for this site bro?
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
google cloud
Peter Davies
Junior Member
- Jul 23, 2019
- 151
- 77
Great! Thank you
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
So... I finished my first version of the scripts. Now everything is running on the server, nothing locally. Checking my points earlier:
- Use better hosting CHECK --> Using Google Cloud
- Use different way of posting articles (maybe not blast all the articles, but keep posting every 1 min or so). 80k articles at 1 min per articles is around: 1000h to post all articles. So I think, best way will be to split these articles into different webpages. 80k articles on 10 websites = 8k per website, which could be reasonable. Half check --> Setup scripts on server, still not started using full range of domains.
- I am able to have the articles more unique, so should also do that. CHECK --> Created better, longer and well formatted templates
- Check if I can do the same on other niches (most probably yes) CHECK --> I found out I can use this on 3 more niches that may be more profitable even
- Try to automate as much as possible with python and wordpress plugins. CHECK --> Everything is automated, only setting up the scraper is what is manual
- Create internal links between articles. What I was thinking on doing is in every article link back to the previous article, and so on and so forth... Can this be enough? CHECK --> Created a related posts and previous and next posts at the end of every article.
- Check if wordpress is actually the best way forward and not pure PHP/HTML or some other CMS? CHECK --> No other CMS can bring this flexibility, can go long with this.
- Use english webpage! I think english can have more potential visitors. CHECK --> Started using a new domain with an english niche
- Use better hosting CHECK --> Using Google Cloud
- Use different way of posting articles (maybe not blast all the articles, but keep posting every 1 min or so). 80k articles at 1 min per articles is around: 1000h to post all articles. So I think, best way will be to split these articles into different webpages. 80k articles on 10 websites = 8k per website, which could be reasonable. Half check --> Setup scripts on server, still not started using full range of domains.
- I am able to have the articles more unique, so should also do that. CHECK --> Created better, longer and well formatted templates
- Check if I can do the same on other niches (most probably yes) CHECK --> I found out I can use this on 3 more niches that may be more profitable even
- Try to automate as much as possible with python and wordpress plugins. CHECK --> Everything is automated, only setting up the scraper is what is manual
- Create internal links between articles. What I was thinking on doing is in every article link back to the previous article, and so on and so forth... Can this be enough? CHECK --> Created a related posts and previous and next posts at the end of every article.
- Check if wordpress is actually the best way forward and not pure PHP/HTML or some other CMS? CHECK --> No other CMS can bring this flexibility, can go long with this.
- Use english webpage! I think english can have more potential visitors. CHECK --> Started using a new domain with an english niche
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
If you can a more affordable hosting than google cloud, would be awesome. Need at least a VPS with min 4 cores, 6gb ram, 100gb ssd and (most important) unmetered bandwith.
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
Also, question:
What would be the best way to build links for these types of websites? Some crappy PBNs? Small high quality links? No links at all?
What would be the best way to build links for these types of websites? Some crappy PBNs? Small high quality links? No links at all?
bottingisfine
Power Member
- Apr 13, 2022
- 679
- 284
How do you avoid orphan pages?
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
Create internal links between articles. What I was thinking on doing is in every article link back to the previous article, and so on and so forth... Can this be enough? CHECK --> Created a related posts and previous and next posts at the end of every article.
Any Adsense earnings yet, outta interest?
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
Nothing much, around 1euro or so in a week
BtcTradeNG
Regular Member
- Feb 22, 2017
- 229
- 191
scraped 2.6 million AI posts and couldnt figure out a way to import the huge CSV into Wordpress... not even Linodes's high cost servers couldn't handle it so I gave up.
blogmen
Elite Member
- May 16, 2021
- 1,984
- 2,384
Break the CSV into smaller files and use WP all import plugin.
You can use cron jobs to automatically import but it will still take a lot of time for your huge data.
You will need good server and maybe the paid version of the plugin.
zcldeapple
Power Member
- Nov 14, 2016
- 501
- 46
How do you know the articles you crawl are created by AI
Shaybib
Senior Member
- Mar 23, 2018
- 831
- 262
Use wordpress rest api
SEOCodeLab
Power Member
- Jun 18, 2012
- 503
- 398
Either that or split them and create an SQL file to add directly to the database