[Journey] 1 million UVs/month in 12 months using AI generated content. Let's do it!

[Archemike]

would you still tune this with hugging, or just directly use this with a paraphraser like pegasus or parrot?

Sartre said:


thank y'all.

hope everyone seen this: https://github.com/yandex/YaLM-100B


Damn, so someone said this a promotional thread and this got off topic.

Can someone in basic terms, explain how you would take this lib and use it? git clone just the english somehow to local, and then its already trained, r use hugging face somehow? And then does it need an adapter like pegasus to use it, or it can be run as is with an API that you input into it and it outputs the paraphrase? Any insight would be helpful, thanks guys. let me know how i can help in return.

 

[Archemike]

This piece right here is the gold nugget and there are so many ways to extract and twist the content.

For readers with Python experience, you can download images from stock photo sites like unsplash, pixabay, pexels, and use the Pillow library to add your own text on top of the image. This is also great to use for people who want to create their own memes.

exactly, this thread gave a ton of ideas. Some threads beg for some negative responses, but I'm at a loss how this thread can get negative attention.

This thread lead me down a rabbit hole on automation, for example this library and discord group: https://github.com/elebumm/RedditVideoMakerBot. Using python to generate TTS reddit vids, also very useful for inserting into blogs! I'm using a modified version for tiktok

 

[Smurfyy]

Hello @Sartre, I'm really fascinated by your threads & postings, I only have one question if you'd like to answer and provide me with some guidance. How much time did it take you like from point 0 to the point you're able to do these amazing things with Python and to make your own automation tools? I don't have any knowledge in python, but I know some Java & C#. how much time will it take me to build my own tools "not as good as yours" but enough to get the job done? Thank you very much

 

[Walterwhit]

Haha! I believe I know you from somewhere. Something you mentioned in the above post just clicked.

Been building PAA sites for a year now. My largest one is 182k pages in an extremely competitive niche. With around 120k indexed.

Been using GPT3 + a custom fine tuned GPTNeoX for content generation. Pegasus for rephrasing.

QA Format is very similar to yours as well.

I also harness the qapage schema in some instances.

Setup is very similar to yours. My tweaking and optimization is towards a different direction, though.

I use a different image generation library that gives me better shadows, more sigma control and crisper fonts.

I don't use the Indexing API (at least haven't yet, but may just use it as a test today on a brand new site).

All, but 5 of my sites, are on brand new Domains with zero back link profiles.

Here's a site with its first post 4 days ago (domain regged same day).

No backlinks, brand new domain, some social signals and web 2.0, again very competitive niche.

Happy to connect and brainstorm if you're interested.

i m thinking of using this for image generation ->

Yake -> bing image scraper -> PIL -> G An.

What is your approach for Image generation?

Wouldn't be great if all of us can connect via some form of slack group?

 

[BlogPro]

i m thinking of using this for image generation ->

Yake -> bing image scraper -> PIL -> G An.

What is your approach for Image generation?

Wouldn't be great if all of us can connect via some form of slack group?

Scraping images from Google/Bing could result in you ending up with a bunch of copyrighted images. Rather use Unsplash, Pexels, Pixabay etc. - they have an API as well. Your options would get extremely limited, but you'll get public domain images.

 

[prvncqr]

Where do you get images from? I tried Pixabay but it returned 0 results for most of the long tail queries. How do you get rid of such issues?

 

[Walterwhit]

Scraping images from Google/Bing could result in you ending up with a bunch of copyrighted images. Rather use Unsplash, Pexels, Pixabay etc. - they have an API as well. Your options would get extremely limited, but you'll get public domain images.

Yup, i use cc license while using them.

How was your success with qa schema? Did it make any difference?

 

[Walterwhit]

Anyone scraped enough websites using newspaper 3K to see and realize that sometimes even if you add some of your own cleanups to the HTML you may still end up with content coming from the MENU or FOOTER or Author BIO etc… because each person can write the code a little different on a site - you may always end up getting stuck with non-relevant and bad content in the output texts…. Anyone familiar with what I’m saying and managed to find a solution or just running manually reading outputs and making sure you didn’t got by accident all those unrelated texts?

If you scrape only h2 and the text after you are quite safe from menu and footer. Other than that you add a blacklist of words like "bio" and "covid" and dont add text containing them.
Also i dont know why people use newspaper instead of beautiful soup

Try this - https://github.com/adbar/trafilatura

BenchMark - https://trafilatura.readthedocs.io/en/latest/evaluation.html

 

[Sartre]

This piece right here is the gold nugget and there are so many ways to extract and twist the content.

For readers with Python experience, you can download images from stock photo sites like unsplash, pixabay, pexels, and use the Pillow library to add your own text on top of the image. This is also great to use for people who want to create their own memes.

+ Nider is pretty cool

exactly, this thread gave a ton of ideas. Some threads beg for some negative responses, but I'm at a loss how this thread can get negative attention.

This thread lead me down a rabbit hole on automation, for example this library and discord group: https://github.com/elebumm/RedditVideoMakerBot. Using python to generate TTS reddit vids, also very useful for inserting into blogs! I'm using a modified version for tiktok

wow I didn't know about this one!

Hello @Sartre, I'm really fascinated by your threads & postings, I only have one question if you'd like to answer and provide me with some guidance. How much time did it take you like from point 0 to the point you're able to do these amazing things with Python and to make your own automation tools? I don't have any knowledge in python, but I know some Java & C#. how much time will it take me to build my own tools "not as good as yours" but enough to get the job done? Thank you very much

This is a tough question. I'd say it depends on how much time you got and if you're willing to sit at a computer for 10hours/day and learn. Persistence is the most important skill here. I'd recommend the following:

Download Visual Studio Code. Create a Github account. Both are free.

Learning resources:

1. [Free] Start with https://automatetheboringstuff.com/ I think they have a paid video version of the book, but I prefer learning from regular books. Videos are hard to follow for me. Get a second monitor or a tablet/kindle to follow the books while you're learning.
2. [Free] Go through the https://docs.python.org/3/tutorial/ 2-3 times and type out all the code and make sure you understand it and it runs properly.
3. This book https://realpython.com/products/python-tricks-book/ - At this point you should start making your own, interesting, productive apps.
4. One year later: https://www.oreilly.com/library/view/fluent-python-2nd/9781492056348/
5. Once you know your stuff: https://www.oreilly.com/library/view/architecture-patterns-with/9781492052197/ - this book helped me a lot with making real-life apps and orchestrating bigger projects.

Don't really need to pay for any expensive courses. 2-3 books and you're good.

Make sure to never copy-paste code.

i m thinking of using this for image generation ->

Yake -> bing image scraper -> PIL -> G An.

What is your approach for Image generation?

Wouldn't be great if all of us can connect via some form of slack group?

Taking people off-BHW to groups/forums/chats is against the terms of service here. If I were you I would message a mod and ask to delete that post.

Where do you get images from? I tried Pixabay but it returned 0 results for most of the long tail queries. How do you get rid of such issues?

Unsplash. Don't use long tail queries. Do subcategories with things that are non-ambigous and universal like "yoga".

Try this - https://github.com/adbar/trafilatura

BenchMark - https://trafilatura.readthedocs.io/en/latest/evaluation.html

WOW. You just blew my mind with this. Insane accuracy. thank you <3

 

[SamiMM]

Just finished reading the whole 41 pages ! What a journey !

I did notice that your system or app as you like to call it changed a little bit over the time, first you used some GPT-3 app out there then you started tweeking your own model which you decided to completely switch and start using now ( or since march if i remember right ).

I have been looking to start something similar quite some time, i have had a couple of failed attempts to start niche sites in the past, some of them i still own to this day ( with 1 - 2 articles in average, content writing was never my thing ). I did think about something similar in the past but i never thought it could be automated to this level. I even purchased a lifetime license on an NLP AI provider last year when it was still available but never had the time to play around with it. I am studying data science now (will graduate in 6 months) and i think it's time to use the skills.

I started coding something earlier today and still trying to get a clearer image of the whole process, so i might have a couple of questions for you :

1- Do you do KW research and cleaning manually ? Or do you have a script that extracts keywords from the pages you scraped on some website ? If so, how do you know what keywords should be going together as i imagine one page will be ranked for multiple keywords ( I guess i am overthinking this ).
2- Do you still cluster classify paragraphs from different pages ? Do you do that based on articles or on H2s ? I thought about using some simple ML SVM model for that.
3- You mentionned that you are using a "simple" paraphrasing model now that is not an NLP DL model. Is it still a machine/deep learning model ? How big of a dataset did you use for training it ?
4- What would be your advice for someone starting now something similar to your project ? Start with the service i already paid for for paraphrasing ? Or create my own model/script ? I think you had better results with your own model so i am wondering how it works and how big it is.

Thanks for showing us that this is actually possible and don't forget to leave a small bite of the cake for us starting 9 months later

 

[Archemike]

childrens books genreated with walle. walle seems hot now for text to image generation. https://www.surgehq.ai//blog/generating-childrens-stories-using-gpt-3-and-dall-e

 

[prvncqr]

Are you still using your own NLP model for paraphrasing? Did you make any further improvements?

 

[prvncqr]

Most probably, the issue sides with your proxies. They are probably used to death.

I have had no problem with 2 requests/second for a few months, running 24/7 on a cheap google cloud vm.

If you are going past first page, set delays between pages.

Do you use selenium or requests?

 

[Alexion]

Do you use selenium or requests?

I'm using selenium so I can have more control and to check on possible errors if they appear.

If the proxy/ip is clean, requests will do just fine, I just like a bit more control.

You can also automate ReCaptcha with selenium, there's a library on Github that resolves reCaptcha using voice recognition, works wonders.

 

[prvncqr]

I'm using selenium so I can have more control and to check on possible errors if they appear.

If the proxy/ip is clean, requests will do just fine, I just like a bit more control.

You can also automate ReCaptcha with selenium, there's a library on Github that resolves reCaptcha using voice recognition, works wonders.

I already used the recaptcha breaker and it worked fine. However, we should limit the requests otherwise Google will block the IP. Did you face any such issues?

 

[Ruseusti]

I won't share the script for the simple reason that I'm using an AI app that doesn't have an official API and I'm breaking the terms of service and want to fly under the radar(this app isn't very well-known). It's not based on GPT-3 and it's not Quillbot.

Is the App available in the public market?

 

[Alexion]

I already used the recaptcha breaker and it worked fine. However, we should limit the requests otherwise Google will block the IP. Did you face any such issues?

I didn't face any problems as I'm using a very low rate. One IP does one PAA in ~2 minutes, I just use multiple socks5.

 

[prvncqr]

I didn't face any problems as I'm using a very low rate. One IP does one PAA in ~2 minutes, I just use multiple socks5.

Don't you get recaptcha verification or what? And socks5 proxies work better than http?

 

[SamiMM]

I didn't face any problems as I'm using a very low rate. One IP does one PAA in ~2 minutes, I just use multiple socks5.

Do you mean you are writing a whole PAA article in 2minutes or is it scraping the questions only ?

 

[Alexion]

Do you mean you are writing a whole PAA article in 2minutes or is it scraping the questions only ?

It's basically the same thing. Once 40ish questions are scraped, that's the article.