[Journey] 1 million UVs/month in 12 months using AI generated content. Let's do it!

[Sartre]

How are u able to use Curl Requests for scraping Google? doesnt using cURL bring out Recaptha just after few requests?
which mean using Curl need too many private proxies. which is a waste of resources, or did i miss out on anything ?

Using selenium uses 100x more resources because we are using full instance of google chrome with each thread. but using selenium we can avoid recaptcha 99% of the time, which avoid the requirement of proxies.

i am not using any big servers but only 2GB ram servers and running only single thread in each server. i m currently 10 of such servers. bringing out about 100,000-150,000 PAA everyday. i haven't tried multi threading as i dint have access to bigger servers

I dont collect more then 20 PAA from each keyword because after 20 i noticed the the relevancy goes down a lot.

BTW . may i know if the E3-1245V2 Xeon 4c/8t you are running is windows or Linux ? and if they are from hetzner?

I don't use curl requests. Selenium is more efficient for PAAs. You could use requests if you very elaborately forge them and emulate the javascript clicks so that Google thinks you're a real browser.

I use windows OVH for the Xeon and a private hosting for the Threadripper, also Windows. I would prefer Linux but for my whole app there is a reason now that we need to use Windows because of dependencies.

> I dont collect more then 20 PAA from each keyword because after 20 i noticed the the relevancy goes down a lot.

I store all PAAs for future use. Now I usually scrape 50-150 depending on query.

 

[Sartre]

I recommend anyone trying this to read this article, very useful: https://towardsdatascience.com/keyword-extraction-python-tf-idf-textrank-topicrank-yake-bert-7405d51cd839

Also wanted to brag that we now have our own ML model for text generation so there is 0 points of failure and it's approaching cheap-writer levels:

prompt:
How can I become more intelligent?

result:
There are many things you can do to become more intelligent.

A good place to start is by reading every day. This will help you build a deeper understanding of the world around you. Additionally, it's important to constantly question and seek clarification on what you're learning.

Diversifying your activities also helps - try different types of books, magazines, articles, etc. Finally, review information regularly and keep track of your ideas in a journal or notebook.

Allow yourself to be open-minded and change your opinions based on new evidence or perspectives.

---

Passes Copyscape premium and AI detection algorithms that I know of.





I would never have imagined this will go that far.

 

[doctorhm007]

I recommend anyone trying this to read this article, very useful: https://towardsdatascience.com/keyw...idf-textrank-topicrank-yake-bert-7405d51cd839

Also wanted to brag that we now have our own ML model for text generation so there is 0 points of failure and it's approaching cheap-writer levels:

prompt:
How can I become more intelligent?

result:
There are many things you can do to become more intelligent.

A good place to start is by reading every day. This will help you build a deeper understanding of the world around you. Additionally, it's important to constantly question and seek clarification on what you're learning.

Diversifying your activities also helps - try different types of books, magazines, articles, etc. Finally, review information regularly and keep track of your ideas in a journal or notebook.

Allow yourself to be open-minded and change your opinions based on new evidence or perspectives.

---

Passes Copyscape premium and AI detection algorithms that I know of.

View attachment 218074

View attachment 218075

I would never have imagined this will go that far.

Nice accomplishment! Were you able to achieve it alone or by hiring devs? Also, I thought you only were working on paraphrasing, did you shift to generating texts based on input?

 

[Sartre]

Nice accomplishment! Were you able to achieve it alone or by hiring devs? Also, I thought you only were working on paraphrasing, did you shift to generating texts based on input?

I developed the core algorithm alone, but outsourced most of the work since I'm working on a couple projects now. I got a full time dev now.

Yup, generating only now, no paraphrasing anymore.

 

[Alma]

I recommend anyone trying this to read this article, very useful: https://towardsdatascience.com/keyw...idf-textrank-topicrank-yake-bert-7405d51cd839

Also wanted to brag that we now have our own ML model for text generation so there is 0 points of failure and it's approaching cheap-writer levels:

prompt:
How can I become more intelligent?

result:
There are many things you can do to become more intelligent.

A good place to start is by reading every day. This will help you build a deeper understanding of the world around you. Additionally, it's important to constantly question and seek clarification on what you're learning.

Diversifying your activities also helps - try different types of books, magazines, articles, etc. Finally, review information regularly and keep track of your ideas in a journal or notebook.

Allow yourself to be open-minded and change your opinions based on new evidence or perspectives.

---

Passes Copyscape premium and AI detection algorithms that I know of.

View attachment 218074

View attachment 218075

I would never have imagined this will go that far.

literal Deus Mechanicus

 

[neOtron]

@Sartre may i know which library you like to use to match similar keywords/sentences against pandas dataframe?
I mean sure at core its all Levenshtein but you must be using some wrapper library like :
fuzzywuzzy
fuzzy_pandas
fuzzymatcher
thefuzz

May i know which library you prefer for better accuracy or you use Levenshtein directly?

 

[Sartre]

@Sartre may i know which library you like to use to match similar keywords/sentences against pandas dataframe?
I mean sure at core its all Levenshtein but you must be using some wrapper library like :
fuzzywuzzy
fuzzy_pandas
fuzzymatcher
thefuzz

May i know which library you prefer for better accuracy or you use Levenshtein directly?

We're not really using Pandas in the app anymore for the most part, everything is classified in the database. Fuzzywuzzy is a good library for this!

 

[AmberJones]

did you make any link building? how many posts do you have this time?

 

[doperdex90]

the fact is we've changed our paraphrasing model entirely 3 times already Yes, before I was doing it like you described.

Right now I created a completely original model with my dev from scratch that is a little out-of-the-box and different than most stuff out there. Works very similary to Quilbot or Wordtune but is scalable on CPUs so I'm just scaling Threadripper/EPYC servers when I need and it's much more cost-efficient for me than GPUs. I hope that makes sense and sorry I can't reveal the details right now, but I invested a lot of money into developing this.

My dev is working on a simple white paper to explain this and it will be posted in public soon.

You created a model form scratch? I find that very hard to believe. Building such models requires massive computational resources. If someone like you can build such a brilliant model with 2 threadrippers then what's keeping Quillbot in business?

 

[Sartre]

You created a model form scratch? I find that very hard to believe. Building such models requires massive computational resources. If someone like you can build such a brilliant model with 2 threadrippers then what's keeping Quillbot in business?

Oh no, of course not. I should clarify. First thing is we used GPUs to train the model. Second thing is when I say from scratch, I mean we used an existing very "base" model and fine-tuned it for a month on strong hardware.

I can't mention the details in public but I know a guy in one of the smaller AI companies and he let me self host their model.

The model they gave us was very basic, and fine-tuning it was the majority of the work, that's why I phrased it that way.

On the other hand, with current algo improvements it would be feasible to completely train your own model for as little as $10-50k. ML is really getting easier exponentially every ~2 months.

I'm also experimenting with creating images using AI. I can show some examples if anyone is interested.

 

[doperdex90]

Oh no, of course not. I should clarify. First thing is we used GPUs to train the model. Second thing is when I say from scratch, I mean we used an existing very "base" model and fine-tuned it for a month on strong hardware.

I can't mention the details in public but I know a guy in one of the smaller AI companies and he let me self host their model.

The model they gave us was very basic, and fine-tuning it was the majority of the work, that's why I phrased it that way.

On the other hand, with current algo improvements it would be feasible to completely train your own model for as little as $10-50k. ML is really getting easier exponentially every ~2 months.

I'm also experimenting with creating images using AI. I can show some examples if anyone is interested.

Now that make sense

 

[doperdex90]

How much epmv are you getting on Mediavine? Have your pagespeed scores dropped after adding Mediavine to your site?

 

[Sartre]

How much epmv are you getting on Mediavine? Have your pagespeed scores dropped after adding Mediavine to your site?

Page speed scores dropped very slighly, but I'm still in the green on GSC.

 

[Daemon]

the method to avoid captcha with using python selenium is Do not call the google search result page directly as you will get blocked soon (which is what required to be done in CURL) .
The trick is to open google.com home page and then simulate typing keywords and then simulate hitting enter button.. This way you wont get find captcha unless ur IP is already shady in the eyes of google.

That's a great trick Is it possible to just pass Google homepage as a spoofed referrer? Or would Google's security not count that?

Though the people_also_ask library uses Python Requests, not Selenium. So that's why I was surprised why I wasn't blocked.

This library, GQuestions, uses Chromium via Selenium:
https://github.com/nittolese/gquestions

However I think it needs to be updated.

 

[neOtron]

That's a great trick Is it possible to just pass Google homepage as a spoofed referrer? Or would Google's security not count that?

Though the people_also_ask library uses Python Requests, not Selenium. So that's why I was surprised why I wasn't blocked.

This library, GQuestions, uses Chromium via Selenium:
https://github.com/nittolese/gquestions

However I think it needs to be updated.

seems like i already jinxed it. i m getting recaptchas today

i dint try to spoof so i cant comment on it.

regarding https://github.com/nittolese/gquestions . it good for scraping just questions. however i m scraping both quesers + answers

 

[desperado596]

captcha breaking is not cost-efficient. With Selenium running 70-100 threads I can scrape ~1 mil PAAs in 1 day with 100 proxies without any captchas.

Are thouse proxies rotation proxy ? 100 rotations proxies must cost a lot ?
can you give more details please how much cost for you

 

[Sartre]

Are thouse proxies rotation proxy ? 100 rotations proxies must cost a lot ?
can you give more details please how much cost for you

I've got 1000 cheap proxies + 250 good ones, but running a pretty big operation.

 

[cafaga]

I recommend anyone trying this to read this article, very useful: https://towardsdatascience.com/keyw...idf-textrank-topicrank-yake-bert-7405d51cd839

Also wanted to brag that we now have our own ML model for text generation so there is 0 points of failure and it's approaching cheap-writer levels:

prompt:
How can I become more intelligent?

result:
There are many things you can do to become more intelligent.

A good place to start is by reading every day. This will help you build a deeper understanding of the world around you. Additionally, it's important to constantly question and seek clarification on what you're learning.

Diversifying your activities also helps - try different types of books, magazines, articles, etc. Finally, review information regularly and keep track of your ideas in a journal or notebook.

Allow yourself to be open-minded and change your opinions based on new evidence or perspectives.

---

Passes Copyscape premium and AI detection algorithms that I know of.

View attachment 218074

View attachment 218075

I would never have imagined this will go that far.

So now you use the generated text to answer the first question only, or are you now answering all questions with AI?

 

[Rufin jazz]

Oh no, of course not. I should clarify. First thing is we used GPUs to train the model. Second thing is when I say from scratch, I mean we used an existing very "base" model and fine-tuned it for a month on strong hardware.

I can't mention the details in public but I know a guy in one of the smaller AI companies and he let me self host their model.

The model they gave us was very basic, and fine-tuning it was the majority of the work, that's why I phrased it that way.

On the other hand, with current algo improvements it would be feasible to completely train your own model for as little as $10-50k. ML is really getting easier exponentially every ~2 months.

I'm also experimenting with creating images using AI. I can show some examples if anyone is interested.

Can you please post some example of images you are creating using AI.

 

[Sartre]

Can you please post some example of images you are creating using AI.