[Journey] 1 million UVs/month in 12 months using AI generated content. Let's do it!

[Sartre]

I initially made a setup but was not happy with the results of the scraping (how to get relevant paragraphs) so went back to the start trying semi manually to try and figure out how to automate. I am still kind of stuck here.

I tried the paa question and answer way and paraphrased just the answers, to avoid the problem, but then it looked like a 'paa site'. I would like to have articles which look like 'normal' articles but unsure how to scrape content there.

So currently for me I am making articles by manually going to the serps for main kw phrase, then opening up the top ten. I then take some decent paragraphs from the results by looking through each article manually for relevant stuff and place them into the article template. I repeat the process for a few more paa question searches until it has over 1k words.

Then I run it through pegasus and will manually edit the result so it makes sense. This seems to be a similar process to the semi automated way the other seos use with commercial ai tools.

This produces good articles I am happy to post, rather than the semi coherent stuff the bots came up with doing it totally automated, however it still takes a good chunk of my time and of course the goal is automation.

What are some tips to be able to have a higher degree of accuracy via bot to automate this process of choosing paragraphs that will make sense in the article as a whole as this is the main sticking point currently in terms of automation. There is so much variance in articles and content I am unsure what logic/nlp to use in order to get reliable results in what the bot would pull.

I know I could pull the h2s and 1-3 paragraphs beneath from articles, but still not sure how to parse for coherent content.

well this is so complex and difficult that it's beyond the scope of a simple post on a forum.

First step is generating an outline that you're satisfied with. something that looks like a real article. not only paa titles. then you can start thinking about content.

 

[nreispqwi]

well this is so complex and difficult that it's beyond the scope of a simple post on a forum.

First step is generating an outline that you're satisfied with. something that looks like a real article. not only paa titles. then you can start thinking about content.

Thanks for the response.

Well I have the general idea of an outline already since I have done it a couple of times the semi manual way mentioned in my last post and also fully manually.

The titles/headings are the easy bit however the garnering content to fill them, where the content is going to being relevant and flow from one paragraph to the next is the tricky part which I was asking advice on.

If grabbing paragraphs directly from the heading of a scraped article that could help with that issue but still there is a lot of rubbish in articles that have to be weeded out. You have things like check lists and stuff where the continuity is ruined when you just pull one part.

Maybe an in depth explanation would be beyond the scoep but how about a check list of a few libraries which would be relevant to the task I can go off and study just to point me in the right direction .

Thanks.

Btw what is the problem with ripping a whole article and just paraphrasing that? I haven't seen it mentioned and I imagine there is a good reason people don't do it but wondered what the issue there is? Would google not rank that?

 

[Sartre]

Btw what is the problem with ripping a whole article and just paraphrasing that? I haven't seen it mentioned and I imagine there is a good reason people don't do it but wondered what the issue there is? Would google not rank that?

you will eventually get into DMCA problems if you're doing thousands of articles.

I will try to answer the rest in a few days, I've got a personal problem to deal with.

 

[gerald23]

interesting read

 

[Archemike]

https://bigscience.huggingface.co/blog/bloom
Who's going to start using bloom?

 

[BlogPro]

https://bigscience.huggingface.co/blog/bloom
Who's going to start using bloom?

Already on it.

This and Yandex. The AI game is about to get a ton more interesting.

I really need to save up and invest in some A100s.

 

[Daemon]

I found this library: lagranges/people_also_ask

It was last updated on May 15th and works, though the documentation is a bit thin. Seems pretty easy to use.

I ran "pip install people_also_ask" and created a file with the following contents to print basic results. You un-comment one "paaresult" line at a time:

import people_also_ask
from pprint import pprint

#paaresult = people_also_ask.get_related_questions("coffee")
#paaresult = people_also_ask.get_related_questions("coffee", 5)
#paaresult = people_also_ask.get_answer("Why is coffee bad for you?")
#paaresult = people_also_ask.get_simple_answer("Why is coffee bad for you?")

#pprint(paaresult)

#paaresult = people_also_ask.generate_related_questions("coffee")
paaresult = people_also_ask.generate_answer("coffee")

limit = 5
for index, nextresult in zip(range(limit), paaresult):
        print(index,":","="*50)
        pprint(nextresult)

The "people_also_ask.generate_" functions return Generator objects. I think these are generated on the fly as the generator is looped through, so you have to have a limit.

I'm not sure how hard it would be to get it to work with a proxy, or a captcha solver. I don't know very much Python so correct me if I'm wrong, but I think this is the file you would edit if you wanted to add proxies: https://github.com/lagranges/people_also_ask/blob/master/people_also_ask/google.py

Based on this article I think you could add proxies by adding lines like:

SESSION.proxies = {
   'http': 'http://10.10.10.10:8000',
   'https': 'http://10.10.10.10:8000',
}

After line 29, "SESSION = requests.Session()".

I don't know how you could integrate it with a Catptcha solver like these though:

• https://github.com/MacKey-255/GoodByeCatpcha
• https://github.com/ohyicong/recaptcha_v2_solver

Does anyone know how to do that?

 

[Takobeh]

If you want to do it seriously and success will be reachable, hire a content writer - a good one. Publish new, useful, and trending content that is worth your audience reading it.

 

[Daemon]

captcha breaking is not cost-efficient. With Selenium running 70-100 threads I can scrape ~1 mil PAAs in 1 day with 100 proxies without any captchas.

How many PAAs do you scrape for each keyword/SERP? How long does each keyword/SERP take to scrape all the PAAs?

70-100 threads seems like a lot, doesn't that use up a ton of RAM? My server will start giving errors with only 30 concurrent cURL threads, Selenium I'm assuming takes up way more resources, so I'm surprised a machine can go that high. Maybe my server is just crappy though I guess

 

[nextlvlig]

I found this library: lagranges/people_also_ask

It was last updated on May 15th and works, though the documentation is a bit thin. Seems pretty easy to use.

I ran "pip install people_also_ask" and created a file with the following contents to print basic results. You un-comment one "paaresult" line at a time:

import people_also_ask
from pprint import pprint

#paaresult = people_also_ask.get_related_questions("coffee")
#paaresult = people_also_ask.get_related_questions("coffee", 5)
#paaresult = people_also_ask.get_answer("Why is coffee bad for you?")
#paaresult = people_also_ask.get_simple_answer("Why is coffee bad for you?")

#pprint(paaresult)

#paaresult = people_also_ask.generate_related_questions("coffee")
paaresult = people_also_ask.generate_answer("coffee")

limit = 5
for index, nextresult in zip(range(limit), paaresult):
        print(index,":","="*50)
        pprint(nextresult)

The "people_also_ask.generate_" functions return Generator objects. I think these are generated on the fly as the generator is looped through, so you have to have a limit.

I'm not sure how hard it would be to get it to work with a proxy, or a captcha solver. I don't know very much Python so correct me if I'm wrong, but I think this is the file you would edit if you wanted to add proxies: https://github.com/lagranges/people_also_ask/blob/master/people_also_ask/google.py

Based on this article I think you could add proxies by adding lines like:

SESSION.proxies = {
   'http': 'http://10.10.10.10:8000',
   'https': 'http://10.10.10.10:8000',
}

After line 29, "SESSION = requests.Session()".

I don't know how you could integrate it with a Catptcha solver like these though:

• https://github.com/MacKey-255/GoodByeCatpcha
• https://github.com/ohyicong/recaptcha_v2_solver

Does anyone know how to do that?

Yea I'm using this one right now and tweaked it a bit. The issue I think with this repo is that it sends a lot of requests. For example, when you "get_answer" for a question it lists everything on the page, but when you want an answer to the related questions, it doesn't use the source code it already generated. It makes another search to Google to find the featured snippet for that question. So instead of having 5 requests to find questions and answer for 16 PAAs, It makes 16 requests which makes the process much slower.

But overall it's really good and easy to use. It covers a lot of different featured snippet cases making it pretty versatile for searches. I would test different searches, see what results you get, and tweak the code accordingly to what you want to see.

I would add your proxies to the SESSION.get function. For example, response = SESSION.get(URL, params=params, headers=HEADERS, proxies=PROXIES). You can test this setup by changing the URL to a site that checks your IP and return the response to see if the IP is changing properly.

And I don't think you'll need a captcha solver. You need a rotating proxy that changes IPs whenever you send a request.

 

[ashtonwright]

Interesting. I'm going to follow this. Continued luck!

 

[ismail777]

Hello, I would like to get involved. If you want someone to recheck with you those articles. I'm your guy.

 

[Sartre]

How many PAAs do you scrape for each keyword/SERP? How long does each keyword/SERP take to scrape all the PAAs?

70-100 threads seems like a lot, doesn't that use up a ton of RAM? My server will start giving errors with only 30 concurrent cURL threads, Selenium I'm assuming takes up way more resources, so I'm surprised a machine can go that high. Maybe my server is just crappy though I guess

40 cURL threads taking CPU or RAM? How is that even possible? What kind of hardware are you running?

I'm running 22 Selenium threads on an old E3-1245V2 Xeon 4c/8t, 100% load.

70 Selenium threads on a Threadripper 24c/48t with 30-40% CPU load.

Remember you can disable loading images and embeds in Selenium.

 

[Cpt_Ghost]

40 cURL threads taking CPU or RAM? How is that even possible? What kind of hardware are you running?

I'm running 22 Selenium threads on an old E3-1245V2 Xeon 4c/8t, 100% load.

70 Selenium threads on a Threadripper 24c/48t with 30-40% CPU load.

Remember you can disable loading images and embeds in Selenium.

Noob here: Is there possible to run something similar on a cheap VPS or solo computer?

 

[ownz]

40 cURL threads taking CPU or RAM? How is that even possible? What kind of hardware are you running?

SSL/TLS handshakes costs cpu cycles. Too many simultaneous requests = high cpu load.

 

[neOtron]

40 cURL threads taking CPU or RAM? How is that even possible? What kind of hardware are you running?

I'm running 22 Selenium threads on an old E3-1245V2 Xeon 4c/8t, 100% load.

70 Selenium threads on a Threadripper 24c/48t with 30-40% CPU load.

Remember you can disable loading images and embeds in Selenium.

How are u able to use Curl Requests for scraping Google? doesnt using cURL bring out Recaptha just after few requests?
which mean using Curl need too many private proxies. which is a waste of resources, or did i miss out on anything ?

Using selenium uses 100x more resources because we are using full instance of google chrome with each thread. but using selenium we can avoid recaptcha 99% of the time, which avoid the requirement of proxies.

i am not using any big servers but only 2GB ram servers and running only single thread in each server. i m currently 10 of such servers. bringing out about 100,000-150,000 PAA everyday. i haven't tried multi threading as i dint have access to bigger servers

I dont collect more then 20 PAA from each keyword because after 20 i noticed the the relevancy goes down a lot.

BTW . may i know if the E3-1245V2 Xeon 4c/8t you are running is windows or Linux ? and if they are from hetzner?

 

[neOtron]

Noob here: Is there possible to run something similar on a cheap VPS or solo computer?

u can always run single thread like me using $5 VPS such server can run about 1 query per minute collecting about 20 questions per Query. i.e. total 30000 PAA per day. but as tonss of PAA will get repeated so i you will barely get 10000, 15000 PAA per day

 

[Daemon]

The issue I think with this repo is that it sends a lot of requests. For example, when you "get_answer" for a question it lists everything on the page, but when you want an answer to the related questions, it doesn't use the source code it already generated. It makes another search to Google to find the featured snippet for that question. So instead of having 5 requests to find questions and answer for 16 PAAs, It makes 16 requests which makes the process much slower.

Really? I was messing around with it a bunch, wasn’t using any proxies. I’m surprised Google didn’t block me.

Is there something about Python Requests that makes Google more tolerant of it than PHP cURL?

And I don't think you'll need a captcha solver. You need a rotating proxy that changes IPs whenever you send a request.

OK thanks. How many requests per proxy IPs/IP rotation would you recommend? At what scale daily do you scrape?

 

[Daemon]

40 cURL threads taking CPU or RAM? How is that even possible? What kind of hardware are you running?

Well it's 30 instances of a single PHP file that uses PHP cURL and saves the output. More than about 30 concurrent instances (using "&" at the command line) and I start getting "bash error". Sorry for not being more precise. The use of PHP probably limits the concurrency as compared to just command line curl.

I have an Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz with 8 processors and 16GB RAM.

I'm running 22 Selenium threads on an old E3-1245V2 Xeon 4c/8t, 100% load.

70 Selenium threads on a Threadripper 24c/48t with 30-40% CPU load.

Ah OK that makes sense, that's way above my machine.

How many PAAs do scrape per SERP? Sorry if you've answered that already but I couldn't find it in the thread

 

[neOtron]

Really? I was messing around with it a bunch, wasn’t using any proxies. I’m surprised Google didn’t block me.

Is there something about Python Requests that makes Google more tolerant of it than PHP cURL?

OK thanks. How many requests per proxy IPs/IP rotation would you recommend? At what scale daily do you scrape?

the method to avoid captcha with using python selenium is Do not call the google search result page directly as you will get blocked soon (which is what required to be done in CURL) .
The trick is to open google.com home page and then simulate typing keywords and then simulate hitting enter button.. This way you wont get find captcha unless ur IP is already shady in the eyes of google.

You can scrape up to 500 or even more paa with just one google search but the relavancy keeps dropping as you go on.
Sarte metntioned earlier that he scrapes about 40 PAA with one query. i scrape only 20 PAA per query as i find relavenacy drops after 20 PAA