My new PAA Journey

[thegcfrolamide]

Nice one and Goodluck on your journey. Do you plan on selling your scrapper in the near future?

 

[TheDankChocolate]

Good Luck on this one. Just be watchful of the upcoming Google Update (that possibly is targetting PAAs)

I hope you are unaffected

 

[fearlessvn]

Right now I have 17K posts in the WordPress database.
Every post is separate question with answer.
I generate tags for every post based on the title.
Then I use related posts plugin which I have modified to include the related posts (based on tags, title and main body) in the main body of the primary post as title H3 and answer. So the whole post now looks like it has 20 subheadings with content but actually those 20 subheadings are real posts in the wordpress and you can open every single one of them and get different related posts for that post too.
Also, my PHP script, before adding new post to the database, compares the content of all posts and checks if there is already post with the same content. If there is, then I do not add that as new post, I only take the question and append it to the post as meta data.
Then in my template I use function to extract the meatdata and show it in the top of the post as "Similar questions"

Then I use Table of content plugin in the top of the primary post to show all the subheadings and link them for quick access.

PS: Google adsense is still reviewing my page.
PSS: I use Google keyword planner to generate keywords, that I later feed to my Python scraper

Thank you so much for your very clear answer. Hope the best with your journey.

 

[SharonHe]

Good Luck on this one. Just be watchful of the upcoming Google Update (that possibly is targetting PAAs)

I hope you are unaffected

To be honest I am really unlucky so I am sure I will get hit

Nice one and Goodluck on your journey. Do you plan on selling your scrapper in the near future?

It is more of a "system". More scripts and components working together to generate PAA sites. Right now I am also working on implementing GP2 AI generator in the "system". So it's not simply like complete script with interface. It's much more "coder" orienteadted. But why not, if the system proves it works I can sell it like that, with few improvements for easier working and tutorial.

 

[thegcfrolamide]

To be honest I am really unlucky so I am sure I will get hit


It is more of a "system". More scripts and components working together to generate PAA sites. Right now I am also working on implementing GP2 AI generator in the "system". So it's not simply like complete script with interface. It's much more "coder" orienteadted. But why not, if the system proves it works I can sell it like that, with few improvements for easier working and tutorial.

Congratulations once again on your journey and the very best while working on it

 

[SharonHe]

Current number of posts in wordpress 20.140

Indexed by google 80

Yesterday visitors 0

I installed YOAST SEO pro plugin, and google search console immediately found 20K urls in the sitemaps.

 

[light and bolts]

yea but with rotating proxies its no issue and you get much more data faster as you dont need delays like when you scrape google from 1 IP, but was just a suggestion.
if your setup works for you, then thats fine too. there are always multiple ways to do things

how do you rotate proxies?

 

[SharonHe]

I think I am in sandbox

 

[fearlessvn]

I think I am in sandbox

I think G is preparing for big update this week, so we can see many of content will be slower to index. So you should be patient during this time.

Hope everything will be better.

 

[itz_styx]

how do you rotate proxies?

use a provider that has rotating proxies, so every request is sent through a different IP.

 

[SharonHe]

use a provider that has rotating proxies, so every request is sent through a different IP.

Never thought of this. I am always relaying on my script. I use text file with ip\ort:user\ass and parse this file so I can change it on every few requests. But your idea is much more elegant.

 

[itz_styx]

i used to do this too, but rotating proxies are so much better for scraping data and pricing is similar to normal proxies.

 

[HustleTong]

What kind of AI model do you use to answer them?
Good luck!

 

[SharonHe]

What kind of AI model do you use to answer them?
Good luck!

Nothing, I just take the answer from google PAA. Now I am working to generate bigger content using neo AI, but that is still work in progress. I am limited with GPU power.

 

[sagarpatil]

use a provider that has rotating proxies, so every request is sent through a different IP.

Even if you use rotating proxies with new IP for every request, Google will still rate limit you. You need to fake your user agent and have some delay.

 

[itz_styx]

Even if you use rotating proxies with new IP for every request, Google will still rate limit you. You need to fake your user agent and have some delay.

no they wont if you get a new IP for every request and that IP hasnt been used for scraping. delays would be pointless.
faking the UA is obvious, but has not much effect on how much you can scrape. you might get a little more from the same IP, but its the number of requests they count until they show a captcha.
with rotating proxies you dont have the captcha problem and you dont need any delays so you can scrape tens of thousands in a short amount of time.
been there done that with many types of different data scraping from google.

 

[desperado596]

no they wont if you get a new IP for every request and that IP hasnt been used for scraping. delays would be pointless.
faking the UA is obvious, but has not much effect on how much you can scrape. you might get a little more from the same IP, but its the number of requests they count until they show a captcha.
with rotating proxies you dont have the captcha problem and you dont need any delays so you can scrape tens of thousands in a short amount of time.
been there done that with many types of different data scraping from google.

İ am using requests with rotating proxy with just 5 keywords and 10 multithread still getting captchas also using fake useragent.

its also giving errors i found some coders but they couldnt fix the problem. sometimes it gives pandas error sometimes other. i got really sick of it. i dont know progrramming.

i found the code on github. if you can help can you pm me ? i can pay you.

 

[sagarpatil]

no they wont if you get a new IP for every request and that IP hasnt been used for scraping. delays would be pointless.
faking the UA is obvious, but has not much effect on how much you can scrape. you might get a little more from the same IP, but its the number of requests they count until they show a captcha.
with rotating proxies you dont have the captcha problem and you dont need any delays so you can scrape tens of thousands in a short amount of time.
been there done that with many types of different data scraping from google.

Do you have any recommendations for proxies? I tried HydraProxy with new IP for every request but was still getting 429, it only worked if I put a delay of 7-10 seconds.

 

[benman]

Why do you store the q&a in database?

 

[SharonHe]

Why do you store the q&a in database?

Because I then use PHP script that connects with the wordpress framework to parse the data and post it as wordpress post together with tags and category.

also it is easier for my python scraper to skip the repeated data.