[Journey] 1 million UVs/month in 12 months using AI generated content. Let's do it!

[bbloverz]

I still have a lot to learn

 

[Sartre]

I still have a lot to learn

I'm learning all my life. It's a privilege to be able to do that.

 

[neOtron]

I just use a broad keyword as if i have a website about female fitness i will just use 'girl working out'. Or if you want to be specific you could try yake. It's a Python lib.

That's awesome that you took action and posted 80 posts in 2 days mate!!

are you using Yake in your project or any other keyword extractor lib? i tried Yake and tested with few articles but the data is provides is too inconsistent and non relevant. i am working on a similar projects but this part is one the biggest headache for me.

Example. lets say is feed it an article
towardsdatascience.com/web-scraping-news-articles-in-python-9dd605799558

and it gives in e data like :

[
[
"HTML code",
0.006774771507866986
],
[
"HTML",
0.008319411752053977
],
[
"Web Scraping",
0.013292987709593118
],
[
"articles",
0.014517443148616896
],
[
"articles web scraping",
0.01569511281132911
],
[
"article",
0.016591363598419312
],
[
"content",
0.017878965914899715
],
[
"text",
0.018162606644746553
],
[
"code",
0.018178441084268044
],
[
"HTML Web scraping",
0.02051221395701447
]
]

can you please tell me How do you figure out the relevant keywords from the about result. as clearly the lowest score keyword is not relevant . is there any better way to achieve this?
my intention is to get some relevant keywords and generate People also ask questions related to the topic .

 

[Niffo]

are you using Yake in your project or any other keyword extractor lib? i tried Yake and tested with few articles but the data is provides is too inconsistent and non relevant. i am working on a similar projects but this part is one the biggest headache for me.

Example. lets say is feed it an article
towardsdatascience.com/web-scraping-news-articles-in-python-9dd605799558

and it gives in e data like :


can you please tell me How do you figure out the relevant keywords from the about result. as clearly the lowest score keyword is not relevant . is there any better way to achieve this?
my intention is to get some relevant keywords and generate People also ask questions related to the topic .

You should probably give yake only the text, not the complete html. You can use beautifulsoup for this.

 

[neOtron]

You should probably give yake only the text, not the complete html. You can use beautifulsoup for this.

Yes ofcos. I meant taking article from that web page. .I extracted plain text article with newspaper3 then fed this article to yake

 

[Sartre]

Yes ofcos. I meant taking article from that web page. .I extracted plain text article with newspaper3 then fed this article to yake

this. I extract the top 10 serps for all keywords using a SERPs API, then newspaper3k (add cloudscraper for better success rate) and calculate topical relevancy between keywords.

 

[Sartre]

sorry for double post, but couldn't edit anymore:

import cloudscraper import newspaper def get_article(url: str) -> tuple: try: print(f"Getting {url}") scraper = cloudscraper.create_scraper() html = scraper.get(url).content article = newspaper.Article(url=" ") article.set_html(html) article.parse() article.nlp() return (article.title, article.text) except Exception as error: print(error) return ("", "")

cloudscraper is very important, as ~40% of the sites run cloudflare

 

[neOtron]

sorry for double post, but couldn't edit anymore:

import cloudscraper import newspaper def get_article(url: str) -> tuple: try: print(f"Getting {url}") scraper = cloudscraper.create_scraper() html = scraper.get(url).content article = newspaper.Article(url=" ") article.set_html(html) article.parse() article.nlp() return (article.title, article.text) except Exception as error: print(error) return ("", "")

cloudscraper is very important, as ~40% of the sites run cloudflare

thanks for the tip, i integrated cloudscraper in my code as i was still experimenting so i never came across any site scarping restrictions.

Regarding this :

extract the top 10 serps for all keywords using a SERPs API

This is a big headache. i tried my best to avoid those Serp Apis till now. i have about 20 servers each have proxy servers installed so i was relying on them to scrape google. i am only doing 1 request per minute per proxy/server to avoid any ip blocks. but i guess there is no way around but to look for some scalable solution in terms of Google scraping power , as the amount of google request in this project keep increasing exponentially lol.

Btw did u find any noteworthy grammar fixing lib or something similar to that?

Thanks again for your inputs



.

 

[Niffo]

thanks for the tip, i integrated cloudscraper in my code as i was still experimenting so i never came across any site scarping restrictions.

Regarding this :

This is a big headache. i tried my best to avoid those Serp Apis till now. i have about 20 servers each have proxy servers installed so i was relying on them to scrape google. i am only doing 1 request per minute per proxy/server to avoid any ip blocks. but i guess there is no way around but to look for some scalable solution in terms of Google scraping power , as the amount of google request in this project keep increasing exponentially lol.

Btw did u find any noteworthy grammar fixing lib or something similar to that?

Thanks again for your inputs



.

Scraping google isn' that hard from my experience. Can run 6 concurrent scripts scraping google on one good ip. For insance a 4/5G connection from your phone. Scraped arround 60K serps a day, not a single captcha. Can easily scale with more 4G sims and laptops. Also if u live near public wifi, u can abuse those. Those ip's have a great reputation because of all the real users using them, bet u can easily run 20+ concurrent scrapers on there.

 

[Alexion]

thanks for the tip, i integrated cloudscraper in my code as i was still experimenting so i never came across any site scarping restrictions.

Regarding this :

This is a big headache. i tried my best to avoid those Serp Apis till now. i have about 20 servers each have proxy servers installed so i was relying on them to scrape google. i am only doing 1 request per minute per proxy/server to avoid any ip blocks. but i guess there is no way around but to look for some scalable solution in terms of Google scraping power , as the amount of google request in this project keep increasing exponentially lol.

Btw did u find any noteworthy grammar fixing lib or something similar to that?

Thanks again for your inputs



.

If you want to get cheap and get your own google results in a fast manner, I suggest you use this

https://github.com/ohyicong/recaptcha_v2_solver
I've been using it for a few months, and so far works like a charm.
The downside is you'll have to integrate it with Selenium. But I see that as a win-win scenario anyway.

 

[neOtron]

If you want to get cheap and get your own google results in a fast manner, I suggest you use this

https://github.com/ohyicong/recaptcha_v2_solver
I've been using it for a few months, and so far works like a charm.
The downside is you'll have to integrate it with Selenium. But I see that as a win-win scenario anyway.

Thanks I will try it right away

 

[Sartre]

Btw did u find any noteworthy grammar fixing lib or something similar to that?

Not really possible to make it perfectly yet imo. Even grammarly makes some bullshit mistakes often.

 

[absolvo]

but first sites have been accepted by ezoic.

how are earnings doing?

 

[herr]

Looks like an automated way of Advanced Find and Replace, if you know it.

Very interesting though

 

[spectrejoe]

this. I extract the top 10 serps for all keywords using a SERPs API, then newspaper3k (add cloudscraper for better success rate) and calculate topical relevancy between keywords.

When you say you calculate topical relevancy between keywords, you using NLP correct? Do you have examples of this? When extracting different parts from each article you scrape, how are you verifying that its not talking about the same thing you just wrote beforehand?

 

[Sartre]

When you say you calculate topical relevancy between keywords, you using NLP correct? Do you have examples of this? When extracting different parts from each article you scrape, how are you verifying that its not talking about the same thing you just wrote beforehand?

I do this to gather them into clusters. Not to put them inside of articles.

 

[Arceflow]

To the other guy - there's no ready made tool to do something like this well. At least I haven't seen one. You gotta build it yourself with your own twists. I can give you advice but I'm not going to release my entire sourcecode. It's also not for sale.

I would like to hire a developer for my custom tool. How much do you think is the right price to pay a developer to create a custom tool?

 

[Sartre]

I would like to hire a developer for my custom tool. How much do you think is the right price to pay a developer to create a custom tool?

it depends. I would say a decent developer starts from $50-60k/ year. full time

 

[neOtron]

If you want to get cheap and get your own google results in a fast manner, I suggest you use this

https://github.com/ohyicong/recaptcha_v2_solver
I've been using it for a few months, and so far works like a charm.
The downside is you'll have to integrate it with Selenium. But I see that as a win-win scenario anyway.

How do solve the below issue? i can only solve one recaptcha from one proxy. after the first try i get this message everytime the bot clicks the play button.
i tried both residential and 4g proxy.

 

[Sartre]

How do solve the below issue? i can only solve one recaptcha from one proxy. after the first try i get this message everytime the bot clicks the play button.
i tried both residential and 4g proxy.
View attachment 204062

why do you need to solve proxies? I've spent like $200 on SERP APIs in 2 months and have a shit ton of content up. I think I've got around 6 million serp entries in my database right now.