[Journey] 1 million UVs/month in 12 months using AI generated content. Let's do it!

[EmirEm]

Right, that's the thing though, I can get 500-1000 visitors a day from Google using unique content and with 100-200 articles on "whitehat sites" with no backlinks but I can't even get more than 30 clicks a day from Google on a 24k posts website? There must be something I'm missing, especially since the 24k posts site actually got quite some decent backlinks over the year. Which is why I think that Google definitely factors in content uniqueness

Do you made 100-200 posts with AI as well?

 

[spectrejoe]

Do you made 100-200 posts with AI as well?

I use AI tools on my main websites like Sensit.ai and Jarvis, to help me write articles & add extra content, however, the articles are created & edited by myself for those 100-200 posts.

 

[Sartre]

I use AI tools on my main websites like Sensit.ai and Jarvis, to help me write articles & add extra content, however, the articles are created & edited by myself for those 100-200 posts.

did u put all these articles at once? Start a site, put a few posts, let them rank, add more. Post them in order of increasing competition too. And they have to be good, unless you have a massive PBN. That's just my broscience tho.

Also are these 20k posts indexed? If they are and you're not getting clicks then the content sucks.

 

[spectrejoe]

did u put all these articles at once? Start a site, put a few posts, let them rank, add more. Post them in order of increasing competition too. And they have to be good, unless you have a massive PBN. That's just my broscience tho.

Also are these 20k posts indexed? If they are and you're not getting clicks then the content sucks.

This AI site was built using GPT-2 not using my current AI, but there are 11k posts indexed and still only getting 20-40 clicks per day from Google & 500-800 from Bing/Yahoo/Duckduckgo

I also thought that posting speed was the reason why since its a clear flag for google if someone posts thousands upon thousands of posts to a website in a short time span, however, I still see others doing it and having success so I have conflicting thoughts about it, idk.

 

[RowdySadjit]

I'm just running plain old Python locally using virtualenv, Python 3.6.8 actually, because some of the dependencies (Nider) don't run on newer versions.

At first, I was using Spacy to calculate sentence similarity using a ML model, but it took MUCH more time and yielded just a bit better results than plain old counting of repeated words.

I'm also thinking about adding a function to turning photos from Unsplash into illustrations and make some stuff kinda like WikiHow.

Ok so I am about 2/3 through the thread and trying to get a grasp of all the libraries and what their purpose is. I was having a hard time figuring out the purpose of spacy and sentencepiece as well as Levenshtein distance in the stack. Are you saying here they are redundant now? Were they just used for choosing the most unique paraphrased output from Pegasus rather than changing the text in any way?

I had been thinking while I was reading why not just use standard libraries and a bit of maths to figure this out and it seems this is what you settled on after the complicated ML libraries proved too costly. Is that a correct appraisal?

So should I just skip those other libs or do they have some utility still within the whole process, perhaps elsewhere or something I missed that they still are used for? Struggling to connect the dots with the ML/NLP libs tbh, the pegasus one was obvious.

 

[RowdySadjit]

Btw this thread has been awesome, not only for OPs great wealth of knowledge sharing but also loads of other active marketers sharing useful info between each other.

Reminds me of how things used to be on bhw a few years ago before every post seemed to become just veiled product shilling threads of useless buzzfeed type info or just 'no spoonfeeding!' responses.

Glad to see there is still gold to be had here!

 

[hwp994108018]

yes to both

they come from my database, I scrape the serps for each keyword and top 10 articles, find the most popular n-grams in each article to extract the most relevant phrases for each keyword. My app is over 10k lines of code at this point, after refactoring and optimizing it twice.

you have to dig deeper. there are cheaper options, and also I only check keywords with volume.

they got hit by manual action, traffic almost down to 0.

My sites are still doing fine, they are not as spammy. I'm working now to make sure that they will pass manual reviews.

Unfortunately I was sick(almost ended up in the hospital) for almost 7 days, but yeah, things are still going very fine:

View attachment 199949

Hello
You're using Google Analytics to track traffic, right?
Do you create a Google Analytics account for each website or track traffic for all websites with one account?

Another thing bothering me: IP addresses.
Is it necessary to buy an IP address for each website....?

I did some research through Google, and when webmasters create automated blogs in bulk, they hide "personal traces" as much as possible to avoid Google knowing that the sites belong to the same person. Could you share your thoughts, thank you.

Thank you for your time,
Huang

 

[RowdySadjit]

Question to all: I had a thought about the method Sartre has mentioned of the scraping of the top 10 results for a given search suggestion in order to get a paragraph from each to form your own article.

If we are targetting underserved kws then there shouldn't, if you did good kw research, be relevant info existing already to answer that question. It is kind of a catch 22 no? In that you want the least competitive stuff which by essence will have bad results where the question will not already have been answered accurately yet.

As such how do you gather accurate article content to scrape for content for your own article?

 

[mdevo]

Question to all: I had a thought about the method Sartre has mentioned of the scraping of the top 10 results for a given search suggestion in order to get a paragraph from each to form your own article.

If we are targetting underserved kws then there shouldn't, if you did good kw research, be relevant info existing already to answer that question. It is kind of a catch 22 no? In that you want the least competitive stuff which by essence will have bad results where the question will not already have been answered accurately yet.

As such how do you gather accurate article content to scrape for content for your own article?

Well, the thing you want to achieve is to meet search intent. You want to answer the main topic containted in your post title + preferably few other questions that are related to main topic. How you do it, depends on you. It's called topical authority.
There is no possibility to answer questions that haven't been answered yet - reason is quite simple - web contains answer to almost every question. You want to target keywords with good volume (obviously not really competetive keywords), preferably longtails that will bring you traffic. It's volume game.

 

[Shaybib]

It is kind of a catch 22 no?

It is... but mind you in this case google also not know the answer so he might trust your answer to be true and show it for seaechers

 

[RowdySadjit]

Can someone explain how pegasus works, specifically, is it 'phoning home' and accessing some data servers to run it every time you call it from python because I notice there is a long pause every time I start the script (this is even before it starts computing the sentences, more like it is maybe accessing something on the net.

If it is phoning home it makes me concerned they may cut off the supply sometime. So it is really doing that each time (after you download the transformers of course which is a one shot deal) or does it just take a bit to fire up locally (maybe just me as my computer is almost 10 years old). It is important to know whether it has dependencies on anywhere other than locally though for future concerns.

Also, when I run it there is a ton of output spat out to the terminal, how do you run it silently? Of course it could be dev nulled but would like to know if there is an unbuilt function for it first to switch on/off. Of course it should be on while testing things but want to know how to suppress when not needed.

This stuff:

    loading configuration file https://huggingface.co/tuner007/pegasus_paraphrase/resolve/main/co
    nfig.json from cache at /home/user/.cache/huggingface/transformers/d6a784b31cbe212ac3dabbc7
    8bc4e454cf9d8a1b11ff2ef1ba1c7497f2bbfb33.7a3e093739f407bcc025e64dfc5244f91bc573ed4285cba53de9
    0f960cbce58e
    Model config PegasusConfig {
      "_name_or_path": "tuner007/pegasus_paraphrase",
      "activation_dropout": 0.1,
      "activation_function": "relu",
      "add_bias_logits": false,
      "add_final_layer_norm": true,
      "architectures": [
        "PegasusForConditionalGeneration"
      ],
      "attention_dropout": 0.1,
      "bos_token_id": 0,
      "classif_dropout": 0.0,
      "classifier_dropout": 0.0,  
      "d_model": 1024,
      "decoder_attention_heads": 16,
      "decoder_ffn_dim": 4096,
      "decoder_layerdrop": 0.0,
      "decoder_layers": 16,
      "decoder_start_token_id": 0,
      "dropout": 0.1,
      "encoder_attention_heads": 16,
      "encoder_ffn_dim": 4096,
      "encoder_layerdrop": 0.0,
      "encoder_layers": 16,
      "eos_token_id": 1,
      "extra_pos_embeddings": 1,  
      "force_bos_token_to_be_generated": false,
      "forced_eos_token_id": 1,
      "id2label": {
        "0": "LABEL_0",
        "1": "LABEL_1",
        "2": "LABEL_2"
      },
      "init_std": 0.02,
      "is_encoder_decoder": true,
      "label2id": {
        "LABEL_0": 0,
        "LABEL_1": 1,
        "LABEL_2": 2
      },
      "length_penalty": 0.8,
      "max_length": 60,
      "max_position_embeddings": 60,
      "model_type": "pegasus",
      "normalize_before": true,
      "normalize_embedding": false,
      "num_beams": 8,
      "num_hidden_layers": 16,
      "pad_token_id": 0,
      "scale_embedding": true,
      "static_position_embeddings": true,
      "transformers_version": "4.18.0",
      "use_cache": true,
      "vocab_size": 96103
    }

 

[neOtron]

Can someone explain how pegasus works, specifically, is it 'phoning home' and accessing some data servers to run it every time you call it from python because I notice there is a long pause every time I start the script (this is even before it starts computing the sentences, more like it is maybe accessing something on the net.

If it is phoning home it makes me concerned they may cut off the supply sometime. So it is really doing that each time (after you download the transformers of course which is a one shot deal) or does it just take a bit to fire up locally (maybe just me as my computer is almost 10 years old). It is important to know whether it has dependencies on anywhere other than locally though for future concerns.

Also, when I run it there is a ton of output spat out to the terminal, how do you run it silently? Of course it could be dev nulled but would like to know if there is an unbuilt function for it first to switch on/off. Of course it should be on while testing things but want to know how to suppress when not needed.

This stuff:

    loading configuration file https://huggingface.co/tuner007/pegasus_paraphrase/resolve/main/co
    nfig.json from cache at /home/user/.cache/huggingface/transformers/d6a784b31cbe212ac3dabbc7
    8bc4e454cf9d8a1b11ff2ef1ba1c7497f2bbfb33.7a3e093739f407bcc025e64dfc5244f91bc573ed4285cba53de9
    0f960cbce58e
    Model config PegasusConfig {
      "_name_or_path": "tuner007/pegasus_paraphrase",
      "activation_dropout": 0.1,
      "activation_function": "relu",
      "add_bias_logits": false,
      "add_final_layer_norm": true,
      "architectures": [
        "PegasusForConditionalGeneration"
      ],
      "attention_dropout": 0.1,
      "bos_token_id": 0,
      "classif_dropout": 0.0,
      "classifier_dropout": 0.0, 
      "d_model": 1024,
      "decoder_attention_heads": 16,
      "decoder_ffn_dim": 4096,
      "decoder_layerdrop": 0.0,
      "decoder_layers": 16,
      "decoder_start_token_id": 0,
      "dropout": 0.1,
      "encoder_attention_heads": 16,
      "encoder_ffn_dim": 4096,
      "encoder_layerdrop": 0.0,
      "encoder_layers": 16,
      "eos_token_id": 1,
      "extra_pos_embeddings": 1, 
      "force_bos_token_to_be_generated": false,
      "forced_eos_token_id": 1,
      "id2label": {
        "0": "LABEL_0",
        "1": "LABEL_1",
        "2": "LABEL_2"
      },
      "init_std": 0.02,
      "is_encoder_decoder": true,
      "label2id": {
        "LABEL_0": 0,
        "LABEL_1": 1,
        "LABEL_2": 2
      },
      "length_penalty": 0.8,
      "max_length": 60,
      "max_position_embeddings": 60,
      "model_type": "pegasus",
      "normalize_before": true,
      "normalize_embedding": false,
      "num_beams": 8,
      "num_hidden_layers": 16,
      "pad_token_id": 0,
      "scale_embedding": true,
      "static_position_embeddings": true,
      "transformers_version": "4.18.0",
      "use_cache": true,
      "vocab_size": 96103
    }

It download Files from huggingface.com after every update at the server sided.. infact it download several GBs of files .
There are ways to Download all files locally when you run the script for the first time and after that load all files locally.

 

[RowdySadjit]

It download Files from huggingface.com after every update at the server sided.. infact it download several GBs of files .
There are ways to Download all files locally when you run the script for the first time and after that load all files locally.

I thought that is what it is doing as standard. I also just noticed at the output code I posted above and it says it is calling it from home folder's cache, rather than downloading.

 

[neOtron]

I thought that is what it is doing as standard. I also just noticed at the output code I posted above and it says it is calling it from home folder's cache, rather than downloading.

I guess i failed to explain properly.
The first time you run the script . it will download a lot of files from huggigface.co and save it to your .cache directory. and then use the files from there
But. after the cache period expires it will again call huggigface.co and download all the files again.
i m not sure what the cache time limit i never checked . all i know is every few days it download file again and again.
so i looked for an alternative and instead of saving those files in .cache directory i save it to another specific folder and use it from there. otherwise downloading those files again and again fill up my server.

model_name="tuner007/pegasus_paraphrase"
model_location=os.getcwd()+'/models/'+model_name
token_location=os.getcwd()+'/tokens/'+model_name

# tokenizer = AutoTokenizer.from_pretrained(model_name)
# tokenizer.save_pretrained(token_location)
# model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# model.save_pretrained(model_location)

tokenizer = AutoTokenizer.from_pretrained(token_location)
model = model = AutoModelForSeq2SeqLM.from_pretrained(model_location).to(torch_device)

 

[RowdySadjit]

I guess i failed to explain properly.
The first time you run the script . it will download a lot of files from huggigface.co and save it to your .cache directory. and then use the files from there
But. after the cache period expires it will again call huggigface.co and download all the files again.
i m not sure what the cache time limit i never checked . all i know is every few days it download file again and again.
so i looked for an alternative and instead of saving those files in .cache directory i save it to another specific folder and use it from there. otherwise downloading those files again and again fill up my server.

model_name="tuner007/pegasus_paraphrase"
model_location=os.getcwd()+'/models/'+model_name
token_location=os.getcwd()+'/tokens/'+model_name

# tokenizer = AutoTokenizer.from_pretrained(model_name)
# tokenizer.save_pretrained(token_location)
# model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# model.save_pretrained(model_location)

tokenizer = AutoTokenizer.from_pretrained(token_location)
model = model = AutoModelForSeq2SeqLM.from_pretrained(model_location).to(torch_device)

few DAYS? wtf lol. What is the purpose of that? I had it several days and haven't noticed that yet. Do you know how to suppress the mass of output I mentioned above when it runs? It spams the terminal bigtime. Is there a method to run it quiet or do you have to send to /dev/null?

 

[Sartre]

few DAYS? wtf lol. What is the purpose of that? I had it several days and haven't noticed that yet. Do you know how to suppress the mass of output I mentioned above when it runs? It spams the terminal bigtime. Is there a method to run it quiet or do you have to send to /dev/null?

I tried downloading the huggingface model locally and it works without any calls home.

 

[RowdySadjit]

I tried downloading the huggingface model locally and it works without any calls home.

Do you know how to suppress all that output listed above that pegasus spews out or should I just /dev/null it?

I am amazed how quickly I am getting things stitched together myself now. I thought I would really struggle given the seeming complexity of the posts laid out however I now have nearly the whole bare bones article process up and running now; still only on minor sentence level tests and testing for most unique which I just figured out a few minutes ago. That is the main proof of concept now though and just a case of building out all the other parts around it.

 

[Alma]

Hey @Sartre, what is your cost per 1k scraped question + answer from PAA? Would like to compare it with my costs.

 

[bellamartinez]

All the best for the launch of those 3 sites. Seems interesting to me

 

[Sartre]

Hey @Sartre, what is your cost per 1k scraped question + answer from PAA? Would like to compare it with my costs.

pennies. electricity. + some negligible cost for proxies at this scale.