Alexion
Banned - Shitlisted and did not do the right thing
- Sep 9, 2021
- 1,456
- 3,788
Ok, sorry about that clickbait title, a small weekend joke.
But seriously, if you are using Indexing API to index stuff in volume(talking about spam blogs, auto blogs, PAA, etc) I just noticed something.
This only applies for people looking to index thousands of pages a day.
Out of laziness, I was submitting 100 links in one go(since it's the max per load) all the time.
I noticed today that if you send 100 links per request, around 40% get lost, or ignored.
I tracked 10 APIs, so that's about 2,000 pages sent to the API.
Around 800 got lost, the crawler never bothered to show up.
Now, I did some changes.
Created another google profile, created 10 GCP projects, and redid the operation.
This time, I sent only 10 pages / request.
Result?
100% links visited by a crawler. Like clockwork.
The problem isn't really with Google, but with the rank math script. When it doesn't get a reply, it resends the request.
This also wastes the number of requests from your daily quota.
So yeah, thought it's worth mentioning.
Trying to spread some karma in the universe, so good things come back to me, as I need a new on grid invertor.
But seriously, if you are using Indexing API to index stuff in volume(talking about spam blogs, auto blogs, PAA, etc) I just noticed something.
This only applies for people looking to index thousands of pages a day.
Out of laziness, I was submitting 100 links in one go(since it's the max per load) all the time.
I noticed today that if you send 100 links per request, around 40% get lost, or ignored.
I tracked 10 APIs, so that's about 2,000 pages sent to the API.
Around 800 got lost, the crawler never bothered to show up.
Now, I did some changes.
Created another google profile, created 10 GCP projects, and redid the operation.
This time, I sent only 10 pages / request.
Result?
100% links visited by a crawler. Like clockwork.
The problem isn't really with Google, but with the rank math script. When it doesn't get a reply, it resends the request.
This also wastes the number of requests from your daily quota.
So yeah, thought it's worth mentioning.
Trying to spread some karma in the universe, so good things come back to me, as I need a new on grid invertor.