- Jun 30, 2014
- 13,772
- 5,228
I've been examining a website that is only 4 months old, has a weak backlink profile but has 6 million pages indexed and is getting tons of traffic...
So Google obviously is naturally crawling all those pages. I've had mass page sites before on scalable cloud infrastructure and new sites don't have the crawl budget to get all that indexed so quick.
All I can think of is he's got tons of throw away domains and he's 301'ing the Google bot from those domains to his website and he's triggering the requests with the indexing API. In a similar way to how omega works.
Who knows their indexing shit and can throw some ideas in?
So Google obviously is naturally crawling all those pages. I've had mass page sites before on scalable cloud infrastructure and new sites don't have the crawl budget to get all that indexed so quick.
All I can think of is he's got tons of throw away domains and he's 301'ing the Google bot from those domains to his website and he's triggering the requests with the indexing API. In a similar way to how omega works.
Who knows their indexing shit and can throw some ideas in?