Google indexing everything
- Thread starter Ai Samuel
- Start date
How do I remove it and stop Google from indexing it further?
Robots txt file google it bro.
Heres an example
# robots.txt for example.com
# Allow everything by default except sensitive areas
User-agent: *
Disallow: /admin/
Disallow: /login
Disallow: /search
Allow: /assets/
Allow: /public/
# Block a few aggressive crawlers completely
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
# Sitemap location
Sitemap: https://example.com/sitemap.xml
Heres an example
# robots.txt for example.com
# Allow everything by default except sensitive areas
User-agent: *
Disallow: /admin/
Disallow: /login
Disallow: /search
Allow: /assets/
Allow: /public/
# Block a few aggressive crawlers completely
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
# Sitemap location
Sitemap: https://example.com/sitemap.xml
LouisaLily
Newbie
- Jan 6, 2024
- 23
- 4
I encountered a very interesting thing, when my domain was about to expire, Google announced that my website had been indexed, that was the anniversary of the first time I made a website.
Lol you think they wanted your site? Google have been known for doing dodgy things in the past. If thats what you mean.
LouisaLily
Newbie
- Jan 6, 2024
- 23
- 4
I just post on entertainment blog but google makes me find it very interesting
