What do you think about this robots.txt ?

maildigger

Senior Member
Joined
Jan 30, 2009
Messages
897
Reaction score
162
I'm having ranking issues with my Woocommerce.
Could it be robots.txt related?
Do you see a problem in my robots.txt file?


# Sitemap

Sitemap: https://url.com/sitemap_index.xml

# Global rules

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /feed/
Disallow: /*?add-to-cart=
Disallow: /?s=*
Disallow: /search/*
Disallow: /page/*

# Bots

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# AI

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

# Search

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /
 
references your actual domain and you've just redacted it here?
You've obviously got experience asking "Have you restarted the computer?" :D
 
Your robots file is very carefully customized, which shows that you have a strong technical understanding. The ranking issue probably isn’t related to your robots file.
 
Looks mostly fine to me. You're blocking the normal pages like cart and checkout, which is standard. Just make sure you're not blocking any important category or product pages by mistake. Also check that your sitemap is working. Usually ranking issues come from other areas, not robots.txt.
 
My ranking problems started when we switched to Cloudfare.
 
Back
Top