robots.txt

https://openrobotstxt.org/ Just someone explain why there is a Neil Patel bot? lol

My website sitemap is not getting indexed and it has been two months even my robots txt file is also right i publish 100 % unique article and posts also not getting index so now what i do can any one explain me

I want to Block all URL With Numeric Here is one of the url is - monkeynut449009.html How can i block a 5000+ url like this

Is it ok to exclude plugins directory from robot.txt in wordpress site ? does it affect anything ?

Hi, anyone who can give me block everything except google bot robots.txt file ?

Hey guys. Short story here. Someone was asking me about this in a PM. This is my robots.txt file that I use for my white hat sites to block popular SEO tools like Ahrefs, Semrush, and also Archive.org, etc: User-agent: 01h4x.com Disallow:/ User-agent: 360Spider Disallow:/ User-agent...

Hello Guys. I need to know something about robots.txt file. please help. I have converted my website images .jpg to .webp But it did not work, so I deleted that plugin which converted .jpg to .webp Now google bots are crawling the URL /wp-content/uploads/2017/720x400.jpg.webp...

Although robots.txt files seem like an easy to understand topic, in reality it's one of the more complex topics within Technical SEO. Why? For starters, because robots.txt files influence crawl budget, duplicate content and JavaScript rendering. What's more, different search engines interpret...

Hey, I'm looking to learn more about the technical side of SEO in regards to coding, I'm trying to learn how to access the htaccess files and what to edit for SEO, add X-Robots-Tag for mockup sites, and everything in between :) It seems like not many tutorials online in this respect. Cheers

Hello, Today i received an email from Google Search Console saying that i have "Coverage issues detected" The warning is: "Indexed, though blocked by robots.txt" When i investigated the problem, i found out that the issue comes from blocking the "affiliate cloaking links", i use /visit/ for...

Hello, My site stopped getting traffic for a couple of days now, and i was wondering if there was anything wrong with my updated robots.txt file. I'm counting that i can ask for your expertise to check if there is anything wrong with my file: User-agent: Googlebot Allow: / User-Agent: *...

Hello everybody ! I have a question about the robot.txt : i would like to block some tracking urls like /?utm or /?gclid because these Urls impact my crawl budget. is it ok if i block all of these URLs ? will it impact the comportment of ad bot or other bots ? Thank you !

guys - i was reading this https://yoast.com/wordpress-robots-txt-example/ and have looked at their robots file: https://yoast.com/robots.txt, which has 1 line: User-Agent: * so basically, suggestion is not to put anything in robots.txt. what works for you guys, specially if you have...

when I fetch my website's robots.txt file I receive this error "Google couldn't crawl your site because we were unable to access the robots.txt file" please help me out

Hi community! I have a question regarding PBN's and hiding them from Majestic (MJ12bot) bot. I have a PBN, actually a lot of PBN's. I was hiding them using robots.txt method known by everyone. But then in one sunny day Majestic started to ignore my robots.txt and now others can see my PBN sites...

A nr. of URLs are being indexed that I don't want to... But since it is impossible to edit the robots.txt file shopify says in this guide to hide a url you should put the following code in the header of your template: {% if handle contains 'page-handle-you-want-to-exclude' %} <meta...

Hi, i have a site a year old..i added content but left it like that, but i also checked the "Discourage search engines from indexing this site" on my wordpress platform. anyhow, now started using the site so iv unchecked that..but when i submitted my sitemap again, its showing many errors..the...

Hi everyone, We recently saw a drop in our overall search result rankings so we checked our Google search console and it looks like at the same time as the SERP drop there was a boost in the number of pages being indexed on our site + a boost in the pages being blocked by robots. Please...

Hi guys, so a friend of mine plans to create a copy-niche site so to say, a niche site with rewritten content, which is ranking parallel to his other one. Reason is, that he got a penalty on an important money-keyword and doesn't know if it will come back - So he wants to rank now a rewritten...

I want to allow every thing but 3 pages in robots.txt is it correct to do it like this : User-agent: * Allow: Disallow: /p/terms.html Disallow: /p/privacy.html Disallow: /p/contact.html Sitemap: http://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500 i google it and that's...