robots.txt

  1. Alphaman77

    Sitemap Not Indexing "could Not Fetch" Error

    My website sitemap is not getting indexed and it has been two months even my robots txt file is also right i publish 100 % unique article and posts also not getting index so now what i do can any one explain me
  2. P

    Robots.txt Block All page with Numeric URL

    I want to Block all URL With Numeric Here is one of the url is - monkeynut449009.html How can i block a 5000+ url like this
  3. Sam1s

    plugin exclude from robots.txt

    Is it ok to exclude plugins directory from robot.txt in wordpress site ? does it affect anything ?
  4. S

    anyone who can give me block everything except google bot robots.txt file ?

    Hi, anyone who can give me block everything except google bot robots.txt file ?
  5. Sartre

    ☑️☑️☑️ My robots.txt file and how we hide advanced PBNs in 2022 from SEO tools and fly under the radar of the competition to make $$,$$$

    Hey guys. Short story here. Someone was asking me about this in a PM. This is my robots.txt file that I use for my white hat sites to block popular SEO tools like Ahrefs, Semrush, and also Archive.org, etc: User-agent: 01h4x.com Disallow:/ User-agent: 360Spider Disallow:/ User-agent...
  6. kuldeepsingh689

    Robots.txt File Question

    Hello Guys. I need to know something about robots.txt file. please help. I have converted my website images .jpg to .webp But it did not work, so I deleted that plugin which converted .jpg to .webp Now google bots are crawling the URL /wp-content/uploads/2017/720x400.jpg.webp...
  7. T

    Refresh your robots.txt knowledge

    Although robots.txt files seem like an easy to understand topic, in reality it's one of the more complex topics within Technical SEO. Why? For starters, because robots.txt files influence crawl budget, duplicate content and JavaScript rendering. What's more, different search engines interpret...
  8. C

    Coding For SEO - htaccess files, X-Robots-Tag, etc...

    Hey, I'm looking to learn more about the technical side of SEO in regards to coding, I'm trying to learn how to access the htaccess files and what to edit for SEO, add X-Robots-Tag for mockup sites, and everything in between :) It seems like not many tutorials online in this respect. Cheers
  9. kurosaki4d

    Warning from Google Search Console

    Hello, Today i received an email from Google Search Console saying that i have "Coverage issues detected" The warning is: "Indexed, though blocked by robots.txt" When i investigated the problem, i found out that the issue comes from blocking the "affiliate cloaking links", i use /visit/ for...
    • kurosaki4d
    • Thread
    • affiliate cloacking crawl disallow google robots.txt seo
    • Replies: 7
    • Forum: White Hat SEO
  10. kurosaki4d

    Is there anything wrong with my robots.txt

    Hello, My site stopped getting traffic for a couple of days now, and i was wondering if there was anything wrong with my updated robots.txt file. I'm counting that i can ask for your expertise to check if there is anything wrong with my file: User-agent: Googlebot Allow: / User-Agent: *...
  11. C

    Robots.txt and tracking URLs

    Hello everybody ! I have a question about the robot.txt : i would like to block some tracking urls like /?utm or /?gclid because these Urls impact my crawl budget. is it ok if i block all of these URLs ? will it impact the comportment of ad bot or other bots ? Thank you !
  12. fb3003

    robots.txt and wordpress

    guys - i was reading this https://yoast.com/wordpress-robots-txt-example/ and have looked at their robots file: https://yoast.com/robots.txt, which has 1 line: User-Agent: * so basically, suggestion is not to put anything in robots.txt. what works for you guys, specially if you have...
  13. T

    robots.txt fetch failed

    when I fetch my website's robots.txt file I receive this error "Google couldn't crawl your site because we were unable to access the robots.txt file" please help me out
    • tarunvij7
    • Thread
    • google on page on page seo robots.txt seo webmaster
    • Replies: 2
    • Forum: White Hat SEO
  14. Narix

    How to block Majestic bot?

    Hi community! I have a question regarding PBN's and hiding them from Majestic (MJ12bot) bot. I have a PBN, actually a lot of PBN's. I was hiding them using robots.txt method known by everyone. But then in one sunny day Majestic started to ignore my robots.txt and now others can see my PBN sites...
    • Narix
    • Thread
    • block robots majestic mj12 robots.txt
    • Replies: 25
    • Forum: Black Hat SEO
  15. LeftyLuke

    HELP! How can I hide a string of URLs in Shopify?

    A nr. of URLs are being indexed that I don't want to... But since it is impossible to edit the robots.txt file shopify says in this guide to hide a url you should put the following code in the header of your template: {% if handle contains 'page-handle-you-want-to-exclude' %} <meta...
  16. saaqib

    search console crawl errors

    Hi, i have a site a year old..i added content but left it like that, but i also checked the "Discourage search engines from indexing this site" on my wordpress platform. anyhow, now started using the site so iv unchecked that..but when i submitted my sitemap again, its showing many errors..the...
    • saaqib
    • Thread
    • robots.txt search console seo wordpress
    • Replies: 1
    • Forum: White Hat SEO
  17. L

    Surge in number of indexed pages on ecommerce store + a drop in SERPs

    Hi everyone, We recently saw a drop in our overall search result rankings so we checked our Google search console and it looks like at the same time as the SERP drop there was a boost in the number of pages being indexed on our site + a boost in the pages being blocked by robots. Please...
  18. kleinerwauwau

    Preventing Googlebot from reading the imprint

    Hi guys, so a friend of mine plans to create a copy-niche site so to say, a niche site with rewritten content, which is ranking parallel to his other one. Reason is, that he got a penalty on an important money-keyword and doesn't know if it will come back - So he wants to rank now a rewritten...
  19. googlebis

    Robots.txt Allow every thing but three pages blogger?

    I want to allow every thing but 3 pages in robots.txt is it correct to do it like this : User-agent: * Allow: Disallow: /p/terms.html Disallow: /p/privacy.html Disallow: /p/contact.html Sitemap: http://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500 i google it and that's...
  20. Werombi

    Blocking Bots - Which method is best in htaccess?

    Hi All, I have seen 2 methods of blocking bots through htaccess floating around and cannot see the operating difference. Can someone please explain the difference? If you know of what is best practice that be great too... Im using wordpress themes if this make any difference. First and...
    • Werombi
    • Thread
    • block badbots htaccess robots robots.txt
    • Replies: 0
    • Forum: White Hat SEO
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock