1. TomTheCat


    If you don't want your content to be indexed by AI, you should create the ai.txt text file, which is similar to robots.txt. User-Agent: * Disallow: *.txt Disallow: *.pdf Disallow: *.doc Disallow: *.docx Disallow: *.odt Disallow: *.rtf Disallow: *.tex Disallow: *.wks Disallow: *.wpd Disallow...
  2. BaldHat616

    Crawling issues

    One of my sites is relatively new, about 6 months old. I already build some links (not too much) and request indexing via search console for every new post. Still, there are about 50 posts which Google found in the site map, but did not crawl yet. Many of them are live for about a month...
  3. T

    Refresh your robots.txt knowledge

    Although robots.txt files seem like an easy to understand topic, in reality it's one of the more complex topics within Technical SEO. Why? For starters, because robots.txt files influence crawl budget, duplicate content and JavaScript rendering. What's more, different search engines interpret...
  4. Yez16

    What's wrong with Google Indexing API?

    I added my websites to Google Index with help Google Indexing API. But they stopped being added at about Oct. 21. I found the news about "Request Indexing Feature": But it's function in Search...
  5. S

    My new posts are getting de-indexed after 24 hours

    Hi, I'm publishing on daily 2 posts. Google is taking 3-4 hours to index my new posts. After 24 hrs those posts are getting de-indexed. Has anyone faced the same issue like me? Thanks
  6. littlewebdragon

    ███►CRAWLING.WEBSITE◄███ Get Your Web Properties Crawled & Indexed ✅✅✅ FREE Crawling For Everyone!

    Give your website boost that it deserves! We know that you have spent a lot of effort, time and money into building various web properties to link to your money site. Don't let them go to waste! Google needs to crawl and acknowledge your backlinks before they are counted as backlinks towards...
  7. alexrochelle

    How to prevent crawling my website from

    How to prevent crawling my website from
  8. D

    Web Scraping Google Shopping | i need your help

    Im in need of a setup for web crawling BHW Community | I need your help...
  9. R

    Website Pages have got DeIndexed

    Hello to all. Recently since November 27 to be more specifically our website had heavy fluctuations within the SERPs. At first we thought it was just usual update fluctuations since they happen from time to time and do not affect the outcome that heavily. For the past week we have been...
  10. Isabella Taylor

    Help me in Link Building Crawling Issues

    Hello connections, I am creating link building for my SEO projects but Google is not crawling any of my Links. So what should I do in this case?
  11. Guybrushthepirate

    It is possible to know from which site the bot/crawler is coming when following a link?

    Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in there is a link pointing to Is there a chance for the owner of, to know that the...
  12. P

    Excluded pages were down from 7 lakh to 41k but valid pages are still 17.4k? What does it shows?

    How does it impact the overall performance? If the pages are dropped from excluded, where did they go?
  13. showmaker

    Journey to Create Self Growing Websites

    I have started a journey to create 10 self growing websites. I bought a few good domains from auctions and plan to develop a website on each of those. I then plan to write scripts that can crawl the web and find unidexed/old/deleted content and feed that content to the created websites; thus a...
  14. bangun tidur

    (HELP) Google Search Result Keep Showing My Mobile Page Instead Of My Dekstop Version why?

    hello all i have issue here when ever google bot especially google bot smartphone indexing my site it keep changing my search result into my mobile version that have no seo feature because it only be used for my mobile visitor to download and app for my game instead of playing through browser...
  15. M

    How can block ahref bot to crwal my website?

    How can block ahref bot to crwal my website?
  16. A

    How to block backlink checking sites, like Majestic, Ahrefs and SEMRush

    Hi guys, I am building a small PBN for myslef. Can any one please guide me step by step how to block backlink checking sites, like Majestic, Ahrefs and SEMRush, from crawling or indexing my PBN?
  17. Jbaco

    NewsMedia site: 304 vs QDF

    Hi there! So this is a post looking for experiences and thoughts about QDF vs 304. I'm getting you into context. I work as an SEO (a rookie one) in a newsmedia site quite huge 60k+ url. So I'm struggling to get crawled and indexed the super-fresh news we launch by the time we launch it, not 3...
  18. Sunil Thakur

    Website Crowling issue !!

    Hello there, I have a website, it's not cache by google. I will tried all methods but it still same. Please guys help me.
  19. ArmondSP

    ROTATING RESIDENTIAL PROXIES! Get Access to 40 million+ IPs and Be Ready to Go Above and Beyond With BHW-exclusive RECURRING DISCOUNT OFFER

    >> Click Here to Order Now << Our proxy pricing is based on traffic Want a special 15% BHW discount? What you have to do: simply like the thread and post "NEED SPECIAL BHW OFFER" in this thread. We will get back to you with a discount code via private message! Plans eligible to receive a...
  20. greytophat

    Sources of raw Instagram data on a large scale?

    I'm looking for sources of raw (i.e. JSON or CSV) Instagram data. In particular looking for (a single place doesn't have to provide all, just any one of these): list of the usernames of all of an account's followers (or at least a sample of several thousand) lists of usernames that have liked...
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock