crawling

  1. BaldHat616

    Crawling issues

    One of my sites is relatively new, about 6 months old. I already build some links (not too much) and request indexing via search console for every new post. Still, there are about 50 posts which Google found in the site map, but did not crawl yet. Many of them are live for about a month...
  2. T

    Refresh your robots.txt knowledge

    Although robots.txt files seem like an easy to understand topic, in reality it's one of the more complex topics within Technical SEO. Why? For starters, because robots.txt files influence crawl budget, duplicate content and JavaScript rendering. What's more, different search engines interpret...
  3. Yez16

    What's wrong with Google Indexing API?

    I added my websites to Google Index with help Google Indexing API. But they stopped being added at about Oct. 21. I found the news about "Request Indexing Feature": https://www.techradar.com/news/google-killed-one-of-its-most-popular-seo-features-ahead-of-black-friday But it's function in Search...
  4. S

    My new posts are getting de-indexed after 24 hours

    Hi, I'm publishing on daily 2 posts. Google is taking 3-4 hours to index my new posts. After 24 hrs those posts are getting de-indexed. Has anyone faced the same issue like me? Thanks
  5. littlewebdragon

    ███►CRAWLING.WEBSITE◄███ Get Your Web Properties Crawled & Indexed ✅✅✅ FREE Crawling For Everyone!

    Give your website boost that it deserves! We know that you have spent a lot of effort, time and money into building various web properties to link to your money site. Don't let them go to waste! Google needs to crawl and acknowledge your backlinks before they are counted as backlinks towards...
  6. alexrochelle

    How to prevent crawling my website from alexa.com?

    How to prevent crawling my website from alexa.com?
  7. D

    Web Scraping Google Shopping | i need your help

    Im in need of a setup for web crawling BHW Community | I need your help...
  8. R

    Website Pages have got DeIndexed

    Hello to all. Recently since November 27 to be more specifically our website had heavy fluctuations within the SERPs. At first we thought it was just usual update fluctuations since they happen from time to time and do not affect the outcome that heavily. For the past week we have been...
  9. Isabella Taylor

    Help me in Link Building Crawling Issues

    Hello connections, I am creating link building for my SEO projects but Google is not crawling any of my Links. So what should I do in this case?
  10. Guybrushthepirate

    It is possible to know from which site the bot/crawler is coming when following a link?

    Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...
  11. P

    Excluded pages were down from 7 lakh to 41k but valid pages are still 17.4k? What does it shows?

    How does it impact the overall performance? If the pages are dropped from excluded, where did they go?
  12. showmaker

    Journey to Create Self Growing Websites

    I have started a journey to create 10 self growing websites. I bought a few good domains from auctions and plan to develop a website on each of those. I then plan to write scripts that can crawl the web and find unidexed/old/deleted content and feed that content to the created websites; thus a...
  13. bangun tidur

    (HELP) Google Search Result Keep Showing My Mobile Page Instead Of My Dekstop Version why?

    hello all i have issue here when ever google bot especially google bot smartphone indexing my site it keep changing my search result into my mobile version that have no seo feature because it only be used for my mobile visitor to download and app for my game instead of playing through browser...
  14. M

    How can block ahref bot to crwal my website?

    How can block ahref bot to crwal my website?
  15. A

    How to block backlink checking sites, like Majestic, Ahrefs and SEMRush

    Hi guys, I am building a small PBN for myslef. Can any one please guide me step by step how to block backlink checking sites, like Majestic, Ahrefs and SEMRush, from crawling or indexing my PBN?
  16. Jbaco

    NewsMedia site: 304 vs QDF

    Hi there! So this is a post looking for experiences and thoughts about QDF vs 304. I'm getting you into context. I work as an SEO (a rookie one) in a newsmedia site quite huge 60k+ url. So I'm struggling to get crawled and indexed the super-fresh news we launch by the time we launch it, not 3...
  17. Sunil Thakur

    Website Crowling issue !!

    Hello there, I have a website, it's not cache by google. I will tried all methods but it still same. Please guys help me.
  18. greytophat

    Sources of raw Instagram data on a large scale?

    I'm looking for sources of raw (i.e. JSON or CSV) Instagram data. In particular looking for (a single place doesn't have to provide all, just any one of these): list of the usernames of all of an account's followers (or at least a sample of several thousand) lists of usernames that have liked...
  19. Peter Manson

    How crawler crawl a web page?

    A question pings me many times If a website disallow bot for specific directory like ( Disallow: /user/* ) but I crawl a URL ( /user/someone) through submit tool so what will happen. Will this page crawl or not? I forgot to tell that I am getting the do-follow links here. Will I get the link...
  20. Link Player

    Google in not Crawling my links after 301 Redirects

    Hi Experts, I have a small HTML website with a wordpress blog. My website domain was redirected to www from non www version and also indexed with www except blog page and blog posts. I just set 301 redirects on my Wordpress blog and all its post pages and trying to index these pages again...
Top