crawling

Hello Everyone, I’m encountering a difficult issue with some filtered URLs on my site being indexed in Google. URL's Pattern - example.com/category-name/colour/Dark%20Grey example.com/category-name/trousers/size/9/10 example.com/category-name/gender/size/6/9...

I am not able to see fb page's details like followers details, posts and some more. When i scroll through contents without logging in to fb. it gives a popup which is not closable. How to bypass this? Please help.

If you don't want your content to be indexed by AI, you should create the ai.txt text file, which is similar to robots.txt. User-Agent: * Disallow: *.txt Disallow: *.pdf Disallow: *.doc Disallow: *.docx Disallow: *.odt Disallow: *.rtf Disallow: *.tex Disallow: *.wks Disallow: *.wpd Disallow...

One of my sites is relatively new, about 6 months old. I already build some links (not too much) and request indexing via search console for every new post. Still, there are about 50 posts which Google found in the site map, but did not crawl yet. Many of them are live for about a month...

Although robots.txt files seem like an easy to understand topic, in reality it's one of the more complex topics within Technical SEO. Why? For starters, because robots.txt files influence crawl budget, duplicate content and JavaScript rendering. What's more, different search engines interpret...

I added my websites to Google Index with help Google Indexing API. But they stopped being added at about Oct. 21. I found the news about "Request Indexing Feature": https://www.techradar.com/news/google-killed-one-of-its-most-popular-seo-features-ahead-of-black-friday But it's function in Search...

Hi, I'm publishing on daily 2 posts. Google is taking 3-4 hours to index my new posts. After 24 hrs those posts are getting de-indexed. Has anyone faced the same issue like me? Thanks

Give your website boost that it deserves! We know that you have spent a lot of effort, time and money into building various web properties to link to your money site. Don't let them go to waste! Google needs to crawl and acknowledge your backlinks before they are counted as backlinks towards...

How to prevent crawling my website from alexa.com?

Im in need of a setup for web crawling BHW Community | I need your help...

Hello to all. Recently since November 27 to be more specifically our website had heavy fluctuations within the SERPs. At first we thought it was just usual update fluctuations since they happen from time to time and do not affect the outcome that heavily. For the past week we have been...

Hello connections, I am creating link building for my SEO projects but Google is not crawling any of my Links. So what should I do in this case?

Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...

How does it impact the overall performance? If the pages are dropped from excluded, where did they go?

I have started a journey to create 10 self growing websites. I bought a few good domains from auctions and plan to develop a website on each of those. I then plan to write scripts that can crawl the web and find unidexed/old/deleted content and feed that content to the created websites; thus a...

hello all i have issue here when ever google bot especially google bot smartphone indexing my site it keep changing my search result into my mobile version that have no seo feature because it only be used for my mobile visitor to download and app for my game instead of playing through browser...

How can block ahref bot to crwal my website?

Hi guys, I am building a small PBN for myslef. Can any one please guide me step by step how to block backlink checking sites, like Majestic, Ahrefs and SEMRush, from crawling or indexing my PBN?

Hi there! So this is a post looking for experiences and thoughts about QDF vs 304. I'm getting you into context. I work as an SEO (a rookie one) in a newsmedia site quite huge 60k+ url. So I'm struggling to get crawled and indexed the super-fresh news we launch by the time we launch it, not 3...

Hello there, I have a website, it's not cache by google. I will tried all methods but it still same. Please guys help me.