googlebot

Surprised there hasn't been a thread about this here? Basically, Cloudflare is accusing Google of using its crawler for both search and AI simultaneously. Cloudflare is giving Google the ultimatum to split the two or get blocked. Started September 15, last week.

ive been using virustotal as a sort of indexer now and it worked quite well. since virustotal is owned by google they basically send a googlebot to check your site for malware. takes like 5 seconds as well. i used it for a new blogger recently and it indexed it in 24 hours...

Lately, I'm seeing google is not indexing my account's tweets (automatically) I've to index them manually. Anyone of you noticed that google is slowly indexing / not at all indexing tweets both automatically or manually?

Hi, Couple of years ago Google was following, crawling, indexing external links on pages having meta robtos tag: noindex, follow. example my PBN page: example.com/cars.html having meta robots tag noindex, follow, having an external link to my money site with anchor text Cars. But now it seems...

I need a WordPress theme or plugin own cloaking script for Adsense arbitrage

Hello, My site stopped getting traffic for a couple of days now, and i was wondering if there was anything wrong with my updated robots.txt file. I'm counting that i can ask for your expertise to check if there is anything wrong with my file: User-agent: Googlebot Allow: / User-Agent: *...

Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...

Hey guys, Question, my site gets an enourmous amounts of requests from Googlebot, and has been getting these for atleast 3 days straight. However when I search site:mysite.com or check the serps, I dont see ANY differences at all. I finally thought I was about to kick off. Does anyone else...

hello all i have issue here when ever google bot especially google bot smartphone indexing my site it keep changing my search result into my mobile version that have no seo feature because it only be used for my mobile visitor to download and app for my game instead of playing through browser...

We require a bot for a website traffic anybody have knowledge regarding this ,comment in box

Sorry for my bad english. The most effective and free method of indexing in Google is to add a site to Google Search Console:) But it happens that this is impossible, for various reasons. Here I will consider other ways. The purpose of the experiment: I wanted to find out how to get...

Hello there, I have a website, it's not cache by google. I will tried all methods but it still same. Please guys help me.

Hi guys, so a friend of mine plans to create a copy-niche site so to say, a niche site with rewritten content, which is ranking parallel to his other one. Reason is, that he got a penalty on an important money-keyword and doesn't know if it will come back - So he wants to rank now a rewritten...

He everyone, i have a site in which you can get nutritional value on foods (compare them with one another). Right now the URL is like that : https://compareyourfood.com/?Bananas=09040&Apples-with-skin=09003 I'm wondering if google crawler will be able to index this page differently from my...

Hi all I just want to warn everyone who block alot of bots from their site to make sure Googlebot not blocked today went to one of my sites and I notice last visit from Googlebot was from almost a week . I checked .htaccess and i found wrong code that blocking Googlebot ouch .. to prevent...

Can anyone tell what i need to do ? here is a message which I got domain dot com: Googlebot can't access your site Feb 8, 2014 Over the last 24 hours, Googlebot encountered 14 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we...

Hi, I have a website with stores in english and italian. When we change the store, the product URL changes - 1 URL for english product page and 1 URL for italian product page. English store is the default store. The switch works but when I go directly to the italian product URL I'm redirected...

I have been noticing a random url in the middle of my crawl stamps. [09/Jun/2012:08:41:56 -0400] "GET / HTTP/1.1" 200 9521 "http://randomurl.com/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" I have been seeing a lot of them, but not sure what it means. Does...

"Googlebot keeps getting smarter. Now has the ability to execute AJAX/JS to index some dynamic comments" https://twitter.com/#!/mattcutts/status/131425949597179904 http://searchengineland.com/google-can-now-execute-ajax-javascript-for-indexing-99518

This isn't new information, but it is recent evidence that re-confirms older understandings and for myself it revives some old ideas to look at again. The history: I set up an in-house click logging system for some traffic generation projects I have going on. I decided to manually submit...