FYI: Googlebot and No Follow Links

[ipopbb]

This isn't new information, but it is recent evidence that re-confirms older understandings and for myself it revives some old ideas to look at again.

The history:

I set up an in-house click logging system for some traffic generation projects I have going on. I decided to manually submit one of the sites to the free service at http://www.submitexpress.com/free-submission.html (which has been around for about a decade) to see if it works still and how it performs in terms of time to visit from Googlebot (since Google removed it's anonymous add url page and now requires a webmaster login).

Googlebot started crawling the site 17 hours later.

My click logging system shows that Googlebot does visit external "No Follow" links "As it crawls" and in "Top Down and Left to Right" document parsing order.

So What?

IF Google considers "link order on page" in the quality of a link (which historically and currently I do not believe, but who knows what has changed recently) then this is likely how and where the ordinal is being set. It also alludes to how to organize links of value in your "parsable" HTML source (even though everybody knew this already... this at least puts one more blip of plausibility behind the idea).

A "No Follow" link is worth Googlebot's time and effort to crawl, so there is some kind of perceived value in them because crawling them in serial order as a site is being crawled is very inefficient and in terms of the "whole internet" very expensive. (or maybe this is a bug that explains why SERP rankings seem to change so much slower now)

I'd like to hear your thoughts on this.

Cheers,

Ted

 

[nikao]

This isn't new information, but it is recent evidence that re-confirms older understandings and for myself it revives some old ideas to look at again.

The history:

I set up an in-house click logging system for some traffic generation projects I have going on. I decided to manually submit one of the sites to the free service at (which has been around for about a decade) to see if it works still and how it performs in terms of time to visit from Googlebot (since Google removed it's anonymous add url page and now requires a webmaster login).

Googlebot started crawling the site 17 hours later.

My click logging system shows that Googlebot does visit external "No Follow" links "As it crawls" and in "Top Down and Left to Right" document parsing order.

So What?

IF Google considers "link order on page" in the quality of a link (which historically and currently I do not believe, but who knows what has changed recently) then this is likely how and where the ordinal is being set. It also alludes to how to organize links of value in your "parsable" HTML source (even though everybody knew this already... this at least puts one more blip of plausibility behind the idea).

A "No Follow" link is worth Googlebot's time and effort to crawl, so there is some kind of perceived value in them because crawling them in serial order as a site is being crawled is very inefficient and in terms of the "whole internet" very expensive. (or maybe this is a bug that explains why SERP rankings seem to change so much slower now)

I'd like to hear your thoughts on this.

Cheers,

Ted

I did a test for nofollow links a few years back. At that time I think it was google who was using it for indexing, but not for ranking. Yahoo was just ranking for it as well..
pretty sure things are different right now, but yeah.. if google is following the link it has SOME value clearly, if not just for a 'normal balance' in nofollow vs follow links? (like suggested to me in a different threat )

 

[RightInTwo]

From my experiences it still helps, it just doesn't pass any PageRank to your site.

 

[wowhaxor]

NoFollow certainly helps, I have one tool that runs 100% nofollow links and I see SERP increases on newer sites after just 1 blast. Its only one of many factors IMO.

 

[ipopbb]

UPDATE:


42 hours after website submission

25 hours after Googlebot's first visit

Progress towards a complete crawl: 3293 / 3650 or 90%

Only 15 Visits in the last hour

Only 14 Visits in the hour before last

Only 17 Visits in the hour before that

Only 15 Visits in the hour before that

looks like I'm being rate limited by Googlebot. Its all static content so not many options for performance tuning. I wonder if factors other than load times influence crawl rate. The average is 132 Visits per hour, so the crawl started fast and then slowed down. Perhaps googlebot goes more parallel with submission activity resulting in slower visits per hour. That seems odd to me. I would have thought they would use a FIFO (first in first out) queue for submissions OR something loosely based on one.

89% drop in crawl rate seems extreme. I wonder if google bot sees spikes where it gets 89% busier. That doesn't seem likely to me, but I honestly don't know at that scale.

 

[BluRoze]

According to google webmaster help, as of July 2011:

"In general, we don't follow [rel=nofollow links]. This means that Google does not transfer PageRank or anchor text across these links. Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap. Also, it's important to note that other search engines may handle nofollow in slightly different ways."

Google's spiders will crawl the links (apparently, they used to not even crawl them), so your findings are correct. But, nofollow links provide no 'link juice' whatsoever!

 

[Fwiffo]

a few thoughts:

1) have heard it said that google has enough horsepower to keep track of the web in almost real time, however that would be too much of a load on everyone else's servers (from some google video, forget which) - therefore I don't think that overall horsepower use is a top priority at the moment, if anything they may be orienting so that everyone else's horsepower use is reduced. Are you noticing any crawl difference between different times of day?

2) I'm currently operating under a slightly non-mainstream theory of "no follow" links

my understanding of mainstream theory: follow links pass linkjuice, nofollow links don't pass link juice

however, I think that google breaks down a possible search querry match under roughly the following questions:

1) how relevant is this page to the querry (ie accuracy of information on the page to the querry)
2) how "quality" is this page (ie quality as far as the visitor will percieve it - theme, overall reputation, etc.)
3) how "popular" is this page (ie backlinks / linkjuice)

personally, I think that link juice and a PR type (or similar unknown background system) affects question #3 and to a certain extent #2, but not necessarily #1

however, a no follow link, even while passing link juice to a page, would likely be a good indicator of question #1 - especially in long tail type searches.

Even if a nofollow link passes no linkjuice, theoretically the anchor/titile/surrounding text showing relevancy would still be an indicator (a nofollow link with anchor text saying "crappy stereo systems" is still an indicator that there's stereo systems at the other end of the link) - therefore likely of value for helping answer questions #1, especially in long tail searches

3) Im also operating under the theory that having a majority of follow links leaves a fairly identifiable pattern - ie a site with 1,000 follow links and no nofollow would likely be outside of standard deviation and worth a further look with the spam filter.

 

[goinviral]

Very interesting post Ipopbb. A while ago (2 maybe 3 years ago) a friend in IM told me that he considers no follow and do follow pretty well the same. I was like pffttt you know nothing. He felt abashed by my comment so we setup a test. We picked a relatively mildly competitive keyword and we setup a 10 page blog (unique content) and built 30 no follow links a day to it. I was waiting to be vindicated and win the bottle of scotch we bet on the project.

Long story short, That site ranked like a bitch and I had to fork out for a bottle of Johnny Green. Siteexplorer only showed the no follow links we have built and no others although months later the site got plenty of do follows as people started linking to it.

That got me thinking and made me change my linkbuilding philosophy. I now actively go after all types of links follow and nofollows. For one thing it looks more natural. However it also made me think about the future and I came up with this premise.

What if Google wins and eventually "ALL" webmasters are educated about nofollow links and "ALL" webmasters start using them. I mean really if you know about it you will probably use it right? Why let your precious linkjuice head off to some random just because you like their article? The link itself should be enough!

Now if we keep thinking about the future, that point in time, when Google finally wins the follow/nofollow battle and everyone is using nofollows. What links are they going to use for ranking the popularity of websites? Well, the NOFOLLOWS of course! of course they will have other ranking factors like social markers etc. However they will have to take notice of all those nofollow links "they used to ignore".

I believe they are approaching that point quicker than we may think and are already factoring no follows (as the little experiment above showed). On seo moz I believe they tried to audit nofollows vs follow links out there and it is about 50/50 now and nofollows are growing at a faster rate than the "********s".

So where am I going with all this? no where really. Just saying that the old adage "a link is a link". I personally rate HIGH PR do follows as the cream of the crop, "but only because PR is all we really have to go by". Really if you want to think about the best backlinks you can possibly get is by searching a niche and getting the top 10 sites to link to you, but we all know that will never happen!

However I will take any link I can get, I still believe there is no bad link out there. If there was a bad link we would all be spamming our competitors with these so called bad links! One of my competitors tried to spam one of my sites with porn and pharma links, Guess what? I am still pos 1! I wont actively persue these backlinks but if the competition wants hit me with them then thanks

 

[Greybeard]

Goinviral, maybe you weren't penalized for the bad links your competitor sent because you have a history of good links. Google's algorithm is more sophisticated than we know, and I'm sure it factors in possible spam in the form of bad links. However, if you were a newer site with few links, I'm sure those bad links would have affected you.

I go with the notion that a link is a link, but it is good to have higher pr, do follow links. A site with absolutely no "no follow" links does seem unnatural and would be on high alert.

 

[goinviral]

Goinviral, maybe you weren't penalized for the bad links your competitor sent because you have a history of good links. Google's algorithm is more sophisticated than we know, and I'm sure it factors in possible spam in the form of bad links. However, if you were a newer site with few links, I'm sure those bad links would have affected you.

I go with the notion that a link is a link, but it is good to have higher pr, do follow links. A site with absolutely no "no follow" links does seem unnatural and would be on high alert.

This could very well be true. Although I did buy a few fiverr gigs for a site recently and one of them commented on porn blogs and linked back to money site (not what I asked for) but the site is doing well so far! The gig didn't provide lots of backlinks but did target high pr blog posts, so maybe thats why it didn't cause to much trouble. I could imagine if you had a new sites and all of sudden had 10k plus porn backlinks it would be a red flag.

and yes I agree a link is a link. They are all good to have but not all links are the same.