Scritty
Elite Member
- May 1, 2010
- 3,019
- 4,707
And why are many of the interim states such a frikking mess... (with link to animation which shows why it looks so bad while in process)
Here goes my explanation. Nothing below is "fact" I'm not claiming it to be. It is summization based on years of watching the process and listening to what Google say and other brainier people who "know this stuff" have worked out.
-----------------------------------------------------------------------
Been asked a couple of times on my forum, PM's and emails about why I think Googles sort process is iterative.
What IS iterative sort? and.. Why don't Google just "set the order" in one go
Here's what an iterative sort and my best shot at explaining why something like this is the ONLY effective way of doing it... and also a link to an animation which shows how and why some of the interim states that these methods cause are, for some data nodes, temporarily such a mess and so seemingly out of place.
Penguin is links based, but there are many criteria in links. Authority, relevance, age, anchor text, platform, context, follow attribution
So not only is there a sort - it is a multi criteria sort or a weighted criteria sort.
In other words Google has to assess each URL against each possible keyword on several or many different criterias.
Now it may give a URL this weighted result and do a one pass sort.
Or it may do the most heavily influential criteria first then refine a URL's position within a given scope with iterations of sort based on increasingly less important criteria.
Either way - the sort process itself is long winded and NOT instant fix.
The likely process used is something along the lines of a partition exhange sort (Quick sort) I say this because large chunks are moved out and back in. We know (for instance) that Google treats the top 10 results as different (one "heap" or partition) and also the rest of the top 20. This lends itself to doing a heap sort.
Without access to a carbon copy static version of the internet to do a "total data set swap" with. Google HAS to do an iterative sort of some kind. \Here's some of the reasons why this is so:
a) The internet is not static. New URLS' are added at a rate of tens of thousands per minute, Others are changed. It's not possible to have a carbon copy of the internet to "practice" on then replace your sorted version for the real one with a push button
b) The internet is dispersed as is Google's view of it in its directory listings. It isn't all in one place making the idea of a worldwide "single swap" at the least a logistical nightmare, but more likely just "not possible"
c) The internet is FUCKING HUGE. Googles directory listings are also FUCKING HUGE
d) Google are not allowed to "stop" the internet, they are not allowed to "download it on to their own servers for analysis and sorting" (much of it is copyright - and of course the minute it's downloaded...it's obsolete)
Neither the internet or their index is fixed, can be fixed or would work if it was fixed. fixed.
"Fixed" and "Static" are not states that the internet and Googles' view of it ever are... or ever COULD BE.
A push button replacement of an entire data set that actually works and was worth doing 10 seconds later is just not possible.
So they sort - and they have to sort the live data "on the fly" - a lot of it - trillions of URL's.
Sit down - take a deep breath. Consider that there is no choice other than an iterative sort that is realistically possible given the constraints I've outlined above.
Now I don't know what method they use or what the real criteria are but "partition exchange sort" seems most likely.
http://en.wikipedia.org/wiki/Quicksort
It won't be exactly this. It won't be exactly anything you can find ona WIKI page. It will be a Google copyright method BASED on the sound logic of a standard sort process, and likely altered all the time itself as required
If you look at the animation on this WIKI page you can see just how messed up some of the interim stages are. How outliers and "sore thumb" results stick out for many passes.
Similar anomolies occur with most effective sorting methods (Bubble, Merge...etc)
The take away?
Sit back.. Chillax. If anyone tries to sell you a Penguin 2.1 beater in the next 4-5 days they are talking outta their ASS. No-one outside Google's head office will know what P2.1 really is, and even many of those inside won't understand it. In fact I doubt any one person has the IQ capacity to know all the possible ramifications. The algo is too complex and the data set to big and the situation too fluid.
Scritty
Here goes my explanation. Nothing below is "fact" I'm not claiming it to be. It is summization based on years of watching the process and listening to what Google say and other brainier people who "know this stuff" have worked out.
-----------------------------------------------------------------------
Been asked a couple of times on my forum, PM's and emails about why I think Googles sort process is iterative.
What IS iterative sort? and.. Why don't Google just "set the order" in one go
Here's what an iterative sort and my best shot at explaining why something like this is the ONLY effective way of doing it... and also a link to an animation which shows how and why some of the interim states that these methods cause are, for some data nodes, temporarily such a mess and so seemingly out of place.
Penguin is links based, but there are many criteria in links. Authority, relevance, age, anchor text, platform, context, follow attribution
So not only is there a sort - it is a multi criteria sort or a weighted criteria sort.
In other words Google has to assess each URL against each possible keyword on several or many different criterias.
Now it may give a URL this weighted result and do a one pass sort.
Or it may do the most heavily influential criteria first then refine a URL's position within a given scope with iterations of sort based on increasingly less important criteria.
Either way - the sort process itself is long winded and NOT instant fix.
The likely process used is something along the lines of a partition exhange sort (Quick sort) I say this because large chunks are moved out and back in. We know (for instance) that Google treats the top 10 results as different (one "heap" or partition) and also the rest of the top 20. This lends itself to doing a heap sort.
Without access to a carbon copy static version of the internet to do a "total data set swap" with. Google HAS to do an iterative sort of some kind. \Here's some of the reasons why this is so:
a) The internet is not static. New URLS' are added at a rate of tens of thousands per minute, Others are changed. It's not possible to have a carbon copy of the internet to "practice" on then replace your sorted version for the real one with a push button
b) The internet is dispersed as is Google's view of it in its directory listings. It isn't all in one place making the idea of a worldwide "single swap" at the least a logistical nightmare, but more likely just "not possible"
c) The internet is FUCKING HUGE. Googles directory listings are also FUCKING HUGE
d) Google are not allowed to "stop" the internet, they are not allowed to "download it on to their own servers for analysis and sorting" (much of it is copyright - and of course the minute it's downloaded...it's obsolete)
Neither the internet or their index is fixed, can be fixed or would work if it was fixed. fixed.
"Fixed" and "Static" are not states that the internet and Googles' view of it ever are... or ever COULD BE.
A push button replacement of an entire data set that actually works and was worth doing 10 seconds later is just not possible.
So they sort - and they have to sort the live data "on the fly" - a lot of it - trillions of URL's.
- Set the criteria
- Weight it
- Start the sorting process off
Sit down - take a deep breath. Consider that there is no choice other than an iterative sort that is realistically possible given the constraints I've outlined above.
Now I don't know what method they use or what the real criteria are but "partition exchange sort" seems most likely.
http://en.wikipedia.org/wiki/Quicksort
It won't be exactly this. It won't be exactly anything you can find ona WIKI page. It will be a Google copyright method BASED on the sound logic of a standard sort process, and likely altered all the time itself as required
If you look at the animation on this WIKI page you can see just how messed up some of the interim stages are. How outliers and "sore thumb" results stick out for many passes.
Similar anomolies occur with most effective sorting methods (Bubble, Merge...etc)
The take away?
Sit back.. Chillax. If anyone tries to sell you a Penguin 2.1 beater in the next 4-5 days they are talking outta their ASS. No-one outside Google's head office will know what P2.1 really is, and even many of those inside won't understand it. In fact I doubt any one person has the IQ capacity to know all the possible ramifications. The algo is too complex and the data set to big and the situation too fluid.
Scritty
without any major seo work.
