Reverse Engineering Google's Algos
- Thread starter Gophering
- Start date
Gophering
Junior Member
- Mar 21, 2013
- 115
- 281
I'm not sure why we're assuming that the Google search appliance is the same as the Google internal algo (even back in 2009). The basic functions such as crawling is probably very similar but I doubt that there's any additional parallels especially with the index. And when it comes to the index I think that there are two things that we're interested in: ranking and penalty factors.
There might be some ranking info bits in this appliance (we'll never be sure).
The penalty factors from what I understand are applied retroactively against the existing index based on some pattern or footprint, which means that it will not be part of the "index" otherwise the system will automatically "penalize" websites as it "indexes" them.
Anyways, I'm not trying to take away from the OP's work and there might be some value after all, it's just we will never know
Appreciate your input, thanks.
I never claimed this to be an exact copy of Big Brother. I mentioned that it "somewhat replicates Google's algos" which I think is a fair enough description. About the index, not sure what you mean exactly, but yeah there aren't too many ways to do proper weighted categorization. Have a look at elasticsearch or solr, pretty much the same concept. Obviously Google makes their money by being the best and constantly fine tuning their categorization algorithms.
The Python source shows one part of this process. Records are taken, weighted then stored. What was interesting to me is taking a glimpse at the weighting process (which most probably differs from the real thing). I've already mentioned that this won't be practical for SEO purposes. Firstly GSA doesn't have access to user metrics, which I guarantee Google proper takes heavy advantage of. Secondly, you are absolutely right, we'll never know. There might be 100s of pre-categorization as well as post-categorization processes involved, so we'll never know for sure.
However, it never hurts to dig around a little. We'll most probably never get the complete picture, but pull a lot of information together and most everything becomes a little predictable, don't you agree?
Riders On The Storm
Elite Member
- Feb 27, 2012
- 2,081
- 853
Great stuff mate.
Stufferizer
Regular Member
- Nov 6, 2012
- 284
- 96
Extremely nice approach to get to know what is under the hood of big g! People threw rep and thanks and I am glad to do the same!
I disagree marishal. In any kind of big long-lasting project, there is simply not the time to reinvent the wheel twice. I would be that the actual algorithm now is still around 60-70% of the actual ranking engine today. It's like DNA, we are over 90% equal to apes
I disagree marishal. In any kind of big long-lasting project, there is simply not the time to reinvent the wheel twice. I would be that the actual algorithm now is still around 60-70% of the actual ranking engine today. It's like DNA, we are over 90% equal to apes
bertbaby
Elite Member
- Apr 15, 2009
- 2,025
- 1,535
Based on the appliance page:
New with 7.0, we?ve tweaked the secret sauce to dramatically improve those results, and we?ve made it faster than ever. Go ahead, try it on your ?big data?.
I guess we want the secret sauce but nowadays the secret sauce for the Internet contains a lot of negative SEO penalties that would not be appropriate for a corporate appliance.
New with 7.0, we?ve tweaked the secret sauce to dramatically improve those results, and we?ve made it faster than ever. Go ahead, try it on your ?big data?.
I guess we want the secret sauce but nowadays the secret sauce for the Internet contains a lot of negative SEO penalties that would not be appropriate for a corporate appliance.
Gophering
Junior Member
- Mar 21, 2013
- 115
- 281
I think GSA itself was a bit different, as someone mentioned before you might be looking at the Google Mini. The GSA can be used to index your regular external pages (as I've done with BHW), effectively running your own indexes/crawlers. I've looked through some more Python sources today and parts of the algo do take certain negative metrics into account. Not sure if they are used or not, but they are there.
No clue about the Mini btw. I think it is severely limited compared to the GSA. I might be wrong though.
Gophering, if you're up to PMing me some of the Python source files, I'll be more than happy to help you analyse them here.
EDIT: Spoken to Gophering, and I'm going to collaborate with him here and help with the reverse engineering. Will post more information soon, so subscribe and keep watching this thread!
EDIT: Spoken to Gophering, and I'm going to collaborate with him here and help with the reverse engineering. Will post more information soon, so subscribe and keep watching this thread!
Last edited:
Gophering
Junior Member
- Mar 21, 2013
- 115
- 281
Wouldn't have the slightest idea how to monetize this tbh, apart from somebody hiring me to do some coding based on my skillset displayed here (although next to no coding was done). You got any suggestions?
Tony Barns
Regular Member
- Oct 6, 2009
- 264
- 191
This is the best thread I have read in a long time on here!!
danailo
Supreme Member
- Apr 1, 2011
- 1,209
- 778
I do not have any programmimg skills but I am thinking.
If outside man can buy this for a few k how much can he learn from it? Google wont sell their secrets for few millions.
Even if you have the algo you need to crawl and cache a Lot of data = many servers.
Anyway I will follow your thread.
Sent from my GT-N7000 using Tapatalk 2
If outside man can buy this for a few k how much can he learn from it? Google wont sell their secrets for few millions.
Even if you have the algo you need to crawl and cache a Lot of data = many servers.
Anyway I will follow your thread.
Sent from my GT-N7000 using Tapatalk 2
Gophering has been stressing that he's not sure where this thread is going, it may well be that there is very little usable information we can find out, but it'll all be a good learning path. Nobody is saying that we'll be able to find the 'magic formula' to insta-rank any website #1, because... we won't. But it will provide an interesting insight into some of the functionality of Google, and even if a lot has changed you can bet some aspects will be similar to the current Google set up.