1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Can Google Crawl Other Search Engines?

Discussion in 'BlackHat Lounge' started by rannybo, Apr 20, 2009.

  1. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54
    Hi,

    I was just wondering and maybe someone on this forum can help. Can Google crawl other search engines like Yahoo and MSN? I believe they can but do they actually do it?
     
  2. aмillionaírе

    aмillionaírе Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 20, 2008
    Messages:
    532
    Likes Received:
    358
    They never said anything about it, but I bet they probably have something on the backend that does this.
     
  3. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    There would be no purpose for Google to do this, but I can almost guarantee that both Yahoo and MSN do it to Google.
     
  4. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54
    I just checked their Googles's robots.txt file and they definitely don't let other search engines to access their search results.
     
  5. pyronaut

    pyronaut Executive VIP

    Joined:
    Dec 9, 2008
    Messages:
    1,229
    Likes Received:
    1,422
    Maybe not the MSN and Yahoo. But they do index other search results of pages from other misc web searches.
     
  6. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    Well, that assumes that all bots follow the rules dictated in robots.txt, which is simply not the case.
     
  7. rannybo

    rannybo Junior Member

    Joined:
    Jul 7, 2007
    Messages:
    175
    Likes Received:
    54
    Makes sense. Thanks!
     
  8. entreprenuer

    entreprenuer Regular Member

    Joined:
    Apr 19, 2009
    Messages:
    200
    Likes Received:
    73
    Location:
    Orlando Fl
    In my opinion google wouldn't need to crawl other search engines because they seem to be the quickest at finding sites in most cases anyways.
     
  9. SawUrMomOn/b/

    SawUrMomOn/b/ Newbie

    Joined:
    Apr 20, 2009
    Messages:
    0
    Likes Received:
    0
    No but I wouldn't rule it out. Any thoughts on Google's relative monopoly?
     
  10. ukescuba

    ukescuba Jr. VIP Jr. VIP Premium Member

    Joined:
    Feb 24, 2008
    Messages:
    994
    Likes Received:
    634
    Occupation:
    Mobile Marketer & QR Code Junkie
    Location:
    San Antonio, TX
    Home Page:
    they will probably crawl the static pages of se's but not the actual serps as the spiders have no way to post a search :)

    however most search engines pull data from the same resource pool anyway
     
  11. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    Why don't you look? Put this in Google:

    Code:
    site:http://search.live.com/results
    You will see yes, they have indexed results yet it's blocked by robots.txt and shouldn't be crawled. As for Google scraping other engines results for internal reasons like quality comparisons etc of course they do.

    Yeah, Google fills in search forms and then crawls/indexes the results pages. They have done that for quite some time. I had thousands of unrelated search pages indexed on a test blog i setup due to Google inputting search terms.
     
  12. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    There is a difference between Google actively performing searches on MSN or Yahoo with the intentions of using their indices as sources and Google following links from 3rd party sites to MSN/Yahoo searches and indexing them. Why would they perform quality comparisons on their own index with inferior search engines? You are definitely correct that the spiders perform searches on site search engines, but in terms of using Yahoo or MSN, there is absolutely no proof that this occurs as an active part of Google's indexing activities.
     
  13. tibetangojica

    tibetangojica Newbie

    Joined:
    Apr 19, 2009
    Messages:
    14
    Likes Received:
    0
    Location:
    toronto
    no,google does not
     
  14. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    But why are they inferior? The page layout, usability, colors, branding or the results themselves?

    I've seen studies where the Yahoo results were placed in a Google page, and vice-versa, and the vast majority said Google results were the best when it was actually Yahoo data which shows how much their branding effects perception. They also do blind comparisons (no branding), track result satisfaction over time, and 101 other things. I had a friend who got paid to go participate in one of their quality evaluations, there used to be a signup form on the web for it.

    They have a million reasons to track their competitors search index state.
     
  15. surajprakash31

    surajprakash31 Regular Member

    Joined:
    Oct 7, 2008
    Messages:
    261
    Likes Received:
    459
    Home Page:
    uhh...honestly, I don't get ur question....how will Google crawl a search engine!? You mean entering a keyword and then crawling the results?...thts really not a thing, Google would want to do. Big G as is has the most sites indexed and has the most accurate reults!
    Posted via Mobile Device
     
  16. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    After posting i read oldens post again, i'm sure he's referring to Google scraping other engines results and using that data in Google's own index. In that case, yes i totally agree Google has no reason to do that.

    If certain queries on Google are bad, they make adjustments until it comes good or in the case of one or two spam results manually give them the flick.

    But for internal comparative testing, then yes no doubt and i wouldn't be surprised if Google had a whole server powered with scraped mini indexes from other engines to test against.