1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

New feature from Twitter?

Discussion in 'White Hat SEO' started by lancis, Jul 3, 2013.

  1. lancis

    lancis Elite Member

    Joined:
    Jul 31, 2010
    Messages:
    1,632
    Likes Received:
    2,384
    Occupation:
    Entrepreneur
    Location:
    Milky Way
    Home Page:
    Just checked the robots.txt of their shortener:

    Code:
    User-agent: *
    Disallow: /
    
    Which means they have blocked proper indexing of shortened URLs.
    Google still indexes something, but if you search for "site:t.co" and limit the results to past 24 hours, you'll get these types of results:

    Moreover, they have been pretty active with their robots.txt on twitter.com lately. This is how it looks like now:

    Code:
    #Google Search Engine Robot
    User-agent: Googlebot
    Allow: /?_escaped_fragment_
    
    Allow: /search?q=%23
    Disallow: /search/realtime
    Disallow: /search/users
    Disallow: /search/*/grid
    
    Disallow: /*?
    Disallow: /*/followers
    Disallow: /*/following
    
    Meaning you're no longer getting indexed via followers & following page.
    FYI