1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt and 301 redirect

Discussion in 'Black Hat SEO' started by tony_d, Nov 28, 2013.

  1. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,164
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    Which do bots read first - the 301 redirect, or robots.txt?

    I use 301's alot, and it would be rather convenient if I could block Ahrefs et al from picking up the 301 at the source, by blocking them from the site. My #1 rankings with zero backlinks would then be a total mystery to my competitors :)
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  3. tmamedov

    tmamedov Junior Member

    Joined:
    Sep 20, 2012
    Messages:
    117
    Likes Received:
    17
    Google firstly goes to .httaccess, than robots.txt
     
  4. jhakasseo

    jhakasseo Senior Member

    Joined:
    Mar 1, 2012
    Messages:
    827
    Likes Received:
    237
    Occupation:
    Internet Marketing
    Location:
    On d Earth
    Home Page:
    Yeah Google first access robots.txt so if you want to block ahrefs then go ahead and blog them. Let me know also if you get success with it.
     
  5. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,581
    Likes Received:
    3,164
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    How about Ahrefs and Majestic bots - presumably they behave the same way, checking the robots.txt first? Anyone have insight on this?

    I don't actually want to block Google, that will destroy the 301 - I just want to block the tools that my competitors might use...
     
  6. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    LOL, noone is accessing your htaccess. It is automatically read by your Apache server when a request is made to your website..
     
  7. qrazy

    qrazy Senior Member

    Joined:
    Mar 19, 2012
    Messages:
    1,113
    Likes Received:
    1,712
    Location:
    Banana Republic
    Robots.txt first and then 301.

    Robot.txt defines the rule for the spider whereas 301 is the response(about the new url) sent from your server to Google(when it request a url).

    @tmamedov
    .htaccess is the configuration file of the Apache server, so it won't be accessed elsewhere.
     
    • Thanks Thanks x 1