1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

If you use PHP or HTML redirect, will crawlers still obey robots.txt

Discussion in 'Black Hat SEO' started by macdonjo3, Nov 20, 2015.

  1. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP

    Joined:
    Nov 8, 2009
    Messages:
    5,652
    Likes Received:
    4,379
    Location:
    Toronto
    Home Page:
    Kind of an advanced SEO question here. I don't know if anyone will know.

    If I use a PHP or HTML redirect, will crawlers still obey robots.txt rules?

    There are HTML redirects:

    Code:
    <META http-equiv="refresh" content="0;URL=http://www.indiana.edu/~account/new-directory">
    
    And there are PHP redirects:

    Code:
    sleep(5);
    header("Location: http://example.com/myOtherPage.php");
    die();
    
    I've read:

    If I set an interval of 0, which hints a 301, will Google still check robots.txt before checking that redirect?