1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Majestic and forbidden robots.txt

Discussion in 'White Hat SEO' started by Dr Webinstein, Jul 19, 2016.

  1. Dr Webinstein

    Dr Webinstein Newbie

    Joined:
    Jul 19, 2016
    Messages:
    3
    Likes Received:
    2
    Gender:
    Male
    Location:
    Jotunheim
    Hey all long time reader first time poster, was hoping to reach out for some advice here, and sorry if this has been indexed and answered a million times over !

    Basically is Majestic worth getting indexed and crawled with regularly, and their whole trustflow citation, before I begin, and second

    for the life of me no matter what I do I cannot get verified as the response code is forbidden that they cannot access robots.txt I have tried numerous configurations of Allowing user agent , Majestic, NJ12bot you name it, no success and have spent a good hour scrying the web for answers.

    I doubt it is my ISP , as I do not have this problem with any other service in any way,

    any help or advice much appreciated thank you