Dr Webinstein
Newbie
- Jul 19, 2016
- 3
- 2
Hey all long time reader first time poster, was hoping to reach out for some advice here, and sorry if this has been indexed and answered a million times over !
Basically is Majestic worth getting indexed and crawled with regularly, and their whole trustflow citation, before I begin, and second
for the life of me no matter what I do I cannot get verified as the response code is forbidden that they cannot access robots.txt I have tried numerous configurations of Allowing user agent , Majestic, NJ12bot you name it, no success and have spent a good hour scrying the web for answers.
I doubt it is my ISP , as I do not have this problem with any other service in any way,
any help or advice much appreciated thank you
Basically is Majestic worth getting indexed and crawled with regularly, and their whole trustflow citation, before I begin, and second
for the life of me no matter what I do I cannot get verified as the response code is forbidden that they cannot access robots.txt I have tried numerous configurations of Allowing user agent , Majestic, NJ12bot you name it, no success and have spent a good hour scrying the web for answers.
I doubt it is my ISP , as I do not have this problem with any other service in any way,
any help or advice much appreciated thank you