For the past 1-2 weeks (after their recently update), i have had countless "Page cannot be crawled or displayed due to robots.txt." error and it was sooooooo damn annoying. You have to refresh a few times to see the content. The solution i found is staying signed in and the error will be gone :| damn should have known this sooner. Am i the only one that encountered this issue?