1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HELP: Robots.txt showing 90% websitel Links as BLOCKED on Google Webmaster!

Discussion in 'Blogging' started by ran60616, Apr 24, 2015.

  1. ran60616

    ran60616 Regular Member

    Joined:
    Jan 17, 2013
    Messages:
    219
    Likes Received:
    296
    Location:
    A place like no other!
    Hey guys,

    Any help would be much appreciated.

    Here is how my robots.txt file looks like when anyone looks at it online.

    User-agent: *
    Disallow: /wp-admin/

    Also when I do a Google Search to find how many links are indexed, more than 1000+ shows up in the results. "Site:www.mysite.com" search string.

    However, on my Google Webmaster Tools it shows the following results under "Google Index">"Index Status".

    Total Indexed: 1062
    Blocked by robots: 913

    Any thing I should look at to solve this problem?

    Thank you!
     
  2. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    3,476
    Likes Received:
    3,103
    Gender:
    Male
    Try fetching the robots.txt again from Crawl as Google bot ( and press submit to index).
     
    • Thanks Thanks x 1
  3. ran60616

    ran60616 Regular Member

    Joined:
    Jan 17, 2013
    Messages:
    219
    Likes Received:
    296
    Location:
    A place like no other!
    Thank you for the suggestion. Will give it a try!
     
  4. angry cowboy

    angry cowboy Newbie

    Joined:
    Mar 31, 2015
    Messages:
    34
    Likes Received:
    1
    what does robot.txt file do?
     
  5. mysticDrac

    mysticDrac Newbie

    Joined:
    Jan 11, 2015
    Messages:
    7
    Likes Received:
    1
    it is first place where google bots land while crawl your site, they will see what can be index or not , after that sitemap
     
  6. Lermontov

    Lermontov Registered Member

    Joined:
    Nov 19, 2014
    Messages:
    97
    Likes Received:
    18
    Login to the webmaster tool to block the crawl, that will make Google remove them from search result faster.
     
  7. ran60616

    ran60616 Regular Member

    Joined:
    Jan 17, 2013
    Messages:
    219
    Likes Received:
    296
    Location:
    A place like no other!
    No luck so far!
     
  8. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    3,476
    Likes Received:
    3,103
    Gender:
    Male
    Hmm strange. What you have posted here looks okay to me.I can have a look if you PM me your site url.