1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

200 out 170 url blocked by robot.txt what can i do

Discussion in 'White Hat SEO' started by irfanrufi123, Aug 22, 2013.

  1. irfanrufi123

    irfanrufi123 Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2012
    Messages:
    1,052
    Likes Received:
    220
    Occupation:
    Internet Marketer
    Location:
    Pakistan
    Home Page:
    200 out 170 url blocked by robot.txt what can i do these are the stats from google webmaster tool which i visited randomly after my site traffic almost went nada ,
    can anyone tell what robot.txt i should use
    btw its hosted on blogger custom domain
    www.apkdude.com
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  3. irfanrufi123

    irfanrufi123 Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2012
    Messages:
    1,052
    Likes Received:
    220
    Occupation:
    Internet Marketer
    Location:
    Pakistan
    Home Page:
    SO far i have no robot.txt file u know any robot.txt generator ??
    refer me to site
     
  4. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  5. irfanrufi123

    irfanrufi123 Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2012
    Messages:
    1,052
    Likes Received:
    220
    Occupation:
    Internet Marketer
    Location:
    Pakistan
    Home Page:
  6. Dlckhead

    Dlckhead Newbie

    Joined:
    Jul 11, 2013
    Messages:
    31
    Likes Received:
    29
    You do have robots.txt
    It's in the root directory of your website.
    You can check going to http://www.apkdude.com/robots.txt
     
  7. SeoCrimeIndia

    SeoCrimeIndia Newbie

    Joined:
    Jun 5, 2013
    Messages:
    30
    Likes Received:
    2
    Occupation:
    Web Development and SEO
    Location:
    India
    Your robots.txt file containing


    User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: /

    So You just edit the text file Like following

    User-agent: *
    Disallow: /
     
    • Thanks Thanks x 1
  8. irfanrufi123

    irfanrufi123 Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2012
    Messages:
    1,052
    Likes Received:
    220
    Occupation:
    Internet Marketer
    Location:
    Pakistan
    Home Page:
    thats it no more or less thanks dude
    but isnt robot.txt longer than that?
     
  9. CredibleZephyre

    CredibleZephyre Registered Member

    Joined:
    Jun 10, 2013
    Messages:
    95
    Likes Received:
    27
    no... maybe an XML one is but the .txt file really can be that short. Literally what that says is "useragent: anyone, allow: everything" and that's all you need unless you have a reason for blocking pages
     
  10. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,139
    Likes Received:
    28,607
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
  11. irfanrufi123

    irfanrufi123 Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2012
    Messages:
    1,052
    Likes Received:
    220
    Occupation:
    Internet Marketer
    Location:
    Pakistan
    Home Page: