1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt

Discussion in 'White Hat SEO' started by harrycole, Feb 10, 2016.

  1. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Hi Friends,

    Please help me to block few pages through robots.txt.


    I have a seo project when I was updating xml sitemap on my website i found few urls like

    website domain/pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    website domain/contact/income-tax-efiling-app/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    website domain/contact/bluebug-projects-2/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    etc.

    Please let me know, how can I block these types of page?
     
  2. gh0st_0

    gh0st_0 Senior Member

    Joined:
    Mar 3, 2015
    Messages:
    840
    Likes Received:
    503
    Occupation:
    Promoter
    Location:
    behind you
    you can try this....

    http://kb.yoast.com/article/279-how-to-edit-robots-txt-through-yoast-seo
     
  3. webinsight

    webinsight Junior Member

    Joined:
    Dec 17, 2015
    Messages:
    135
    Likes Received:
    12
    User-agent: *
    Disallow: /pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    Disallow: /contact/income-tax-efiling-app/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    Disallow: /contact/bluebug-projects-/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
     
  4. hrishivardhan

    hrishivardhan Registered Member

    Joined:
    Jul 11, 2014
    Messages:
    84
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Marketing Executive
    Location:
    India
    Home Page:
    This is very simple, any page that you want to disallow, just give root url of this file or folder. Just put this into your robots.txt file.

    Disallow: website domain/pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
     
  5. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    562
    Likes Received:
    173
    You need the following robots.txt:

    Because you have no page with "fb-login" to be indexed.
     
    • Thanks Thanks x 1
  6. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Thanks to all.

    I have done.
     
  7. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Thanks you !! MuayThai !! :thumb:
     
    • Thanks Thanks x 1
  8. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    562
    Likes Received:
    173
    Welcome!
    And do not forget to press "Thanks" button ;-)
     
  9. jadezoole

    jadezoole Junior Member

    Joined:
    Nov 11, 2014
    Messages:
    157
    Likes Received:
    17
    Nice Thank you!