Robots.txt

Discussion in 'White Hat SEO' started by harrycole, Feb 10, 2016.

  1. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Hi Friends,

    Please help me to block few pages through robots.txt.


    I have a seo project when I was updating xml sitemap on my website i found few urls like

    website domain/pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    website domain/contact/income-tax-efiling-app/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    website domain/contact/bluebug-projects-2/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    etc.

    Please let me know, how can I block these types of page?
     
  2. gh0st_0

    gh0st_0 Senior Member

    Joined:
    Mar 3, 2015
    Messages:
    853
    Likes Received:
    523
    Occupation:
    Promoter
    Location:
    behind you
    you can try this....

    http://kb.yoast.com/article/279-how-to-edit-robots-txt-through-yoast-seo
     
  3. webinsight

    webinsight Junior Member

    Joined:
    Dec 17, 2015
    Messages:
    135
    Likes Received:
    12
    User-agent: *
    Disallow: /pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    Disallow: /contact/income-tax-efiling-app/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
    Disallow: /contact/bluebug-projects-/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
     
  4. hrishivardhan

    hrishivardhan Registered Member

    Joined:
    Jul 11, 2014
    Messages:
    84
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Marketing Executive
    Location:
    India
    This is very simple, any page that you want to disallow, just give root url of this file or folder. Just put this into your robots.txt file.

    Disallow: website domain/pages/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fb-login/fbconfig.php
     
  5. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    753
    Likes Received:
    231
    You need the following robots.txt:

    Because you have no page with "fb-login" to be indexed.
     
    • Thanks Thanks x 1
  6. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Thanks to all.

    I have done.
     
  7. harrycole

    harrycole Newbie

    Joined:
    Feb 10, 2016
    Messages:
    3
    Likes Received:
    1
    Thanks you !! MuayThai !! :thumb:
     
    • Thanks Thanks x 1
  8. MuayThai

    MuayThai Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    753
    Likes Received:
    231
    Welcome!
    And do not forget to press "Thanks" button ;-)
     
  9. jadezoole

    jadezoole Junior Member

    Joined:
    Nov 11, 2014
    Messages:
    173
    Likes Received:
    22
    Nice Thank you!