1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What this means (robots.txt)

Discussion in 'Black Hat SEO' started by moonlighsunligh, Aug 16, 2014.

  1. moonlighsunligh

    moonlighsunligh Jr. VIP Jr. VIP Premium Member

    Joined:
    May 1, 2010
    Messages:
    1,625
    Likes Received:
    218
    The robots.txt at webnode.com says:

    User-agent: psbot
    Disallow: /
    User-agent: *
    Disallow:
    Crawl-delay: 10
    Sitemap: http://somedomain.webnode.com/sitemap.xml


    What does it mean?

    I guess that

    User-agent: *
    Disallow:

    isn't the same as

    User-agent: *
    Disallow:/
     
  2. ID Internet Marketer

    ID Internet Marketer Senior Member

    Joined:
    Jan 22, 2013
    Messages:
    938
    Likes Received:
    1,442
    Occupation:
    Blackhatworld Member
    Location:
    My Private ***
    User-agent: *
    Disallow:

    (to allow all robots complete access)

    User-agent: *
    Disallow:/

    (to exclude all robots from entire server)
     
    • Thanks Thanks x 1
  3. moonlighsunligh

    moonlighsunligh Jr. VIP Jr. VIP Premium Member

    Joined:
    May 1, 2010
    Messages:
    1,625
    Likes Received:
    218
    So
    User-agent: *
    Disallow:

    is the same as
    User-agent: *
    Allow: /
     
  4. ID Internet Marketer

    ID Internet Marketer Senior Member

    Joined:
    Jan 22, 2013
    Messages:
    938
    Likes Received:
    1,442
    Occupation:
    Blackhatworld Member
    Location:
    My Private ***
    yup... they work same as.
     
    • Thanks Thanks x 1
  5. watsonovedades

    watsonovedades Senior Member

    Joined:
    Mar 10, 2011
    Messages:
    922
    Likes Received:
    233
    Location:
    Mexico
    Whats is the SEO benefit on modifying Robots.txt?