1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Using robots.txt to drip feed content

Discussion in 'White Hat SEO' started by jing0, Jul 1, 2014.

  1. jing0

    jing0 Junior Member

    Joined:
    Apr 15, 2012
    Messages:
    100
    Likes Received:
    12
    Hey guys,

    From a user perspective, I need to get all my content onto my website at once. However, I'm well aware that from an SEO perspective it's best to drip feed content.

    I was thus wondering, if I manually add every page URL to robots.txt and then remove say 3 every few days - will this work as effectively drip feeding content in the search engines' eyes?

    Thanks!
     
  2. Techxan

    Techxan Elite Member

    Joined:
    Dec 7, 2011
    Messages:
    3,093
    Likes Received:
    3,585
    Occupation:
    Local SEOist
    Location:
    TEXAS (you have to yell, its the law.)
    no, because your site may not be crawled everyday.
     
    • Thanks Thanks x 1
  3. jing0

    jing0 Junior Member

    Joined:
    Apr 15, 2012
    Messages:
    100
    Likes Received:
    12
    Ah fair point. Surely there'd still be some amount of drip feeding going on if the site only gets crawled every 2 weeks or something though? Or I could ping the links when I remove them from robots.txt?
     
  4. Techxan

    Techxan Elite Member

    Joined:
    Dec 7, 2011
    Messages:
    3,093
    Likes Received:
    3,585
    Occupation:
    Local SEOist
    Location:
    TEXAS (you have to yell, its the law.)
    just use a post scheduler.
     
  5. jing0

    jing0 Junior Member

    Joined:
    Apr 15, 2012
    Messages:
    100
    Likes Received:
    12
    Thanks for the suggestion but I really can't use a post scheduler as I need the text there for user experience - no other way around it.
     
  6. augustprojekt

    augustprojekt Newbie

    Joined:
    Jun 30, 2014
    Messages:
    12
    Likes Received:
    2
    Occupation:
    editor
    Location:
    Los Angeles
    Home Page:
    What if you did a "fetch as google" each time you update your robots.txt? BHW won't let me post a link (I'm too new), but if you go into Webmaster Tools and then "Crawl" then "Fetch as Google"