Using robots.txt to drip feed content

J

jing0

Junior Member

Apr 15, 2012

100

12

• Jul 1, 2014

• #1

Hey guys,

From a user perspective, I need to get all my content onto my website at once. However, I'm well aware that from an SEO perspective it's best to drip feed content.

I was thus wondering, if I manually add every page URL to robots.txt and then remove say 3 every few days - will this work as effectively drip feeding content in the search engines' eyes?

Thanks!

 

Techxan

Elite Member

Dec 7, 2011

3,091

3,620

• Jul 1, 2014

• #2

no, because your site may not be crawled everyday.

 

J

jing0

Junior Member

Apr 15, 2012

100

12

• Jul 1, 2014
• Thread Starter Thread Starter

• #3

Ah fair point. Surely there'd still be some amount of drip feeding going on if the site only gets crawled every 2 weeks or something though? Or I could ping the links when I remove them from robots.txt?

 

Techxan

Elite Member

Dec 7, 2011

3,091

3,620

• Jul 1, 2014

• #4

just use a post scheduler.

 

J

jing0

Junior Member

Apr 15, 2012

100

12

• Jul 1, 2014
• Thread Starter Thread Starter

• #5

Thanks for the suggestion but I really can't use a post scheduler as I need the text there for user experience - no other way around it.

 

A

augustprojekt

Newbie

Jun 30, 2014

12

2

• Jul 1, 2014

• #6

What if you did a "fetch as google" each time you update your robots.txt? BHW won't let me post a link (I'm too new), but if you go into Webmaster Tools and then "Crawl" then "Fetch as Google"