Spiders on dynamic content?

Shameless

Newbie
Joined
Mar 17, 2008
Messages
41
Reaction score
3
Hey,

I'm programming now a video site, this site is totally dynamic, content is created on request, for example:

site.com/videos/election (where election is the keyword, matching videos for that will be displayed)

site.com/tags/obama (where obama is the tag, videos tagged like that will be displayed)

site.com/author/whatev (where whatev is the author, all videos by the author gets displayed)

site.com/watch/xyz/video-title-here (xyz is the youtube code, video-title-here is the for the seo friendly url)

All pages has links to relevant other pages within the site, if they get clicked the content get's generated for the request too. Hope you are not confused yet :)

So, when a spider visits the site, they crawl the whole site based on the links on each page right? But what happens to dynamic content like that? The spider could hang out forever on the site crawling all links it finds. Just dont know if thats a good or bad thing.

Tell me what you think, cheers.
 
I have a dynamic site and practically crawls everything within time. If the spider so happens to visit an obama video when it was generated, it will get crawled to. Im sure other might agree or disagree.
 
good thing, lotsa of internal linking, be careful to meta robots noindex,follow the joining/category/tag etc pages and let the bots only index the actual content pages, also dump links into each tag/category/sitemap page that is important... this is in whitehat though so do as you please, deeplinking, personally id go blackhat and give each 1k links on their own but :)
 
Good advice with the noindex follow, thank you.
 
The spider's are totally hip to mod_rewrite and handle it extremely well. If there are links, they will follow them. This is one reason why blogs get indexed so quickly, there are more "points-of-entry" for the spiders to approach on.
 
Back
Top