1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Spiders on dynamic content?

Discussion in 'White Hat SEO' started by Shameless, Apr 10, 2008.

  1. Shameless

    Shameless Newbie

    Joined:
    Mar 17, 2008
    Messages:
    41
    Likes Received:
    3
    Home Page:
    Hey,

    I'm programming now a video site, this site is totally dynamic, content is created on request, for example:

    site.com/videos/election (where election is the keyword, matching videos for that will be displayed)

    site.com/tags/obama (where obama is the tag, videos tagged like that will be displayed)

    site.com/author/whatev (where whatev is the author, all videos by the author gets displayed)

    site.com/watch/xyz/video-title-here (xyz is the youtube code, video-title-here is the for the seo friendly url)

    All pages has links to relevant other pages within the site, if they get clicked the content get's generated for the request too. Hope you are not confused yet :)

    So, when a spider visits the site, they crawl the whole site based on the links on each page right? But what happens to dynamic content like that? The spider could hang out forever on the site crawling all links it finds. Just dont know if thats a good or bad thing.

    Tell me what you think, cheers.
     
  2. general34

    general34 Junior Member

    Joined:
    Dec 19, 2007
    Messages:
    167
    Likes Received:
    33
    Occupation:
    Making Cash Money
    Location:
    Florianapolis
    Home Page:
    I have a dynamic site and practically crawls everything within time. If the spider so happens to visit an obama video when it was generated, it will get crawled to. Im sure other might agree or disagree.
     
  3. trophaeum

    trophaeum Senior Member

    Joined:
    Dec 21, 2007
    Messages:
    1,189
    Likes Received:
    706
    good thing, lotsa of internal linking, be careful to meta robots noindex,follow the joining/category/tag etc pages and let the bots only index the actual content pages, also dump links into each tag/category/sitemap page that is important... this is in whitehat though so do as you please, deeplinking, personally id go blackhat and give each 1k links on their own but :)
     
  4. Shameless

    Shameless Newbie

    Joined:
    Mar 17, 2008
    Messages:
    41
    Likes Received:
    3
    Home Page:
    Good advice with the noindex follow, thank you.
     
  5. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    The spider's are totally hip to mod_rewrite and handle it extremely well. If there are links, they will follow them. This is one reason why blogs get indexed so quickly, there are more "points-of-entry" for the spiders to approach on.