Free unindexed content from forums

Discussion in 'Black Hat SEO' started by thejake, Apr 21, 2012.

  1. thejake

    thejake Power Member

    Nov 13, 2009
    Likes Received:
    Lately I've noticed many forum owners have weird robots.txt, noindex, and content cloaking setups, it seems to have thread lists indexed but not the actual post content, I guess because the thread titles have better keyword value, or to prevent content from being previewable, or something. By doing site: searches for a forum and manually inspecting page source and robots.txt you'll see that on some popular forums.

    To me seemed like a golden opportunity to grab some new-to-google content for autoblogging and other purposes. In vbulletin page source you'll find a link that looks like
    even if it's not visible on page, and in most installations that gives you the full content of the 15 most recent posts. You can add count= parameter which depending on site configuration allows you to get feed of up to 500 posts, and other type= parameters may be available but probably not useful. Every thread list in vbulletin will indicate the forumid # somewhere in the page source so you can construct your own external.php query string based on it regardless of external.php being visible in source.

    Keep in mind forum content is often crappy and spammy so filtering url's and bad words is usually a good idea, but to pad out an autoblog between money posts many forum owners are doing us favor, just make sure to not grab feeds too often or from same IP. Good luck bros.