1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

RSS Parsing and SQL

Discussion in 'Visual Basic .NET' started by prometheussoft, Oct 29, 2010.

  1. prometheussoft

    prometheussoft Newbie

    Jun 29, 2010
    Likes Received:
    Kailua, HI
    Home Page:
    For those who are interested. I wont go into much of the details for the usefullness of this code but some of you my like it. I recently was asked to help with RSS parsing. The idea was to catolog RSS Feeds into a sql server. The following might come in handy for some of you. These are the two fucntions that I wrote to break down RSS Feeds.

    Public Sub GetFeed()
    ' Bring back the Feed
    Dim doc As New XPathDocument(rssURL)
    Dim nav As XPathNavigator = doc.CreateNavigator()
    Dim iter As XPathNodeIterator = nav.Select(rssSelect)
    ' Loop through the nodes
    While iter.MoveNext()
    ' Get the data we need from the node
    End While
    End Sub 'GetFeed
    Sub ProcessNode(ByVal lstNav As XPathNavigator)
    Dim title As String = ""
    Dim link As String = ""
    Dim description As String = ""
    Dim pubdate As String = ""
    ' Get the child nodes
    Dim iterNews As XPathNodeIterator = lstNav.SelectDescendants(XPathNodeType.Element, False)
    ' Loop through the child nodes
    While iterNews.MoveNext()
    ' Save the current Name
    Dim rssName As String = iterNews.Current.Name
    ' Is this the title?
    If rssName.ToUpper() = rssTitle.ToUpper() Then
    title = iterNews.Current.Value
    End If
    ' Is this the Link?
    If rssName.ToUpper() = rssLink.ToUpper() Then
    link = iterNews.Current.Value
    End If
    ' Is this the Description?
    If rssName.ToUpper() = rssDescription.ToUpper() Then
    description = iterNews.Current.Value
    End If
    If rssName.ToUpper() = rssPubDate.ToUpper() Then
    pubdate = iterNews.Current.Value
    End If
    End While
    ' Make sure the link and title are at least 10 characters long
    If link.Length > 10 And title.Length > 10 Then
    ' Update the database
    InsertNews(SourceID, link, title, description, pubdate)
    End If
    End Sub 'ProcessNode

    For these functions to work you will need to populate a sql server table with the following fields:


    You can call the function simply like

    In my case the following is a simple example

    Dim Dr2 As DataRow
    For Each Dr2 In dt2.Rows
    sql =
    "Delete TempNews from tempnews"
    objCommand = New SqlCommand(sql, HomeServer)
    For Each dr In dt.Rows
    SourceID = dr(
    rssSelect = dr(
    rssTitle = dr(
    rssLink = dr(
    rssDescription = dr(
    rssURL = dr(
    "RSSURL").ToString() & Dr2("Symbol").ToString() & dr("RSSURLEND").ToString()
    rssPubDate = dr(
    symbol = Dr2(

    The idea is to populate the sql server table with rss feed info. Populate the table with the tags that appear before each of the nodes you're interested in. Then you can break down that RSS feed into the info you need and bring it back in to a sql server for analysis. I wrote this up pretty quick so I'm sorry if my brian is scattered. If you need any help using the function or want to see more of the code let me know.