1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any VB.NET Coders? Having Small Issue With HTTPWebRequest

Discussion in 'Visual Basic .NET' started by gimme4free, Sep 22, 2011.

  1. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    Hey Guys, Looking for anyone that knows a bit about VB.NET & HTTPWebRequest.

    I'm new to VB, been learning for the past 3 months now and have finalyl gotten around to making up a new app from scratch but am running into issues already!

    Below is a function from a class, I am using this function to login to Twitter:

    Code:
    Private cookies As CookieContainer
    
    Public Function Post_Content(ByVal URL As String, ByVal Data As String, Optional ByVal Referrer As String = "", Optional ByVal MultiPart As Boolean = False) As String
            Dim encoding As New UTF8Encoding
            Dim byteData As Byte() = encoding.GetBytes(Data)
            Dim Content As String = Nothing
            Try
                ' New Request
                System.Net.ServicePointManager.Expect100Continue = False
    
                Dim request As HttpWebRequest = DirectCast(WebRequest.Create(URL), HttpWebRequest)
                ' Set Proxy Details
                Dim myProxy As New WebProxy(Proxy, Port)
                myProxy.Credentials = New NetworkCredential(ProxyUser, ProxyPass)
                request.Proxy = myProxy
                ' Request Settings
                request.Method = "POST"
                ' request.KeepAlive = True
                request.AllowAutoRedirect = True
                request.Timeout = MaxTimeout
                request.CookieContainer = cookies
                request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)"
                request.ContentLength = byteData.Length
                If MultiPart = False Then
                    request.ContentType = "application/x-www-form-urlencoded"
                Else
                    request.ContentType = "multipart/form-data"
                End If
                If Referrer <> "" Then request.Referer = Referrer
                ' Read Stream
                Dim postreqstream As Stream = request.GetRequestStream()
                postreqstream.Write(byteData, 0, byteData.Length)
                postreqstream.Close()
                Dim postresponse As HttpWebResponse
                postresponse = DirectCast(request.GetResponse(), HttpWebResponse)
                ' Check HTTP Status
                If postresponse.StatusCode = Net.HttpStatusCode.OK Then
                    Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
                    Content = postreqreader.ReadToEnd
                End If
                ' Add Cookies
                If postresponse.Cookies.Count > 0 Then cookies.Add(postresponse.Cookies)
                ' Close Connection
                postresponse.Close()
            Catch ex As Exception
                Throw New Exception("Error occurred while posting content to '" + URL + "': " + ex.Message, ex)
            End Try
            ' Return HTML Content
            Return Content
        End Function
    The issue is that the cookies are not being saved to my cookie container. After using the above function to register a new account the account is registered just fine and I can use the same credentials in my browser to login but instantly within VB, after the POST the next page I am already logged back out again before EVER seeing a logged in page.

    Would this be due to some sort of redirect in place?

    I have created the same bot in PHP/CURL that works fine and POST's exactly the same data so it must be something to do with my POST function and/or my cookiecollection.
     
    • Thanks Thanks x 2
  2. Bross

    Bross Senior Member

    Joined:
    Feb 6, 2010
    Messages:
    859
    Likes Received:
    355
    What resource do you use to learn VB.net?

    Also looking into that.. Please advise if you have a good resource.
     
  3. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    Hi,

    The best way to solve this is to have a look at the datastream with an external program like wireshark to get to know where it gets fucked up and what is beeing sent and get.

    1. Does your container contain the cookies when you debug?
    2. Does your proxys require auth everytime you make a connection? (if so you might need to recode alot).


    Regards,
     
    • Thanks Thanks x 1
  4. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    Proxies are IP authenticated but in the event that I ever switch to user/pass proxies I have left this part in the function:

    Code:
    Dim myProxy As New WebProxy(Proxy, Port)
    myProxy.Credentials = New NetworkCredential(ProxyUser, ProxyPass)
    request.Proxy = myProxy
    It doesn't seem that the cookies are being added into the cookie container at all at the moment. I am pretty sure that the container is within scope.

    The cookie container is declared in my base class:
    Private cookies As CookieContainer

    And my subclass Inherits from my main class. The subclass also has a constructor that sets the HTTP settings in the base class to use for both classes.

    When I try to print out my cookies I am getting a null reference error like the object is empty (using .ToString() method).
     
  5. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    This site has some pretty decent videos for understanding the basics of VB.NET:
    Code:
    http://howtostartprogramming.com/vb-net/
    I also read the O'Reilly's step-by-step guide which gave me a good overview of the basics but really it didn't teach me much other than the very very basics and terminology. After that I began reading Wrox's Professional VB.NET book, got around halfway through and gave up as the book covered subjects that I would just never use so decided to put what I had learnt into practice from there and google things I was unsure of.
     
    • Thanks Thanks x 1
  6. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    Since CookieContainer is a class and probably not a share one(never used it) you need to make a new statement.

    Somewhere before usage you have to write:
    cookies = new CookieContainer

    Or change (usualy not prefered but) the declare to
    Private cookies As new CookieContainer

    Otherwise cookies will be a null reference
     
    • Thanks Thanks x 1
  7. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    I have been Googling, racking my brain and playing with code for hours all thanks to a missing, "New"!!! Thanks a lot for the help, finally got the code working again :)
     
  8. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    Simple errors like that are usualy the most anoying ones since you can stare at the error without seeing it. Glad you made it work.
     
    • Thanks Thanks x 1
  9. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    One other minor query for VB Coders:

    For what reason would Intellisense in VS2010 be detecting ex as an unused variable in the following code:

    Code:
    Try
                Content = Return_Content("http://twitter.com/welcome")
            Catch ex As Exception
                Throw New Exception("Error Retrieving Twitter Welcome Page: " & ex.Message)
            End Try
     
  10. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    For no reson at all. :)
    My vs2010 does not.
     
    • Thanks Thanks x 1
  11. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    It's a strange one, I have other try catch blocks that use exactly the same code apart from the message & URL have been changed but 2/5 of them are telling me that they are an unused local variable :S
     
  12. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    Hmm took a quick look at msdn with this issue. Somene stated that intellisense detects that there is no way you can make a error that a try/catch can fetch. so in a way it's unused since it will never give you an error. :D

    Edit: and best practice to use try catch imo is not to wrap whole subs/functions but code snippets instead.
     
    • Thanks Thanks x 1
  13. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    Originally I had just the following line in the Try Catch:
    Dim response As Net.HttpWebResponse = request.GetResponse()

    But after posting up on another site with another quick Q somebody told me to wrap the lot in the Try/Catch block. Too many people with varying opinions around!!! The rest of my code is pretty much 1 line try/catch apart from the Get/Post functions.

    Edit: Just re-read your post and think you meant to not wrap the reference to the function (Return_Content(URL)) and to instead just have the try/catch within the function and set that to handle the error?
     
  14. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,060
    You're throwing an exception in response to handling another exception, which means your catch block is essentially doing nothing. That's probably why it's saying it's unused.
     
    • Thanks Thanks x 1
  15. HatIsBlack

    HatIsBlack Regular Member

    Joined:
    Sep 17, 2010
    Messages:
    265
    Likes Received:
    187
    Location:
    Where i belong
    Yes you got my point to move the try catch inside the Return_Content.
    when it comes to large code blocks inside a try catch it depends a little on the code (I'm not against it). At httpwebrequest functions i would rather wrap a single line since you can catch and have difrent soultions depending on the errors. What you need to remember is to use as few try catch functions as possible in your code. if you code correctly you don't need them unless you use classes like httpwebrequest that you don't have controll over. If you wrap a whole code inside a function you can be pretty sure your program don't crash but may end up in less functionality and less accurate errors to work with.

    For example this line
    postresponse = DirectCast(request.GetResponse(), HttpWebResponse)

    why not use webexception instead of exeption in a try catch since that is what is thrown by webresponse class. Anyway good luck with your code.
     
    • Thanks Thanks x 1
  16. w84it

    w84it Regular Member

    Joined:
    Jun 18, 2009
    Messages:
    443
    Likes Received:
    752
    If youre still having trouble, you might wanna use this class I have.
    Webclient with cookie support.
    Code:
    Public Class CookieAwareWebClient
        Inherits WebClient
        Public m_container As New CookieContainer()
        Protected Overrides Function GetWebRequest(ByVal address As Uri) As WebRequest
            Dim request As HttpWebRequest = MyBase.GetWebRequest(address)
            request.ProtocolVersion = HttpVersion.Version10
            request.KeepAlive = False
            request.Timeout = 60000
            request.ReadWriteTimeout = 60000
            If TypeOf request Is HttpWebRequest Then
                TryCast(request, HttpWebRequest).CookieContainer = m_container
            End If
            Return request
        End Function
    End Class
    
    Save cookies to a file.
    Code:
     Dim formatter = New BinaryFormatter()
                        Dim file__1 As String = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), My.Application.Info.DirectoryPath + "\cookiefile.dat")
                        Using s As Stream = File.Create(file__1)
                            formatter.Serialize(s, http.m_container)
                        End Using
    
    Load cookies from a file.
    Code:
    
    Dim retrievedCookies As CookieContainer = Nothing
                            Dim bform As New BinaryFormatter
                            Using s As Stream = File.OpenRead((My.Application.Info.DirectoryPath + "\cookiefile.dat"))
                                retrievedCookies = DirectCast(bform.Deserialize(s), CookieContainer)
                            End Using
                            http.m_container = retrievedCookies
    
     
    • Thanks Thanks x 1
  17. Hydrogen

    Hydrogen Newbie

    Joined:
    Dec 30, 2009
    Messages:
    39
    Likes Received:
    23
    Occupation:
    Co-Owner of AdvertMarketing
    Home Page:
    Code:
    Private cookies As CookieContainer
    
    Should be

    Code:
    Private cookies As NEW CookieContainer
    
    Otherwise cookies will not get filled with the cookies properly. Also everytime you login, post, get whatever you'll want to set the httpwebrequest cookiecontainer to cookies.

    Code:
    request.CookieContainer = cookies
    
    as far as the try statement goes.

    it works like this

    Code:
    Try
         '....Do Code here to 'TRY'
    Catch ex As Exception
         '.... process error here or leave blank and do nothing it's best to handle exceptions as they are thrown.  At no time do you want to throw another exception, as we already have one.
    End Try
    
     
    • Thanks Thanks x 1
  18. v7web

    v7web Registered Member

    Joined:
    Jul 15, 2010
    Messages:
    54
    Likes Received:
    3
    Does this method still work for twitter?
     
  19. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,913
    Likes Received:
    1,940
    Home Page:
    Which method exactly? Both my logon apps for Twitter are working although I only made them last week so not sure if there have been any changes.