1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

VB.net Proxy doesn't change after first HttpWebRequest

Discussion in 'Visual Basic .NET' started by frankweerasinghe, Jun 7, 2013.

  1. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka
    I have a public function to get the pagesource of web URLs. Here is the function.


    Code:
    Public Function gethtml(ByVal id As Integer)
        Try
                        ListView1.Items(id).SubItems(6).Text = "Downloading HTML..."
                        Dim Response3 As HttpWebResponse
                        Dim pagesource3 As String
                        Dim datastream As Stream
                        Dim postdata As String = ""
                        Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postdata)
                        Dim Request3 As HttpWebRequest = HttpWebRequest.Create("http://mysite.com/ip.php")
                        Request3.Referer = "http://google.com/"
                        Request3.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
                        Request3.KeepAlive = True
                        Request3.Timeout = "5000"
                        Request3.AllowAutoRedirect = True
                        If useproxycheckbox.Checked = True Then
                            Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
                            Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)
                            Request3.UseDefaultCredentials = False
                            Request3.Proxy.Credentials = New System.Net.NetworkCredential(words(2).ToString, words(3).ToString)
                        End If
                        Request3.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31"
                        Request3.CookieContainer = CC
                        Request3.Method = WebRequestMethods.Http.Get
                        Request3.ContentType = "application/x-www-form-urlencoded"
                        Response3 = Request3.GetResponse()
                        Dim Stream3 As New StreamReader(Response3.GetResponseStream())
                        pagesource3 = Stream3.ReadToEnd()
                        Response3.Close()
                        Stream3.Close()
                        ListView1.Items(id).SubItems(4).Text = pagesource3
                        
        Catch e1 As ProtocolViolationException
            recheck(id)
            ListView1.Items(id).SubItems(6).Text = "Error1"
        Catch e2 As WebException
            recheck(id)
            ListView1.Items(id).SubItems(6).Text = "Error2"
                                
        End Try
        End Function
    The problem is it works fine with first call like


    Code:
     gethtml(0)
    
    But after that, it doesn't use the proxy I provide to it. it simply uses the same proxy used in the 1st request. I'm calling this function using a thread because if I use this on main thread I'll freeze the UI. What I'm doing wrong? I'm a noob in VB.net


    Thanks.
     
  2. Naveen257

    Naveen257 Regular Member

    Joined:
    Sep 29, 2012
    Messages:
    279
    Likes Received:
    163
    Location:
    next door
    I would like to know this as well. Anyone ??
     
    • Thanks Thanks x 1
  3. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka
    Thanks man! ;)

    ANYONE???
     
  4. qrazy

    qrazy Senior Member

    Joined:
    Mar 19, 2012
    Messages:
    1,103
    Likes Received:
    1,679
    Location:
    Banana Republic
    The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).
     
    • Thanks Thanks x 1
  5. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka
    Hi, Yes I'm sure I was passing the updated value to the function.. This is really wired... If I send you the project file, can you help me?
     
  6. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274
    I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


    Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

    You should probably check out these two lines:

    Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
    Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


    Again, I could be way off, just trying to help ;)
     
    • Thanks Thanks x 1
  7. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka
    Hi, Thanks for the reply. In my list, I have loaded my proxies in PROXY:pORT:USERNAME:pASSWORD format and getting unique one using line number. (id). Then, I separate that string by ":" and provide them to the appropriate fields. New WebProxy class can convert string into a System.WebProxy so I can finally use it to the HTTPWebRequest.

    Yes, I have also tried the way you posted. Still no luck!!! :(
     
  8. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,328
    Likes Received:
    1,950
    Gender:
    Male
    Occupation:
    Conversion Rate Optimization
    Location:
    Somewhere in Europe
    Home Page:
    watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
    Do you know how to set breakpoints/debug the program?
     
  9. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka

    PMed you :)
     
  10. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274
    I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

    Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAME:pASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?
     
  11. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,328
    Likes Received:
    1,950
    Gender:
    Male
    Occupation:
    Conversion Rate Optimization
    Location:
    Somewhere in Europe
    Home Page:
    I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-
     
  12. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274
    Not going to argue with you when I don't know much about VB ;) In C# even when you use ParallelForEach, each HTTPWebRequest can use its own proxy and Network Credentials, they are thread safe. If you have to use any method or variable that is not thread safe, then you use the lock statement.

    Well, let's hope that the OP has figured it out by now
     
  13. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka
    For some reason, I can't edit above post. I found something... This problem only happens with proxies that having basic username password authentication. It works well with normal IP:pORT proxies. Help me!
     
  14. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,440
    Likes Received:
    1,001
    Occupation:
    SEO
    Location:
    IM Wonderland
    If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .
     
    • Thanks Thanks x 1
  15. frankweerasinghe

    frankweerasinghe Regular Member

    Joined:
    Jun 6, 2011
    Messages:
    432
    Likes Received:
    393
    Location:
    Colombo, Sri Lanka

    PMed you :) hope you can solve this damn problem :D
     
  16. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,440
    Likes Received:
    1,001
    Occupation:
    SEO
    Location:
    IM Wonderland
    There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

    I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

    To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.
     
  17. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,328
    Likes Received:
    1,950
    Gender:
    Male
    Occupation:
    Conversion Rate Optimization
    Location:
    Somewhere in Europe
    Home Page:
    I did the same thing.
    Everything works as it should, this is a problem with the Framework not the code I think
     
  18. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,440
    Likes Received:
    1,001
    Occupation:
    SEO
    Location:
    IM Wonderland
    It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

    I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.
     
  19. seorebel

    seorebel Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 15, 2008
    Messages:
    174
    Likes Received:
    920
    How about splitting it up? One function gets a new proxy, and another function makes the webrequest from the new proxy and credentials..
     
  20. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,328
    Likes Received:
    1,950
    Gender:
    Male
    Occupation:
    Conversion Rate Optimization
    Location:
    Somewhere in Europe
    Home Page:
    I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak