VB.net Proxy doesn't change after first HttpWebRequest

[frankweerasinghe]

I have a public function to get the pagesource of web URLs. Here is the function.

Public Function gethtml(ByVal id As Integer)
    Try
                    ListView1.Items(id).SubItems(6).Text = "Downloading HTML..."
                    Dim Response3 As HttpWebResponse
                    Dim pagesource3 As String
                    Dim datastream As Stream
                    Dim postdata As String = ""
                    Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postdata)
                    Dim Request3 As HttpWebRequest = HttpWebRequest.Create("http://mysite.com/ip.php")
                    Request3.Referer = "http://google.com/"
                    Request3.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
                    Request3.KeepAlive = True
                    Request3.Timeout = "5000"
                    Request3.AllowAutoRedirect = True
                    If useproxycheckbox.Checked = True Then
                        Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
                        Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)
                        Request3.UseDefaultCredentials = False
                        Request3.Proxy.Credentials = New System.Net.NetworkCredential(words(2).ToString, words(3).ToString)
                    End If
                    Request3.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31"
                    Request3.CookieContainer = CC
                    Request3.Method = WebRequestMethods.Http.Get
                    Request3.ContentType = "application/x-www-form-urlencoded"
                    Response3 = Request3.GetResponse()
                    Dim Stream3 As New StreamReader(Response3.GetResponseStream())
                    pagesource3 = Stream3.ReadToEnd()
                    Response3.Close()
                    Stream3.Close()
                    ListView1.Items(id).SubItems(4).Text = pagesource3
                    
    Catch e1 As ProtocolViolationException
        recheck(id)
        ListView1.Items(id).SubItems(6).Text = "Error1"
    Catch e2 As WebException
        recheck(id)
        ListView1.Items(id).SubItems(6).Text = "Error2"
                            
    End Try
    End Function

The problem is it works fine with first call like

 gethtml(0)

But after that, it doesn't use the proxy I provide to it. it simply uses the same proxy used in the 1st request. I'm calling this function using a thread because if I use this on main thread I'll freeze the UI. What I'm doing wrong? I'm a noob in VB.net


Thanks.

 

[Naveen257]

I would like to know this as well. Anyone ??

 

[frankweerasinghe]

I would like to know this as well. Anyone ??

Thanks man!

ANYONE???

 

[qrazy]

The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).

 

[frankweerasinghe]

The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).

Hi, Yes I'm sure I was passing the updated value to the function.. This is really wired... If I send you the project file, can you help me?

 

[hpv222]

I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

You should probably check out these two lines:

Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


Again, I could be way off, just trying to help

 

[frankweerasinghe]

I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

You should probably check out these two lines:

Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


Again, I could be way off, just trying to help

Hi, Thanks for the reply. In my list, I have loaded my proxies in PROXYORT:USERNAMEASSWORD format and getting unique one using line number. (id). Then, I separate that string by ":" and provide them to the appropriate fields. New WebProxy class can convert string into a System.WebProxy so I can finally use it to the HTTPWebRequest.

Yes, I have also tried the way you posted. Still no luck!!!

 

[healzer]

watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
Do you know how to set breakpoints/debug the program?

 

[frankweerasinghe]

watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
Do you know how to set breakpoints/debug the program?

PMed you

 

[hpv222]

I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAMEASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?

 

[healzer]

I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAMEASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?

I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-

 

[hpv222]

I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-

Not going to argue with you when I don't know much about VB In C# even when you use ParallelForEach, each HTTPWebRequest can use its own proxy and Network Credentials, they are thread safe. If you have to use any method or variable that is not thread safe, then you use the lock statement.

Well, let's hope that the OP has figured it out by now

 

[frankweerasinghe]

For some reason, I can't edit above post. I found something... This problem only happens with proxies that having basic username password authentication. It works well with normal IPORT proxies. Help me!

 

[accelerator_dd]

If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .

 

[frankweerasinghe]

If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .

PMed you hope you can solve this damn problem

 

[accelerator_dd]

PMed you hope you can solve this damn problem

There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.

 

[healzer]

There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.

I did the same thing.
Everything works as it should, this is a problem with the Framework not the code I think

 

[accelerator_dd]

I did the same thing.
Everything works as it should, this is a problem with the Framework not the code I think

It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.

 

[seorebel]

How about splitting it up? One function gets a new proxy, and another function makes the webrequest from the new proxy and credentials..

 

[healzer]

It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.

I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak