VB.net Proxy doesn't change after first HttpWebRequest

frankweerasinghe

frankweerasinghe

Power Member
Joined
Jun 6, 2011
Messages
555
Reaction score
430
I have a public function to get the pagesource of web URLs. Here is the function.


Code: 
Public Function gethtml(ByVal id As Integer)
    Try
                    ListView1.Items(id).SubItems(6).Text = "Downloading HTML..."
                    Dim Response3 As HttpWebResponse
                    Dim pagesource3 As String
                    Dim datastream As Stream
                    Dim postdata As String = ""
                    Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postdata)
                    Dim Request3 As HttpWebRequest = HttpWebRequest.Create("http://mysite.com/ip.php")
                    Request3.Referer = "http://google.com/"
                    Request3.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
                    Request3.KeepAlive = True
                    Request3.Timeout = "5000"
                    Request3.AllowAutoRedirect = True
                    If useproxycheckbox.Checked = True Then
                        Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
                        Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)
                        Request3.UseDefaultCredentials = False
                        Request3.Proxy.Credentials = New System.Net.NetworkCredential(words(2).ToString, words(3).ToString)
                    End If
                    Request3.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31"
                    Request3.CookieContainer = CC
                    Request3.Method = WebRequestMethods.Http.Get
                    Request3.ContentType = "application/x-www-form-urlencoded"
                    Response3 = Request3.GetResponse()
                    Dim Stream3 As New StreamReader(Response3.GetResponseStream())
                    pagesource3 = Stream3.ReadToEnd()
                    Response3.Close()
                    Stream3.Close()
                    ListView1.Items(id).SubItems(4).Text = pagesource3
                    
    Catch e1 As ProtocolViolationException
        recheck(id)
        ListView1.Items(id).SubItems(6).Text = "Error1"
    Catch e2 As WebException
        recheck(id)
        ListView1.Items(id).SubItems(6).Text = "Error2"
                            
    End Try
    End Function

The problem is it works fine with first call like


Code: 
 gethtml(0)

But after that, it doesn't use the proxy I provide to it. it simply uses the same proxy used in the 1st request. I'm calling this function using a thread because if I use this on main thread I'll freeze the UI. What I'm doing wrong? I'm a noob in VB.net


Thanks.
 
Advertise on BHW
The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).
 
qrazy said:
The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).
Click to expand...

Hi, Yes I'm sure I was passing the updated value to the function.. This is really wired... If I send you the project file, can you help me?
 
I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

You should probably check out these two lines:

Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


Again, I could be way off, just trying to help ;)
 
hpv222 said:
I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

You should probably check out these two lines:

Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


Again, I could be way off, just trying to help ;)
Click to expand...

Hi, Thanks for the reply. In my list, I have loaded my proxies in PROXY:PORT:USERNAME:PASSWORD format and getting unique one using line number. (id). Then, I separate that string by ":" and provide them to the appropriate fields. New WebProxy class can convert string into a System.WebProxy so I can finally use it to the HTTPWebRequest.

Yes, I have also tried the way you posted. Still no luck!!! :(
 
watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
Do you know how to set breakpoints/debug the program?
 
I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAME:PASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?
 
hpv222 said:
I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAME:PASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?
Click to expand...

I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-
 
healzer said:
I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-
Click to expand...

Not going to argue with you when I don't know much about VB ;) In C# even when you use ParallelForEach, each HTTPWebRequest can use its own proxy and Network Credentials, they are thread safe. If you have to use any method or variable that is not thread safe, then you use the lock statement.

Well, let's hope that the OP has figured it out by now
 
For some reason, I can't edit above post. I found something... This problem only happens with proxies that having basic username password authentication. It works well with normal IP:PORT proxies. Help me!
 
If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .
 
frankweerasinghe said:
PMed you :) hope you can solve this damn problem :D
Click to expand...

There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.
 
accelerator_dd said:
There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.
Click to expand...

I did the same thing.
Everything works as it should, this is a problem with the Framework not the code I think
 
healzer said:
I did the same thing.
Everything works as it should, this is a problem with the Framework not the code I think
Click to expand...

It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.
 
How about splitting it up? One function gets a new proxy, and another function makes the webrequest from the new proxy and credentials..
 
accelerator_dd said:
It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.
Click to expand...

I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top