VB.net Proxy doesn't change after first HttpWebRequest

 

Results 1 to 28 of 28
I have a public function to get the pagesource of web URLs. Here is the ...
  1. #1
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default VB.net Proxy doesn't change after first HttpWebRequest

    I have a public function to get the pagesource of web URLs. Here is the function.


    Code:
    Public Function gethtml(ByVal id As Integer)
        Try
                        ListView1.Items(id).SubItems(6).Text = "Downloading HTML..."
                        Dim Response3 As HttpWebResponse
                        Dim pagesource3 As String
                        Dim datastream As Stream
                        Dim postdata As String = ""
                        Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postdata)
                        Dim Request3 As HttpWebRequest = HttpWebRequest.Create("http://mysite.com/ip.php")
                        Request3.Referer = "http://google.com/"
                        Request3.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
                        Request3.KeepAlive = True
                        Request3.Timeout = "5000"
                        Request3.AllowAutoRedirect = True
                        If useproxycheckbox.Checked = True Then
                            Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
                            Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)
                            Request3.UseDefaultCredentials = False
                            Request3.Proxy.Credentials = New System.Net.NetworkCredential(words(2).ToString, words(3).ToString)
                        End If
                        Request3.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31"
                        Request3.CookieContainer = CC
                        Request3.Method = WebRequestMethods.Http.Get
                        Request3.ContentType = "application/x-www-form-urlencoded"
                        Response3 = Request3.GetResponse()
                        Dim Stream3 As New StreamReader(Response3.GetResponseStream())
                        pagesource3 = Stream3.ReadToEnd()
                        Response3.Close()
                        Stream3.Close()
                        ListView1.Items(id).SubItems(4).Text = pagesource3
                        
        Catch e1 As ProtocolViolationException
            recheck(id)
            ListView1.Items(id).SubItems(6).Text = "Error1"
        Catch e2 As WebException
            recheck(id)
            ListView1.Items(id).SubItems(6).Text = "Error2"
                                
        End Try
        End Function
    The problem is it works fine with first call like


    Code:
     gethtml(0)
    But after that, it doesn't use the proxy I provide to it. it simply uses the same proxy used in the 1st request. I'm calling this function using a thread because if I use this on main thread I'll freeze the UI. What I'm doing wrong? I'm a noob in VB.net


    Thanks.




  2. #2
    Naveen257's Avatar
    Naveen257 is offline Regular Member
    Join Date
    Sep 2012
    Location
    next door
    Posts
    233
    Thanks
    127
    Thanked 89 Times in 49 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I would like to know this as well. Anyone ??

  3. The Following User Says Thank You to Naveen257 For This Useful Post:

    frankweerasinghe (06-07-2013)

  4. #3
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by Naveen257 View Post
    I would like to know this as well. Anyone ??
    Thanks man!

    ANYONE???

  5. #4
    qrazy's Avatar
    qrazy is offline Jr. VIP
    Join Date
    Mar 2012
    Posts
    884
    Thanks
    981
    Thanked 1,005 Times in 457 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).

  6. The Following User Says Thank You to qrazy For This Useful Post:

    frankweerasinghe (06-07-2013)

  7. #5
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by qrazy View Post
    The code to set the proxy looks fine. Instantiating new HTTPWebRequest Object doesn't retain the previously set proxy unless it's set in the internet explorer. Make sure that you're getting the updated proxy when you call for the second time (from list view).
    Hi, Yes I'm sure I was passing the updated value to the function.. This is really wired... If I send you the project file, can you help me?

  8. #6
    hpv222 is offline Power Member
    Join Date
    Feb 2010
    Posts
    710
    Thanks
    42
    Thanked 260 Times in 183 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


    Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

    You should probably check out these two lines:

    Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
    Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


    Again, I could be way off, just trying to help

  9. The Following User Says Thank You to hpv222 For This Useful Post:

    frankweerasinghe (06-07-2013)

  10. #7
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by hpv222 View Post
    I don't know much about VB, but it looks like you are loading the proxies in words, but then you are not rotating them. How does the proxy change in your code?


    Say you have loaded ten proxies, then you can create random number b/n 0 and 9, and choose proxy[random number]

    You should probably check out these two lines:

    Dim words As String() = ListView1.Items(id).SubItems(9).Text.Split(New Char() {":"c})
    Request3.Proxy = New WebProxy(words(0).ToString & ":" & words(1).ToString)


    Again, I could be way off, just trying to help
    Hi, Thanks for the reply. In my list, I have loaded my proxies in PROXY:PORT:USERNAME:PASSWORD format and getting unique one using line number. (id). Then, I separate that string by ":" and provide them to the appropriate fields. New WebProxy class can convert string into a System.WebProxy so I can finally use it to the HTTPWebRequest.

    Yes, I have also tried the way you posted. Still no luck!!!

  11. #8
    healzer's Avatar
    healzer is offline TheProblemSolver
    Join Date
    Jun 2011
    Location
    void main() { while (1==1) { MakeMoney(); }}
    Posts
    1,722
    Thanks
    2,501
    Thanked 1,363 Times in 518 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
    Do you know how to set breakpoints/debug the program?

  12. #9
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by healzer View Post
    watch out using Threads, if you don't know what you're doing you're probably doing something wrong.
    Do you know how to set breakpoints/debug the program?

    PMed you

  13. #10
    hpv222 is offline Power Member
    Join Date
    Feb 2010
    Posts
    710
    Thanks
    42
    Thanked 260 Times in 183 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

    Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAME:PASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?

  14. #11
    healzer's Avatar
    healzer is offline TheProblemSolver
    Join Date
    Jun 2011
    Location
    void main() { while (1==1) { MakeMoney(); }}
    Posts
    1,722
    Thanks
    2,501
    Thanked 1,363 Times in 518 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by hpv222 View Post
    I seriously doubt that this has anything to do with threading, simply add a MessageBox() or whatever that is called in VB in your function and see what proxies are being loaded.

    Remove all other code if you want to, just leave the code that loads your proxy, followed by MessageBox(Proxy.ToString()) or similar. So, if the method keeps loading the same proxy, say "12.12.12.12:80:USERNAME:PASS" then you are not rotating the proxies. You say you use "id" to pick a different one, but just test your code .... how do you change "id" - are you using it in a loop?
    I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-

  15. #12
    hpv222 is offline Power Member
    Join Date
    Feb 2010
    Posts
    710
    Thanks
    42
    Thanked 260 Times in 183 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by healzer View Post
    I took a look at his coding and for some VERY VERY VERY Strange and un-understandable reason it had something to do with Threading, couldn't believe myself... can't understand why -_-
    Not going to argue with you when I don't know much about VB In C# even when you use ParallelForEach, each HTTPWebRequest can use its own proxy and Network Credentials, they are thread safe. If you have to use any method or variable that is not thread safe, then you use the lock statement.

    Well, let’s hope that the OP has figured it out by now

  16. #13
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    For some reason, I can't edit above post. I found something... This problem only happens with proxies that having basic username password authentication. It works well with normal IP:PORT proxies. Help me!

  17. #14
    accelerator_dd is offline Jr. VIP
    Join Date
    May 2010
    Location
    IM Wonderland
    Posts
    1,315
    Thanks
    536
    Thanked 557 Times in 318 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .


  18. The Following User Says Thank You to accelerator_dd For This Useful Post:

    frankweerasinghe (06-13-2013)

  19. #15
    frankweerasinghe's Avatar
    frankweerasinghe is offline Regular Member
    Join Date
    Jun 2011
    Location
    Colombo, Sri Lanka
    Posts
    415
    Thanks
    327
    Thanked 369 Times in 70 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by accelerator_dd View Post
    If you want, I can debug it for you and see if i can find the issue - originally a C# programmer, but back when I was starting I started with vb.net .

    PMed you hope you can solve this damn problem

  20. #16
    accelerator_dd is offline Jr. VIP
    Join Date
    May 2010
    Location
    IM Wonderland
    Posts
    1,315
    Thanks
    536
    Thanked 557 Times in 318 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by frankweerasinghe View Post
    PMed you hope you can solve this damn problem
    There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

    I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

    To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.


  21. #17
    healzer's Avatar
    healzer is offline TheProblemSolver
    Join Date
    Jun 2011
    Location
    void main() { while (1==1) { MakeMoney(); }}
    Posts
    1,722
    Thanks
    2,501
    Thanked 1,363 Times in 518 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by accelerator_dd View Post
    There problem isn't that the proxies aren't rotating - they are, but it's somehow caching the content. When sniffing the network, you can clearly see it connecting and authenticating to each proxy. However, for some reason, it's caching the content of the website.

    I did a test on my laptop by setting up a webserver and changing the address in your app, and it was doing the same thing, I wasn't receiving all the pings in the log, so it might be caching content.

    To see what proxy is being used, just put a MessageBox in the try sequence after assinging the proxy, to just print out the proxy used.
    I did the same thing.
    Everything works as it should, this is a problem with the Framework not the code I think

  22. #18
    accelerator_dd is offline Jr. VIP
    Join Date
    May 2010
    Location
    IM Wonderland
    Posts
    1,315
    Thanks
    536
    Thanked 557 Times in 318 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by healzer View Post
    I did the same thing.
    Everything works as it should, this is a problem with the Framework not the code I think
    It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

    I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.


  23. #19
    seorebel's Avatar
    seorebel is offline Junior Member
    Join Date
    Aug 2008
    Posts
    176
    Thanks
    127
    Thanked 921 Times in 40 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    How about splitting it up? One function gets a new proxy, and another function makes the webrequest from the new proxy and credentials..

  24. #20
    healzer's Avatar
    healzer is offline TheProblemSolver
    Join Date
    Jun 2011
    Location
    void main() { while (1==1) { MakeMoney(); }}
    Posts
    1,722
    Thanks
    2,501
    Thanked 1,363 Times in 518 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by accelerator_dd View Post
    It might be, but the fact is that it's caching something because when I started the local server, only the first request was being received although from the sniffer i could clearly see the handshake with the proxy for auth - might even be the proxies, but the code is okay.

    I also tried changing the variable value (response3) to none in the try block, at the end, just to see what would be written after the first DL is done, it was null too.
    I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak

  25. #21
    accelerator_dd is offline Jr. VIP
    Join Date
    May 2010
    Location
    IM Wonderland
    Posts
    1,315
    Thanks
    536
    Thanked 557 Times in 318 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by healzer View Post
    I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak
    If you want I'l show you and screen share, it is connecting to the proxies - after that the issue arises.

    Sent from my Transformer Prime TF201 using Tapatalk 2


  26. #22
    healzer's Avatar
    healzer is offline TheProblemSolver
    Join Date
    Jun 2011
    Location
    void main() { while (1==1) { MakeMoney(); }}
    Posts
    1,722
    Thanks
    2,501
    Thanked 1,363 Times in 518 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by accelerator_dd View Post
    If you want I'l show you and screen share, it is connecting to the proxies - after that the issue arises.

    Sent from my Transformer Prime TF201 using Tapatalk 2
    I know because I got his code as well, I know what I'm saying

  27. #23
    accelerator_dd is offline Jr. VIP
    Join Date
    May 2010
    Location
    IM Wonderland
    Posts
    1,315
    Thanks
    536
    Thanked 557 Times in 318 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Quote Originally Posted by healzer View Post
    I don't think so, because if you run each HttpRequest on a NEW Thread, then it works PERFECTLY, but when it is using the same thread it is created with, then it "caches" as you speak
    Quote Originally Posted by healzer View Post
    I know because I got his code as well, I know what I'm saying
    That's the deal - it's not opening those up okay even in separate threads.

    After testing a few things out with frankweerasinghe it turns out that it's a fault either in the protocol libs or the proxies (my money is on the protocol).

    We were getting that issue with user & pass combo - but when trying out some squidproxies on the same code, it worked out okay.

    So it was working without user/pass, but with, it was not.

    When working with userassword combo, i set up a sniffer to see if the requests are being made - they are being made.

    I changed the address to go to my local server, via those proxies, and apache was only returning the first hit out of the 5 he had coded in.


  28. #24
    gimme4free is offline Executive VIP
    Join Date
    Oct 2008
    Posts
    1,829
    Thanks
    359
    Thanked 1,862 Times in 559 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I have not ever set the proxies using the PRoxy.Credentials method, instead I set the credentials within a new WebProxy & then set the request's proxy to that, E.g.
    If Proxy <> Nothing AndAlso Proxy <> "0" AndAlso Proxy <> "" Then
    Dim myProxy As New WebProxy(Proxy, Port)
    If ProxyUser <> "0" AndAlso ProxyUser <> "" Then myProxy.Credentials = New NetworkCredential(ProxyUser, ProxyPass)
    request.Proxy = myProxy
    End If

    Never had any caching issues when using this method, you mentioned that the issue occurs when using a username/password proxy. Also when you are catching the exceptions change them to a MessageBox incase the exception is empty / not formatted for the exception you are catching so that you can see if an error occurs rather than looking over an empty listview item as if the error is occuring before you are setting the content then the content would remain the same & look like it is caching when in actual fact it has not changed at all but instead an exception occured before you get to that point.

  29. The Following 2 Users Say Thank You to gimme4free For This Useful Post:

    accelerator_dd (06-20-2013), frankweerasinghe (08-05-2013)

  30. #25
    FireSoftLimited is offline Registered Member
    Join Date
    Apr 2013
    Location
    Google UK
    Posts
    97
    Thanks
    22
    Thanked 20 Times in 11 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    PM me with link to solution, I will take a closer look.

  31. The Following User Says Thank You to FireSoftLimited For This Useful Post:

    frankweerasinghe (08-05-2013)

  32. #26
    HatIsBlack's Avatar
    HatIsBlack is offline Jr. VIP
    Join Date
    Sep 2010
    Location
    Where i belong
    Posts
    264
    Thanks
    59
    Thanked 171 Times in 65 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I could take a look if you want. If you have not solved it by now.

    The cacheing issue can come from Squid Proxy usage. When settng credentials like you do webrequest is stupid and does not pass along the credentials to every request. to bypass that issue you have to be more createive. Also if you were to use that code with many threads in the future you might suffer a underlying connection issue since you do not clean up your Connections.

    Regards,
    HatIsBlack

  33. The Following User Says Thank You to HatIsBlack For This Useful Post:

    frankweerasinghe (08-05-2013)

  34. #27
    Psychop1 is offline Registered Member
    Join Date
    Jul 2010
    Posts
    61
    Thanks
    25
    Thanked 18 Times in 15 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    Could be a lot of problems here. For one, this is running on a separate thread with obvious crossthreading violations. I would load my listview into a list or an array and cycle through that to avoid marshalling (it's unnecessary in this case). Another thing, there is what has to be a global cookie container being added to the request after proxies and credentials are loaded. Now, there shouldn't be any cookies with proxy information, but if the issue is what is getting cached from the site as mentioned above (though that doesn't make much sense to me), then what is being loaded into the cookie container could be the problem. I would address issues like these before placing blame on the protocol. My two cents anyway.

  35. The Following User Says Thank You to Psychop1 For This Useful Post:

    frankweerasinghe (08-05-2013)

  36. #28
    RtBeast is offline Newbies
    Join Date
    Aug 2013
    Posts
    12
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default Re: VB.net Proxy doesn't change after first HttpWebRequest

    I am interested please message me


Similar Threads

  1. What all VPN's doesn't get detected as proxy ?
    By ch8878 in forum BlackHat Lounge
    Replies: 2
    Last Post: 04-09-2012, 06:49 AM
  2. Replies: 5
    Last Post: 01-04-2012, 07:52 PM
  3. Proxy or change IP?
    By House99 in forum FaceBook
    Replies: 7
    Last Post: 06-08-2010, 05:28 AM
  4. Replies: 7
    Last Post: 05-12-2010, 08:33 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  




BlackHatWorld on Twitter BlackHatWorld on FaceBook


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109