1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HttpWebRequest failed to login to public website

Discussion in 'Visual Basic .NET' started by Free6, Dec 1, 2016.

  1. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    Hi eveybody, Please I need help.

    I'm using HttpWebRequest in VB.Net.

    I'm trying to login onto a website using the below code, but I'm still unable to login.
    Please see code below and the Live Http Headers' capture.

    Code:
    Dim logincookie As CookieContainer
    
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    
            WebBrowser1.DocumentText = RichTextBox1.Text
    
        End Sub
    
    
        Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
    
            Dim postData As String = "redirect=https%3A%2F%2Fwww.gumtree.co.za%2F&email=kingseven%40gmail.com&password=KaL8n0&rememberMe=on"
            Dim tempCookies As New CookieContainer
            Dim encoding As New UTF8Encoding
            Dim byteData As Byte() = encoding.GetBytes(postData)
    
            Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.gumtree.co.za/login.html"), HttpWebRequest)
            postReq.Method = "POST"
            postReq.KeepAlive = True
            postReq.CookieContainer = tempCookies
            postReq.ContentType = "application/x-www-form-urlencoded"
            postReq.Referer = "https://www.gumtree.co.za/login.html"
            postReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
            postReq.ContentLength = byteData.Length
            'ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3
    
    
            Dim postreqstream As Stream = postReq.GetRequestStream()
            postreqstream.Write(byteData, 0, byteData.Length)
            postreqstream.Close()
    
            Dim postresponse As HttpWebResponse
            postresponse = DirectCast(postReq.GetResponse(), HttpWebResponse)
            tempCookies.Add(postresponse.Cookies)
            logincookie = tempCookies
    
            Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
    
            Dim thepage As String = postreqreader.ReadToEnd
    
            RichTextBox1.Text = thepage
    
        End Sub

    Live HTTP Headers' Capture for POST

    My email is [email protected], you can find it in the "Redirect" section I set in green color.
    I don't know where is the issue.

    Code:
    https://www.gumtree.co.za/login
    
    POST /login HTTP/1.1
    Host: www.gumtree.co.za
    User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:49.0) Gecko/20100101 Firefox/49.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate, br
    Referer: https://www.gumtree.co.za/login.html
    Cookie: machguid=6be49057-8b0b-41b4-8105-443eff9632cd-158ba77e1e6; anonUsrId=ciw6dbqw60o2jmyav0obc5ldh; __utma=29192163.1009344549.1480597288.1480597288.1480597288.1; __utmb=29192163.2.10.1480597288; __utmc=29192163; __utmz=29192163.1480597288.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_siteTracker=1; _ga=GA1.3.1009344549.1480597288; _gat=1; __gads=ID=59e5ca9d5c31366d:T=1480597175:S=ALNI_MausWaJ3qk5yCS5tERlR-r7_yk8KQ; _em_vt=07fe88ba5762f342e1de666de8a658401eb8a1ae56-4902422358401ebf; _em_v=e1321da24531ad0dcca914a5415758401eb8a1ae69-6240733058401ebf; testCookie=true; _gali=login-button
    Connection: keep-alive
    Upgrade-Insecure-Requests: 1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 105
    redirect=https%3A%2F%2Fwww.gumtree.co.za%2F&email=kingseven%40gmail.com&password=KaL8n0&rememberMe=on
    HTTP/1.1 302 Moved Temporarily
    Location: https://www.gumtree.co.za/my/ads.html
    Content-Encoding: gzip
    Server: Apache Tomcat
    X-Cluster: web
    Content-Length: 20
    Date: Thu, 01 Dec 2016 13:01:34 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    Set-Cookie: rememberMe=ZnJlZXNpeDAwN0BnbWFpbC5jb206MTQ4MzE4OTI5NDA2MzpjNzQ5YjE5OTNjZTBkZDQxMDI2YWQzZjAzMDdjNmEwYQ; Expires=Sat, 31-Dec-2016 13:01:34 GMT; Path=/; HttpOnly
    Set-Cookie: bt_auth=rO0ABXNyAD1vcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUuY29udGV4dC5TZWN1cml0eUNvbnRleHRJbXBsAAAAAAAAAZACAAFMAA5hdXRoZW50aWNhdGlvbnQAMkxvcmcvc3ByaW5nZnJhbWV3b3JrL3NlY3VyaXR5L2NvcmUvQXV0aGVudGljYXRpb247eHBzcgBPb3JnLnNwcmluZ2ZyYW1ld29yay5zZWN1cml0eS5hdXRoZW50aWNhdGlvbi5Vc2VybmFtZVBhc3N3b3JkQXV0aGVudGljYXRpb25Ub2tlbgAAAAAAAAGQAgACTAALY3JlZGVudGlhbHN0ABJMamF2YS9sYW5nL09iamVjdDtMAAlwcmluY2lwYWxxAH4ABHhyAEdvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmF1dGhlbnRpY2F0aW9uLkFic3RyYWN0QXV0aGVudGljYXRpb25Ub2tlbtOqKH5uR2QOAgADWgANYXV0aGVudGljYXRlZEwAC2F1dGhvcml0aWVzdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO0wAB2RldGFpbHNxAH4ABHhwAXNyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHQAEExqYXZhL3V0aWwvTGlzdDt4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWNxAH4ABnhwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAB3BAAAAAB4cQB+AA1zcgBIb3JnLnNwcmluZ2ZyYW1ld29yay5zZWN1cml0eS53ZWIuYXV0aGVudGljYXRpb24uV2ViQXV0aGVudGljYXRpb25EZXRhaWxzAAAAAAAAAZACAAJMAA1yZW1vdGVBZGRyZXNzdAASTGphdmEvbGFuZy9TdHJpbmc7TAAJc2Vzc2lvbklkcQB+AA94cHQADTEwLjMyLjI0NC4xMzFwcHNyAEFjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuY29tbW9uLnNlY3VyaXR5LkJvbHRVc2VyRGV0YWlscwvyYa0BfSwTAgACTAALZGlzcGxheU5hbWVxAH4AD0wACnVzZXJDb29raWV0AD5MY29tL2ViYXkvZWNnL2JvbHQvcGxhdGZvcm0vc2hhcmVkL2NvbW1vbi9zZWN1cml0eS9Vc2VyQ29va2llO3hyADJvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUudXNlcmRldGFpbHMuVXNlcgAAAAAAAAGQAgAHWgARYWNjb3VudE5vbkV4cGlyZWRaABBhY2NvdW50Tm9uTG9ja2VkWgAVY3JlZGVudGlhbHNOb25FeHBpcmVkWgAHZW5hYmxlZEwAC2F1dGhvcml0aWVzdAAPTGphdmEvdXRpbC9TZXQ7TAAIcGFzc3dvcmRxAH4AD0wACHVzZXJuYW1lcQB+AA94cAEBAQFzcgAlamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZVNldIAdktGPm4BVAgAAeHEAfgAKc3IAEWphdmEudXRpbC5UcmVlU2V03ZhQk5Xth1sDAAB4cHNyAEZvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUudXNlcmRldGFpbHMuVXNlciRBdXRob3JpdHlDb21wYXJhdG9yAAAAAAAAAZACAAB4cHcEAAAAAHhwdAAUZnJlZXNpeDAwN0BnbWFpbC5jb210AAdGcmVlc2l4c3IAPGNvbS5lYmF5LmVjZy5ib2x0LnBsYXRmb3JtLnNoYXJlZC5jb21tb24uc2VjdXJpdHkuVXNlckNvb2tpZQAAAAAAAAABAgANWgAabG9nZ2VkSW5Vc2luZ1NvY2lhbFNlcnZpY2VaAApyZWdpc3RlcmVkTAAOZG91YmxlUG9zdGFibGV0ABNMamF2YS9sYW5nL0Jvb2xlYW47TAAFZW1haWxxAH4AD0wAE2ZhY2Vib29rQWNjZXNzVG9rZW5xAH4AD0wACmZhY2Vib29rSWRxAH4AD0wACWZpcnN0TmFtZXEAfgAPTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc3RhdGV0ADlMY29tL2ViYXkvZWNnL2JvbHQvcGxhdGZvcm0vc2hhcmVkL2VudGl0eS91c2VyL1VzZXJTdGF0ZTtMAAl0aW1lc3RhbXBxAH4AIUwABnVzZXJJZHQANkxjb20vZWJheS9lY2cvYm9sdC9wbGF0Zm9ybS9zaGFyZWQvZW50aXR5L3VzZXIvVXNlcklkO0wAE3VzZXJQcm9maWxlSW1hZ2VVcmxxAH4AD0wACHVzZXJuYW1lcQB+AA94cAABc3IAEWphdmEubGFuZy5Cb29sZWFuzSBygNWc+u4CAAFaAAV2YWx1ZXhwAHEAfgAddAAAcHEAfgAnc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAGISI0cHNxAH4AKAAAAVi6ecrvc3IANGNvbS5lYmF5LmVjZy5ib2x0LnBsYXRmb3JtLnNoYXJlZC5lbnRpdHkudXNlci5Vc2VySWQAAAAAAAAAAQIAAHhyADZjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuZW50aXR5LkVudGl0eUlkTG9uZzEAAAAAAAAAAQIAAHhyADxjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuZW50aXR5LkVudGl0eUlkQ29tcGFyYWJsZTEAAAAAAAAAAQIAAkwAB2VuY29kZWRxAH4AD0wAEnByaW1hcnlBbmRTaGFyZEtleXQAFkxqYXZhL2xhbmcvQ29tcGFyYWJsZTt4cHQADjEwMTEwMjgzNDc0MDA5cQB+ACpxAH4AJ3EAfgAe__v1__B21E73B2E87D1874D4EA; Domain=.gumtree.co.za; Path=/; HttpOnly
    Set-Cookie: bt_logged_in=true; Domain=.gumtree.co.za; Path=/
    ----------------------------------------------------------
    https://www.gumtree.co.za/my/ads.html
     
  2. phatzilla

    phatzilla Jr. VIP Jr. VIP

    Joined:
    Apr 9, 2009
    Messages:
    1,384
    Likes Received:
    1,023
  3. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    Thanks for your reply.

    Yes, I was sending the POST to the (html) and I changed it to the one you suggested but still not working.

    I'd like to ask, in the code; I have two place where there is that URL, please see below,
    the first line and the postReq.Referer
    these are the two line where you find that URL .

    I tried to change both lines (places) and I've got an exception. Please see file (picture) I attached.

    can I really change to both of them?

    Code:
    Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.gumtree.co.za/login.html"), HttpWebRequest)
            postReq.Method = "POST"
            postReq.KeepAlive = True
            postReq.CookieContainer = tempCookies
            postReq.ContentType = "application/x-www-form-urlencoded"
            postReq.Referer = "https://www.gumtree.co.za/login.html"
            postReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
            postReq.ContentLength = byteData.Length
     

    Attached Files:

  4. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,928
    Likes Received:
    1,944
    Home Page:
    405 Not Allowed = That is what Gumtree is showing as an error when you are trying to make this request. Run something like CharlesProxy - Run the process manually & record the data & then run your program & record the data & compare the two to check for differences.

    If cookies are being set beforehand, E.g. the testcookie, then you will also need to load the login URL before posting to it, if that is the same URL that you do manually. Basically copy the manual process, don't just expect the POST to the login URL to be everything that the backend server is checking for.
     
  5. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    Hi, I downloaded Charles I don't think I understand it completely.

    How do I run the process manually?
    and How do I record the Data?


    Please can you help me???
     
  6. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,928
    Likes Received:
    1,944
    Home Page:
    Run it manually = do it in your browser. Run CharlesProxy & record the data as if you was doing it for real. Then re-create in VB.NET, run CharlesProxy at the same time & compare to make sure the data being processed is the same / check where you are hitting issues & check the HTML content for error messages.
     
  7. qrazy

    qrazy Senior Member

    Joined:
    Mar 19, 2012
    Messages:
    1,115
    Likes Received:
    1,725
    Location:
    Banana Republic
    Just use Fiddler to intercept the requests from your browser and IIRC there's an add-on in fiddler where it would give you the exact code for the requests sent. You can use that code to test it in VB and change those dynamic part with the variables later to suit your needs.

    [​IMG]
     
  8. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    @ gimme4free

    Thanks for your prompt reply.

    But How do I do in order to notify you that I posted something cause I posted yesterday and I don't think you were aware of that.
     
  9. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    @ qrazy

    Thanks very much !!!

    Let me try it first and will let you know about the result.

    Please don't vanish cause I'll still need you.

    So How do I contact you in order to reply because sometimes it seems like the Forum doesn't notify people that someone has replied.

    By the way, which extension should I use as I see many extensions on this add-ons http://www.telerik.com/fiddler/add-ons
     
  10. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    @ qrazy

    I downloaded and installed it.

    it works great but how do I use it?

    because the first thing i need to do according to user (gimme4free) is to capture the login access to the web site?

    How do I do that practically? Can you please walk me through that?
     
  11. qrazy

    qrazy Senior Member

    Joined:
    Mar 19, 2012
    Messages:
    1,115
    Likes Received:
    1,725
    Location:
    Banana Republic
    This is the add-on you need. Here are the steps you'd need.

    1. Install fiddler.
    2. Install the add-on.
    3. Export the Fiddler certificate and import it in your browser.
    4. Set the proxy in your browser to send the requests thru fiddler.
    5. Start fiddler and find the request that you're interested.
    6. Get the code for that request, by drag n drop in that add-on, and use the code in your project.

    You can find the detailed explanation for each of the step here.
     
  12. Free6

    Free6 Newbie

    Joined:
    Nov 21, 2016
    Messages:
    31
    Likes Received:
    0
    Gender:
    Female
    Thanks so so much user qrazy for your suggestion.
    I finally did it, I mean I retrieved the code from Fiddler and pasted it into VB and it works perfect !!!

    I have a small concern :

    How do I do to see the result into a real Web Page ( webBrowser Control )
    because in the https://www.youtube.com/watch?v=UV5_b5oaUQk, the guy insert the WebBrowser control in order to see, he used the below code.

    Code:
    Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
    
            Dim thepage As String = postreqreader.ReadToEnd
    
            RichTextBox1.Text = thepage
    and

    Code:
     WebBrowser1.DocumentText = RichTextBox1.Text