HttpWebRequest failed to login to public website

[Free6]

Hi eveybody, Please I need help.

I'm using HttpWebRequest in VB.Net.

I'm trying to login onto a website using the below code, but I'm still unable to login.
Please see code below and the Live Http Headers' capture.

Dim logincookie As CookieContainer

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        WebBrowser1.DocumentText = RichTextBox1.Text

    End Sub


    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click

        Dim postData As String = "redirect=https%3A%2F%2Fwww.gumtree.co.za%2F&email=kingseven%40gmail.com&password=KaL8n0&rememberMe=on"
        Dim tempCookies As New CookieContainer
        Dim encoding As New UTF8Encoding
        Dim byteData As Byte() = encoding.GetBytes(postData)

        Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.gumtree.co.za/login.html"), HttpWebRequest)
        postReq.Method = "POST"
        postReq.KeepAlive = True
        postReq.CookieContainer = tempCookies
        postReq.ContentType = "application/x-www-form-urlencoded"
        postReq.Referer = "https://www.gumtree.co.za/login.html"
        postReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
        postReq.ContentLength = byteData.Length
        'ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3


        Dim postreqstream As Stream = postReq.GetRequestStream()
        postreqstream.Write(byteData, 0, byteData.Length)
        postreqstream.Close()

        Dim postresponse As HttpWebResponse
        postresponse = DirectCast(postReq.GetResponse(), HttpWebResponse)
        tempCookies.Add(postresponse.Cookies)
        logincookie = tempCookies

        Dim postreqreader As New StreamReader(postresponse.GetResponseStream())

        Dim thepage As String = postreqreader.ReadToEnd

        RichTextBox1.Text = thepage

    End Sub

Live HTTP Headers' Capture for POST

My email is [email protected], you can find it in the "Redirect" section I set in green color.
I don't know where is the issue.

https://www.gumtree.co.za/login

POST /login HTTP/1.1
Host: www.gumtree.co.za
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:49.0) Gecko/20100101 Firefox/49.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.gumtree.co.za/login.html
Cookie: machguid=6be49057-8b0b-41b4-8105-443eff9632cd-158ba77e1e6; anonUsrId=ciw6dbqw60o2jmyav0obc5ldh; __utma=29192163.1009344549.1480597288.1480597288.1480597288.1; __utmb=29192163.2.10.1480597288; __utmc=29192163; __utmz=29192163.1480597288.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_siteTracker=1; _ga=GA1.3.1009344549.1480597288; _gat=1; __gads=ID=59e5ca9d5c31366d:T=1480597175:S=ALNI_MausWaJ3qk5yCS5tERlR-r7_yk8KQ; _em_vt=07fe88ba5762f342e1de666de8a658401eb8a1ae56-4902422358401ebf; _em_v=e1321da24531ad0dcca914a5415758401eb8a1ae69-6240733058401ebf; testCookie=true; _gali=login-button
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
Content-Length: 105
redirect=https%3A%2F%2Fwww.gumtree.co.za%2F&email=kingseven%40gmail.com&password=KaL8n0&rememberMe=on
HTTP/1.1 302 Moved Temporarily
Location: https://www.gumtree.co.za/my/ads.html
Content-Encoding: gzip
Server: Apache Tomcat
X-Cluster: web
Content-Length: 20
Date: Thu, 01 Dec 2016 13:01:34 GMT
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: rememberMe=ZnJlZXNpeDAwN0BnbWFpbC5jb206MTQ4MzE4OTI5NDA2MzpjNzQ5YjE5OTNjZTBkZDQxMDI2YWQzZjAzMDdjNmEwYQ; Expires=Sat, 31-Dec-2016 13:01:34 GMT; Path=/; HttpOnly
Set-Cookie: bt_auth=rO0ABXNyAD1vcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUuY29udGV4dC5TZWN1cml0eUNvbnRleHRJbXBsAAAAAAAAAZACAAFMAA5hdXRoZW50aWNhdGlvbnQAMkxvcmcvc3ByaW5nZnJhbWV3b3JrL3NlY3VyaXR5L2NvcmUvQXV0aGVudGljYXRpb247eHBzcgBPb3JnLnNwcmluZ2ZyYW1ld29yay5zZWN1cml0eS5hdXRoZW50aWNhdGlvbi5Vc2VybmFtZVBhc3N3b3JkQXV0aGVudGljYXRpb25Ub2tlbgAAAAAAAAGQAgACTAALY3JlZGVudGlhbHN0ABJMamF2YS9sYW5nL09iamVjdDtMAAlwcmluY2lwYWxxAH4ABHhyAEdvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmF1dGhlbnRpY2F0aW9uLkFic3RyYWN0QXV0aGVudGljYXRpb25Ub2tlbtOqKH5uR2QOAgADWgANYXV0aGVudGljYXRlZEwAC2F1dGhvcml0aWVzdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO0wAB2RldGFpbHNxAH4ABHhwAXNyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHQAEExqYXZhL3V0aWwvTGlzdDt4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWNxAH4ABnhwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAB3BAAAAAB4cQB+AA1zcgBIb3JnLnNwcmluZ2ZyYW1ld29yay5zZWN1cml0eS53ZWIuYXV0aGVudGljYXRpb24uV2ViQXV0aGVudGljYXRpb25EZXRhaWxzAAAAAAAAAZACAAJMAA1yZW1vdGVBZGRyZXNzdAASTGphdmEvbGFuZy9TdHJpbmc7TAAJc2Vzc2lvbklkcQB+AA94cHQADTEwLjMyLjI0NC4xMzFwcHNyAEFjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuY29tbW9uLnNlY3VyaXR5LkJvbHRVc2VyRGV0YWlscwvyYa0BfSwTAgACTAALZGlzcGxheU5hbWVxAH4AD0wACnVzZXJDb29raWV0AD5MY29tL2ViYXkvZWNnL2JvbHQvcGxhdGZvcm0vc2hhcmVkL2NvbW1vbi9zZWN1cml0eS9Vc2VyQ29va2llO3hyADJvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUudXNlcmRldGFpbHMuVXNlcgAAAAAAAAGQAgAHWgARYWNjb3VudE5vbkV4cGlyZWRaABBhY2NvdW50Tm9uTG9ja2VkWgAVY3JlZGVudGlhbHNOb25FeHBpcmVkWgAHZW5hYmxlZEwAC2F1dGhvcml0aWVzdAAPTGphdmEvdXRpbC9TZXQ7TAAIcGFzc3dvcmRxAH4AD0wACHVzZXJuYW1lcQB+AA94cAEBAQFzcgAlamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZVNldIAdktGPm4BVAgAAeHEAfgAKc3IAEWphdmEudXRpbC5UcmVlU2V03ZhQk5Xth1sDAAB4cHNyAEZvcmcuc3ByaW5nZnJhbWV3b3JrLnNlY3VyaXR5LmNvcmUudXNlcmRldGFpbHMuVXNlciRBdXRob3JpdHlDb21wYXJhdG9yAAAAAAAAAZACAAB4cHcEAAAAAHhwdAAUZnJlZXNpeDAwN0BnbWFpbC5jb210AAdGcmVlc2l4c3IAPGNvbS5lYmF5LmVjZy5ib2x0LnBsYXRmb3JtLnNoYXJlZC5jb21tb24uc2VjdXJpdHkuVXNlckNvb2tpZQAAAAAAAAABAgANWgAabG9nZ2VkSW5Vc2luZ1NvY2lhbFNlcnZpY2VaAApyZWdpc3RlcmVkTAAOZG91YmxlUG9zdGFibGV0ABNMamF2YS9sYW5nL0Jvb2xlYW47TAAFZW1haWxxAH4AD0wAE2ZhY2Vib29rQWNjZXNzVG9rZW5xAH4AD0wACmZhY2Vib29rSWRxAH4AD0wACWZpcnN0TmFtZXEAfgAPTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc3RhdGV0ADlMY29tL2ViYXkvZWNnL2JvbHQvcGxhdGZvcm0vc2hhcmVkL2VudGl0eS91c2VyL1VzZXJTdGF0ZTtMAAl0aW1lc3RhbXBxAH4AIUwABnVzZXJJZHQANkxjb20vZWJheS9lY2cvYm9sdC9wbGF0Zm9ybS9zaGFyZWQvZW50aXR5L3VzZXIvVXNlcklkO0wAE3VzZXJQcm9maWxlSW1hZ2VVcmxxAH4AD0wACHVzZXJuYW1lcQB+AA94cAABc3IAEWphdmEubGFuZy5Cb29sZWFuzSBygNWc+u4CAAFaAAV2YWx1ZXhwAHEAfgAddAAAcHEAfgAnc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAGISI0cHNxAH4AKAAAAVi6ecrvc3IANGNvbS5lYmF5LmVjZy5ib2x0LnBsYXRmb3JtLnNoYXJlZC5lbnRpdHkudXNlci5Vc2VySWQAAAAAAAAAAQIAAHhyADZjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuZW50aXR5LkVudGl0eUlkTG9uZzEAAAAAAAAAAQIAAHhyADxjb20uZWJheS5lY2cuYm9sdC5wbGF0Zm9ybS5zaGFyZWQuZW50aXR5LkVudGl0eUlkQ29tcGFyYWJsZTEAAAAAAAAAAQIAAkwAB2VuY29kZWRxAH4AD0wAEnByaW1hcnlBbmRTaGFyZEtleXQAFkxqYXZhL2xhbmcvQ29tcGFyYWJsZTt4cHQADjEwMTEwMjgzNDc0MDA5cQB+ACpxAH4AJ3EAfgAe__v1__B21E73B2E87D1874D4EA; Domain=.gumtree.co.za; Path=/; HttpOnly
Set-Cookie: bt_logged_in=true; Domain=.gumtree.co.za; Path=/
----------------------------------------------------------
https://www.gumtree.co.za/my/ads.html

 

[phatzilla]

Are you sending the POST to https://www.gumtree.co.za/login.html?

Try

https://www.gumtree.co.za/login

 

[Free6]

Thanks for your reply.

Yes, I was sending the POST to the (html) and I changed it to the one you suggested but still not working.

I'd like to ask, in the code; I have two place where there is that URL, please see below,
the first line and the postReq.Referer
these are the two line where you find that URL .

I tried to change both lines (places) and I've got an exception. Please see file (picture) I attached.

can I really change to both of them?

Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.gumtree.co.za/login.html"), HttpWebRequest)
        postReq.Method = "POST"
        postReq.KeepAlive = True
        postReq.CookieContainer = tempCookies
        postReq.ContentType = "application/x-www-form-urlencoded"
        postReq.Referer = "https://www.gumtree.co.za/login.html"
        postReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
        postReq.ContentLength = byteData.Length

 

[gimme4free]

Thanks for your reply.

Yes, I was sending the POST to the (html) and I changed it to the one you suggested but still not working.

I'd like to ask, in the code; I have two place where there is that URL, please see below,
the first line and the postReq.Referer
these are the two line where you find that URL .

I tried to change both lines (places) and I've got an exception. Please see file (picture) I attached.

can I really change to both of them?

405 Not Allowed = That is what Gumtree is showing as an error when you are trying to make this request. Run something like CharlesProxy - Run the process manually & record the data & then run your program & record the data & compare the two to check for differences.

If cookies are being set beforehand, E.g. the testcookie, then you will also need to load the login URL before posting to it, if that is the same URL that you do manually. Basically copy the manual process, don't just expect the POST to the login URL to be everything that the backend server is checking for.

 

[Free6]

Hi, I downloaded Charles I don't think I understand it completely.

How do I run the process manually?
and How do I record the Data?


Please can you help me???

 

[gimme4free]

Hi, I downloaded Charles I don't think I understand it completely.

How do I run the process manually?
and How do I record the Data?


Please can you help me???

Run it manually = do it in your browser. Run CharlesProxy & record the data as if you was doing it for real. Then re-create in VB.NET, run CharlesProxy at the same time & compare to make sure the data being processed is the same / check where you are hitting issues & check the HTML content for error messages.

 

[qrazy]

Just use Fiddler to intercept the requests from your browser and IIRC there's an add-on in fiddler where it would give you the exact code for the requests sent. You can use that code to test it in VB and change those dynamic part with the variables later to suit your needs.

 

[Free6]

@ gimme4free

Thanks for your prompt reply.

But How do I do in order to notify you that I posted something cause I posted yesterday and I don't think you were aware of that.

 

[Free6]

@ qrazy

Thanks very much !!!

Let me try it first and will let you know about the result.

Please don't vanish cause I'll still need you.

So How do I contact you in order to reply because sometimes it seems like the Forum doesn't notify people that someone has replied.

By the way, which extension should I use as I see many extensions on this add-ons http://www.telerik.com/fiddler/add-ons

 

[Free6]

@ qrazy

I downloaded and installed it.

it works great but how do I use it?

because the first thing i need to do according to user (gimme4free) is to capture the login access to the web site?

How do I do that practically? Can you please walk me through that?

 

[qrazy]

This is the add-on you need. Here are the steps you'd need.

1. Install fiddler.
2. Install the add-on.
3. Export the Fiddler certificate and import it in your browser.
4. Set the proxy in your browser to send the requests thru fiddler.
5. Start fiddler and find the request that you're interested.
6. Get the code for that request, by drag n drop in that add-on, and use the code in your project.

You can find the detailed explanation for each of the step here.

 

[Free6]

Thanks so so much user qrazy for your suggestion.
I finally did it, I mean I retrieved the code from Fiddler and pasted it into VB and it works perfect !!!

I have a small concern :

How do I do to see the result into a real Web Page ( webBrowser Control )
because in the https://www.youtube.com/watch?v=UV5_b5oaUQk, the guy insert the WebBrowser control in order to see, he used the below code.

Dim postreqreader As New StreamReader(postresponse.GetResponseStream())

        Dim thepage As String = postreqreader.ReadToEnd

        RichTextBox1.Text = thepage

and

 WebBrowser1.DocumentText = RichTextBox1.Text