Unable to login with or without cookies (HttpWebRequest)

Free6

Newbie
Joined
Nov 21, 2016
Messages
47
Reaction score
1
Hi, I captured the below login code from Fiddler and the day I captured that, I tried it into VS 2013 and worked well with its original cookies.

When I try to manually set cookies it doesn't work, the output debug shows : (A first chance exception of type 'System.Net.WebException' occurred in System.dll)
Note : I commented out the cookies I set manually. you can uncomment it and try on your side if you want.

I'm sure the website use a daily cookies which expire after day, but even if I set manually cookies it doesn't work.


Can you please help me ?


Code:
Imports System.Net
Imports System.IO
Imports System.IO.Compression
Imports System.Text



Module mod_Login2


    Public Sub MakeRequestsLogin2()

        Dim response As HttpWebResponse
        Dim responseText As String

        If Request_www_gumtree_com_au(response) Then
            responseText = ReadResponse(response)

            response.Close()

        End If

    End Sub

    Private Function ReadResponse(response As HttpWebResponse) As String
        Using responseStream = response.GetResponseStream()
            Dim streamToRead As Stream = responseStream
            If response.ContentEncoding.ToLower().Contains("gzip") Then
                streamToRead = New GZipStream(streamToRead, CompressionMode.Decompress)
            ElseIf response.ContentEncoding.ToLower().Contains("deflate") Then
                streamToRead = New DeflateStream(streamToRead, CompressionMode.Decompress)
            End If

            Using streamReader = New StreamReader(streamToRead, Encoding.UTF8)
                Return streamReader.ReadToEnd()
            End Using
        End Using
    End Function

    Private Function Request_www_gumtree_com_au(ByRef response As HttpWebResponse) As Boolean
        response = Nothing

        Try
            'Dim logincookie As CookieContainer
            Dim request As HttpWebRequest = DirectCast(WebRequest.Create("https://www.gumtree.com.au/t-login.html"), HttpWebRequest)

            'Dim tempCookies As New CookieContainer


            'request.AllowAutoRedirect = True

            request.KeepAlive = True
            request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0")
            request.Headers.Add("Origin", "https://www.gumtree.com.au")
            request.Headers.Add("Upgrade-Insecure-Requests", "1")
            request.ContentType = "application/x-www-form-urlencoded"
            request.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

            'request.CookieContainer = tempCookies

            request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
            request.Referer = "https://www.gumtree.com.au/t-login-form.html?sl=true"
            request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br")
            request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9")

            request.Headers.Set(HttpRequestHeader.Cookie, "machId=Kl7iENaJAekYqzozIQf7zj8S436pQS3CHJsL-8fYzJXsN60H-TPHXj-7AdhFzwgzr2eL9dc2RbshqvjLJl9wEFrzkYgcNxXq2_4; __gads=ID=6f9fa1887ba7a16e:T=1511856578:S=ALNI_MZ_OPw37audK8tGq1WZA6ke38GqWA; ki_r=; _ga=GA1.3.1845295369.1511856571; crtg_rta=; aam_tnt=aamsegid%3D6797281%2Caamsegid%3D6880889; aam_uuid=32615388182877496343265783929349914926; __utma=1.1845295369.1511856571.1515000768.1515000768.1; __utmz=1.1515000768.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); cto_lwid=4481ec65-e154-4f89-b4f6-cbd297610983; aam_dfp=aamsegid%3D6797281%2C6880889%2C7087286%2C8465423%2C8978946%2C8978948%2C8978957%2C8458219; AMCV_50BE5F5858D2477A0A495C7F%40AdobeOrg=2096510701%7CMCIDTS%7C17557%7CMCMID%7C33137287337103129523309459432855465684%7CMCAAMLH-1517216385%7C6%7CMCAAMB-1517562533%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1516964933s%7CNONE%7CMCSYNCSOP%7C411-17561%7CvVersion%7C2.0.0%7CMCCIDH%7C-223693937; __utmz=160852194.1516960284.63.9.utmcsr=trigger|utmccn=abandoned_SYI_flow|utmcmd=email|utmcct=main_cta; __utma=160852194.1845295369.1511856571.1516977245.1516986923.67; ki_t=1511856587832%3B1516957766713%3B1516988474571%3B25%3B406; up=%7B%22ln%22%3A%22467250100%22%2C%22ls%22%3A%22l%3D3000662%26r%3D0%26sv%3DLIST%26sf%3Ddate%22%2C%22rva%22%3A%221149502958%22%2C%22lh%22%3A%22l%3D3003795%26r%3D0%7Cl%3D3000662%26r%3D0%22%2C%22lbh%22%3A%22l%3D3003795%26c%3D20103%26r%3D0%26sv%3DLIST%26sf%3Ddate%7Cl%3D3003795%26c%3D18297%26r%3D0%26sv%3DLIST%26sf%3Ddate%22%2C%22nps%22%3A%2224%22%7D; bs=%7B%22st%22%3A%7B%7D%7D; wl=%7B%22l%22%3A%22%22%7D")



            request.Method = "POST"
            request.ServicePoint.Expect100Continue = False

            Dim body As String = "ctk=DGV-DcO0r3Xv5mYmTePH6Q&csrft=530c2c789d3b24db2717157e471e5c82&targetUrl=&likingAd=false&threatmetrixSessionId=64313464663739622d626533642d343335622d396264312d37306563303232633338336639303939383133333038343730383136323936&loginMail=alphonseantonio%40gmail.com&password=Jaenny007Laure8%2611&rememberMe=true&_rememberMe=on"

            Dim postBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(body)
            request.ContentLength = postBytes.Length
            Dim stream As Stream = request.GetRequestStream()
            stream.Write(postBytes, 0, postBytes.Length)
            stream.Close()

            response = DirectCast(request.GetResponse(), HttpWebResponse)


            'tempCookies.Add(response.Cookies)
            'logincookie = tempCookies


            Console.WriteLine("Code 1 : " & response.StatusCode & "  Description : " & response.ResponseUri.Query)

            Console.WriteLine("Code 2 : " & response.Server & "  Description : " & response.ResponseUri.Query)

            Console.WriteLine("Code 3 : " & response.StatusDescription & "  Description : " & response.ResponseUri.Query)

            Console.WriteLine("Code 4 : " & response.ResponseUri.AbsoluteUri & "  Description : " & response.ResponseUri.Query)

            Console.WriteLine("Code 5 : " & response.ResponseUri.AbsolutePath & "  Description : " & response.ResponseUri.Query)


        Catch e As WebException
            If e.Status = WebExceptionStatus.ProtocolError Then
                response = DirectCast(e.Response, HttpWebResponse)
            Else
                Return False
            End If
        Catch e As Exception
            If response IsNot Nothing Then
                response.Close()
            End If
            Return False
        End Try

        Return True
    End Function



End Module
 
Last edited:

gimme4free

Executive VIP
Joined
Oct 22, 2008
Messages
1,966
Reaction score
1,985
The issue is with the data you are posting:
ctk=DGV-DcO0r3Xv5mYmTePH6Q
csrft=530c2c789d3b24db2717157e471e5c8
threatmetrixSessionId=64313464663739622d626533642d343335622d396264312d37306563303232633338336639303939383133333038343730383136323936

These variables are likely dynamic & checked server side. So you need to load the login page, scrape these variables & then send the actual values for these fields based upon what is shown on the login page as they change each time the page is loaded.
 

naskootbg

Senior Member
Joined
Nov 8, 2010
Messages
846
Reaction score
280
Website
www.blackhatworld.com
I bet:
ctk=DGV-DcO0r3Xv5mYmTePH6Q
csrft=530c2c789d3b24db2717157e471e5c8

Are hidden tokens. You can get them from the first page where Set-Cookie. They are most possibly text type='hidden'. Sometime one is text type='hidden' and the scond is generated with javascript or with some GET request. Sometime one is MD5 of other. Sometime only one of them is enough.

BTW for such tests I'm using UniBot by Mikisoft - have link to download in the forum. It do not record header, but can easy add headers without writing a code on it and check the responces and the retuned source code. I think when unsuccessful it will return something like "wrong csrft" .
 
Last edited:

Free6

Newbie
Joined
Nov 21, 2016
Messages
47
Reaction score
1
Thank you so so much !!!

I've found the two values on the online page and used the fresh one in the app... and worked.

Thank you again guys !!!
 

Free6

Newbie
Joined
Nov 21, 2016
Messages
47
Reaction score
1
Just one more question please.

Do these two values (csrft and ctk) change daily or hourly?
Cause I want to catch them every first time the app start and store them somewhere in order to reuse them anytime but i'm not sure if one refreshs them at which frequency (daily, hourly, after 2-3 hour...)
 

gimme4free

Executive VIP
Joined
Oct 22, 2008
Messages
1,966
Reaction score
1,985
Just one more question please.

Do these two values (csrft and ctk) change daily or hourly?
Cause I want to catch them every first time the app start and store them somewhere in order to reuse them anytime but i'm not sure if one refreshs them at which frequency (daily, hourly, after 2-3 hour...)

They change every page load.
 

Free6

Newbie
Joined
Nov 21, 2016
Messages
47
Reaction score
1
Thanks gimme4free.

So if I want to catch those two values, should I go with VB.Net or DOM element or Javascript as VS also can't be mix with javascript.

What is the best option to go for?
 

naskootbg

Senior Member
Joined
Nov 8, 2010
Messages
846
Reaction score
280
Website
www.blackhatworld.com
Just one more question please.

Do these two values (csrft and ctk) change daily or hourly?
Cause I want to catch them every first time the app start and store them somewhere in order to reuse them anytime but i'm not sure if one refreshs them at which frequency (daily, hourly, after 2-3 hour...)

I think they expire together with the session. Else new come on each page reload (HTTP request) while out of session. They are most regular existing only on the login page before the POST request.

You can find them hidden on the login page. After this post they are going session (cookie: ...sessionid...token...token).

Get them (tokens) from the source code of the login page with simple regex on any language. Hold them and once loged in, use them on the cookies header. They will not change untill session ends.

What is the problem to start from login? Once you have session and the bot running you can use this session to surf inside very long time. Stop the bot and the session will end.
The session in browser is not server side.
 
Last edited:

gimme4free

Executive VIP
Joined
Oct 22, 2008
Messages
1,966
Reaction score
1,985
Thanks gimme4free.

So if I want to catch those two values, should I go with VB.Net or DOM element or Javascript as VS also can't be mix with javascript.

What is the best option to go for?

You should be loading the page with your same cookies & method as you are using to post to the login process. You should use a class for managing your requests to make this process smooth rather than posting the full code every time. Research more into OOP to understand how the code should be written.

You are attempting to simulate a real user so you must load the pages that the browser does & copy the process with the same cookies, useragent & data etc.
 
Top