1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

For some reason HTTP POST is not working for few sites.

Discussion in 'Visual Basic .NET' started by carmamir, Apr 24, 2014.

  1. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    Code:
     Dim URL As String = "http://wallinside.com/createBlog.php"
    
         
                Dim request As HttpWebRequest
                Dim response As HttpWebResponse = Nothing
                Dim reader As StreamReader
                Dim address2 As Uri
                Dim Postdata As StringBuilder
                Dim byteData() As Byte
    
    
                ' System.Net.ServicePointManager.Expect100Continue = False
    
    
    
    
                Dim postStream As Stream = Nothing
    
    
                address2 = New Uri(URL)
    
    
                ' Create the web request  
                request = DirectCast(WebRequest.Create(address2), HttpWebRequest)
                request.ContentType = "application/x-www-form-urlencoded"
    
    
                request.ServicePoint.Expect100Continue = False
              
                request.Host = "wallinside.com"
                     request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
                request.Headers.Set("Origin", "http://wallinside.com")
                request.UserAgent = USERAGENT
                     request.Referer = "http://wallinside.com/"
                request.SendChunked = True
                request.Headers.Set("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,pl;q=0.4")
                request.Headers.Set("Accept-Encoding", "gzip,deflate,sdch")
                     
                request.Headers.Set("Cookie", RegexMatch)
                request.MaximumAutomaticRedirections = 10
                request.AllowAutoRedirect = True
    
    
    
    
                request.Headers.Set("Cache-Control", "max-age=0")
    
    
          
                request.Method = "POST"


    RESULT:
    Code:
    POST http://wallinside.com/createBlog.php HTTP/1.1Content-Type: application/x-www-form-urlencoded
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
    Origin: http://wallinside.com
    User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1
    Referer: http://wallinside.com/
    Accept-Language: en-GB,en-US;q=0.8,en;q=0.6,pl;q=0.4
    Accept-Encoding: gzip,deflate,sdch
    Cookie: PHPSESSID=c53b55216c40f1cfb0cbf07a139f9025; acopendivids=t3,t4,menu; acgroupswithpersist=nada
    Cache-Control: max-age=0
    Host: wallinside.com
    Connection: Keep-Alive
    Content-Length: 118
    
    
    login=12f12faefg%40gsdgsdg&pass1=ko7fLy5pt&ewall=d21d2ff&lg=en&pass2=&nowtime=1398340528&Crear=Sign+Up.+It%27s+free%21

    And here is what browser sent:

    Code:
    POST http://wallinside.com/createBlog.php HTTP/1.1
    Host: wallinside.com
    Connection: keep-alive
    Content-Length: 126
    Cache-Control: max-age=0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
    Origin: http://wallinside.com
    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36
    Content-Type: application/x-www-form-urlencoded
    Referer: http://wallinside.com/
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: en-GB,en-US;q=0.8,en;q=0.6,pl;q=0.4
    Cookie: PHPSESSID=739eb07a7f39261efd382a60a0a256f3; acopendivids=t3,t4,menu; acgroupswithpersist=nada
    
    
    login=f3ef23fdgs%40gsdgsdg.com&pass1=8g89sgqg&ewall=asfa7sfgas7f&lg=en&pass2=&nowtime=1398339894&Crear=Sign+Up.+It%27s+free%21
    I don't think it is cookie as few times I just had PHPSESSID=*****

    Any one ?

    Not only site with same issue. POST looks same, but server some how refuse to register :/ (I tried 4 or 5 which are working for 100% with programmatic registration (like 43things)

    Fighting for a week and I can't solve it :/

    Servers won't register my users :(

    VS2010 + .NET 4.0 Full
     
    Last edited: Apr 24, 2014
  2. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,059
    Can I see the response you get from your code and the response that it sends to your browser?
     
    • Thanks Thanks x 1
  3. auxiliarus

    auxiliarus Regular Member

    Joined:
    Aug 4, 2013
    Messages:
    466
    Likes Received:
    232
    Location:
    Russian Federation.
    You might want to remove your pass from the logs...
     
  4. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    Here is HTTPWebRequest answer
    Code:
    HTTP/1.1 200 OKDate: Thu, 24 Apr 2014 11:17:13 GMT
    Server: Apache/2.2.20 (Unix) mod_ssl/2.2.20 OpenSSL/0.9.8o
    X-Powered-By: PHP/5.2.13-pl1-gentoo
    Cache-Control: max-age=2592000
    Expires: Sat, 24 May 2014 11:17:13 GMT
    Vary: Accept-Encoding,User-Agent
    Content-Length: 39
    Content-Type: text/html
    
    
    <script>window.history.back();</script>

    Here browser:
    Code:
    HTTP/1.1 302 Moved TemporarilyDate: Thu, 24 Apr 2014 11:45:13 GMT
    Server: Apache/2.2.20 (Unix) mod_ssl/2.2.20 OpenSSL/0.9.8o
    X-Powered-By: PHP/5.2.13-pl1-gentoo
    Location: comprueba_en_2.php?login=f3ef23fdgs@gsdgsdg.com&pass=8g89sgqg
    Cache-Control: max-age=2592000
    Expires: Sat, 24 May 2014 11:45:13 GMT
    Vary: Accept-Encoding,User-Agent
    Content-Length: 0
    Keep-Alive: timeout=5, max=100
    Connection: Keep-Alive
    Content-Type: text/html
    
    
    
    Fore some freaking reason it finds that POST is incorrect.
     
    Last edited: Apr 24, 2014
  5. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    It's just random pass.
     
  6. Chris22

    Chris22 Regular Member

    Joined:
    Sep 29, 2010
    Messages:
    400
    Likes Received:
    1,059
    Are you sure the post parameters you are posting are valid?

    Where are you getting your Cookie information from?
     
    • Thanks Thanks x 1
  7. jamesvick

    jamesvick Senior Member

    Joined:
    Jul 26, 2010
    Messages:
    968
    Likes Received:
    653
    Location:
    article directories
    Home Page:
    1) make sure you store cookies in a cookiecontainer
    2) Make sure to check if it's a multi-part post or a single part
     
    • Thanks Thanks x 1
  8. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    http://wallinside.com/

    Time is from form, rest is just email/pass/name.

    Pretty sure that all values are valid.

    For some reason it doesn't work (not only one I have more like this website, some weird responds for no obvious reason).

    I were thinking if maybe I should GET js/css/images with cookie as server maybe checks for it ? (as I have no idea why my POST doesn't get redirect to proper URL)

    cookies are set from string.
     
  9. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    1. I have big issue with cookiecontainer. It doesn't pass cookies (sometimes)(sent with blank cookies), so I start using regex to extract cookies -> string -> set new cookies for POST.
    Before I used:
    resp cookies -> logincookies -> req.cookies
    But often not working (I have no idea why as it should, same domain, same URL)

    2) How ? (I'm using fiddler and I can see only 1 POST)
     
    Last edited: Apr 24, 2014
  10. xxf8xx

    xxf8xx Supreme Member

    Joined:
    Nov 30, 2009
    Messages:
    1,321
    Likes Received:
    596
    Occupation:
    IM
    Alright OP here you go. Here's what I think is wrong:

    1. Your headers look incorrect as I have some different ones. Make sure to get them while browsing in private mode.
    2. You need to go to the homepage and grab the cookies in order to get the SESSIONID.
    3. You have a static timestamp (I'm assuming). While this may not matter, it just might for this site. They might delete blogs that are all created using static timestamp since it is a major footprint.

    I have coded it up for you in C#, but I can convert it to VB. I've included my headers in the comments. Also, the VB code is untested, but it should work. The C# code works for sure.

    Sorry for the formatting. BHW is making it un-prettified.

    C# Code:
    Code:
     
    public void makeBlog(string email, string password, string blogname)
            {
                HttpWebResponse response;
                string thePage = "";
                try
                {
                    //GET [URL]http://wallinside.com/[/URL] HTTP/1.1
                    //Accept: text/html, application/xhtml+xml, */*
                    //Accept-Language: en-US
                    //User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
                    //Accept-Encoding: gzip, deflate
                    //Host: wallinside.com
                    //DNT: 1
                    //Connection: Keep-Alive
    
                    CookieContainer cookies = new CookieContainer();
    
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("[URL]http://wallinside.com/[/URL]");
    
                    request.Accept = "text/html, application/xhtml+xml, */*";
                    request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US");
                    request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
                    request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
                    request.Host = "wallinside.com";
                    request.Headers.Add("DNT: 1");
                    request.KeepAlive = true;
                    request.CookieContainer = cookies;
    
                    response = (HttpWebResponse)request.GetResponse();
                    thePage = ReadResponse(response);
                    response.Close();
    
                    //POST [URL]http://wallinside.com/createBlog.php[/URL] HTTP/1.1
                    //Accept: text/html, application/xhtml+xml, */*
                    //Referer: [URL]http://wallinside.com/[/URL]
                    //Accept-Language: en-US
                    //Content-Type: application/x-www-form-urlencoded
                    //User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
                    //Accept-Encoding: gzip, deflate
                    //Connection: Keep-Alive
                    //Content-Length: 127
                    //DNT: 1
                    //Host: wallinside.com
                    //Pragma: no-cache
                    //Cookie: PHPSESSID=002ae5bf4e5b0a214f41e41d241b324f
    
                    request = (HttpWebRequest)WebRequest.Create("[URL]http://wallinside.com/createBlog.php[/URL]");
                    request.Accept = "text/html, application/xhtml+xml, */*";
                    request.Referer = "[URL]http://wallinside.com/[/URL]";
                    request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US");
                    request.ContentType = "application/x-www-form-urlencoded";
                    request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
                    request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
                    request.KeepAlive = true;
                    request.Host = "wallinside.com";
                    request.Headers.Add("DNT: 1");
                    request.KeepAlive = true;
                    request.Headers.Add("Pragma: no-cache");
                    request.CookieContainer = cookies;
    
                    request.Method = "POST";
                    request.ServicePoint.Expect100Continue = false;
    
                    //((Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds).ToString()
                    string body = "login=" + email + "&pass1=" + password + "&ewall=" + blogname + "&lg=en&pass2=&nowtime=" + (((Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds) - 5000) + "&Crear=Sign+Up.+It%27s+free%21";
    
                    byte[] postBytes = System.Text.Encoding.UTF8.GetBytes(body);
                    request.ContentLength = postBytes.Length;
                    Stream stream = request.GetRequestStream();
                    stream.Write(postBytes, 0, postBytes.Length);
                    stream.Close();
                    response = (HttpWebResponse)request.GetResponse();
                    thePage = ReadResponse(response);
                    response.Close();
                }
                catch(Exception ex)
                {
                }
            }
    
    
    VB Code:
    Code:
    
    Public Sub makeBlog(email As String, password As String, blogname As String)
     Dim response As HttpWebResponse
     Dim thePage As String = ""
     Try
      'GET [URL]http://wallinside.com/[/URL] HTTP/1.1
      'Accept: text/html, application/xhtml+xml, */*
      'Accept-Language: en-US
      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
      'Accept-Encoding: gzip, deflate
      'Host: wallinside.com
      'DNT: 1
      'Connection: Keep-Alive
    
      Dim cookies As New CookieContainer()
      Dim request As HttpWebRequest = DirectCast(WebRequest.Create("[URL]http://wallinside.com/[/URL]"), HttpWebRequest)
      request.Accept = "text/html, application/xhtml+xml, */*"
      request.Headers.[Set](HttpRequestHeader.AcceptLanguage, "en-US")
      request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
      request.Headers.[Set](HttpRequestHeader.AcceptEncoding, "gzip, deflate")
      request.Host = "wallinside.com"
      request.Headers.Add("DNT: 1")
      request.KeepAlive = True
      request.CookieContainer = cookies
      response = DirectCast(request.GetResponse(), HttpWebResponse)
      thePage = ReadResponse(response)
      response.Close()
    
      'POST [URL]http://wallinside.com/createBlog.php[/URL] HTTP/1.1
      'Accept: text/html, application/xhtml+xml, */*
      'Referer: [URL]http://wallinside.com/[/URL]
      'Accept-Language: en-US
      'Content-Type: application/x-www-form-urlencoded
      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
      'Accept-Encoding: gzip, deflate
      'Connection: Keep-Alive
      'Content-Length: 127
      'DNT: 1
      'Host: wallinside.com
      'Pragma: no-cache
      'Cookie: PHPSESSID=002ae5bf4e5b0a214f41e41d241b324f
    
      request = DirectCast(WebRequest.Create("[URL]http://wallinside.com/createBlog.php[/URL]"), HttpWebRequest)
      request.Accept = "text/html, application/xhtml+xml, */*"
      request.Referer = "[URL]http://wallinside.com/[/URL]"
      request.Headers.[Set](HttpRequestHeader.AcceptLanguage, "en-US")
      request.ContentType = "application/x-www-form-urlencoded"
      request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
      request.Headers.[Set](HttpRequestHeader.AcceptEncoding, "gzip, deflate")
      request.KeepAlive = True
      request.Host = "wallinside.com"
      request.Headers.Add("DNT: 1")
      request.KeepAlive = True
      request.Headers.Add("Pragma: no-cache")
      request.CookieContainer = cookies
    
      request.Method = "POST"
      request.ServicePoint.Expect100Continue = False
    
      '((Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds).ToString()
      Dim body As String = "login=" & email & "&pass1=" & password & "&ewall=" & blogname & "&lg=en&pass2=&nowtime=" & (DirectCast((DateTime.UtcNow.Subtract(New DateTime(1970, 1, 1))).TotalSeconds, Int32) - 5000) & "&Crear=Sign+Up.+It%27s+free%21"
      Dim postBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(body)
      request.ContentLength = postBytes.Length
      Dim stream As Stream = request.GetRequestStream()
      stream.Write(postBytes, 0, postBytes.Length)
      stream.Close()
      response = DirectCast(request.GetResponse(), HttpWebResponse)
      thePage = ReadResponse(response)
      response.Close()
     Catch ex As Exception
     End Try
    End Sub
    
    
    
     
    • Thanks Thanks x 2
    Last edited: Apr 24, 2014
  11. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    Thanks mate. I will look closer into your example.

    1. I will check that
    2. That's what I'm doing (going to main page, getting cookies, timestamp (dynamic not static)
    3. As above, timestamp is scraped from main page

    I'm very happy from an example. I will compare it with my code and hopefully I will find what's wrong.
     
  12. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    Quick question.

    On C# project it gives me error

    thePage = ReadResponse(response);

    Error 1 The name 'ReadResponse' does not exist in the current context
     
    Last edited: Apr 24, 2014
  13. Ambassy

    Ambassy BANNED BANNED

    Joined:
    Apr 13, 2011
    Messages:
    642
    Likes Received:
    163
    While I haven't gone into this specific example (seeing as xxf8xx fixed it already) one extra thing to watch out for that is a very common mistake for rookies is expect100continue - often you'll need to set this to false (the servicepoint) in order to fully receive a response.
    This is the case for several large sites such as last.fm etc.
     
    • Thanks Thanks x 1
  14. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK
    Yeap I found expect100continue some time ago... If you have any more advice, tips on what I should watch, I'm really happy to hear. Never enough of knowledge :>
     
  15. xxf8xx

    xxf8xx Supreme Member

    Joined:
    Nov 30, 2009
    Messages:
    1,321
    Likes Received:
    596
    Occupation:
    IM
    Sorry about that. You can either comment that line out, or add this method in. All it does is read the page that gets returned to a string, which you'll probably end up needing anyway to verify it was successful. Otherwise, you can just run the method and see the results in Fiddler.

    Code:
    private static string ReadResponse(HttpWebResponse response)
    {
                using (Stream responseStream = response.GetResponseStream())
                {
                    Stream streamToRead = responseStream;
                    if (response.ContentEncoding.ToLower().Contains("gzip"))
                    {
                        streamToRead = new GZipStream(streamToRead, CompressionMode.Decompress);
                    }
                    else if (response.ContentEncoding.ToLower().Contains("deflate"))
                    {
                        streamToRead = new DeflateStream(streamToRead, CompressionMode.Decompress);
                    }                using (StreamReader streamReader = new StreamReader(streamToRead, Encoding.UTF8))
                    {
                        return streamReader.ReadToEnd();
                    }
                }
    }
    
    
     
    • Thanks Thanks x 1
  16. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK

    Greatly appreciate your help ! Looks that I need to be much more careful while I'm using more complex classes like HTTP Web Request (or at least classes communicating with servers :)), I will investigate problem with my code and hopefully I will come out with something.

    Your code is working, no issue at all, huge thanks to all of you !
     
  17. xxf8xx

    xxf8xx Supreme Member

    Joined:
    Nov 30, 2009
    Messages:
    1,321
    Likes Received:
    596
    Occupation:
    IM
    No problem! I suggest you try and recreate the code yourself from scratch. It's quite odd that your cookies and headers we're different but I guess that happens. Let us know if you can recreate it on your own. If you have anymore questions feel free to let me know.

    Cheers,
    xxf8xx

    EDIT: I should also note that the above reader class was from a Fiddler plugin I use that converts webrequests to code. You should definitely look into it. I don't remember the name but it makes it a lot easier to just copy and paste a large portion of the code you need.
     
  18. carmamir

    carmamir Junior Member

    Joined:
    May 8, 2011
    Messages:
    118
    Likes Received:
    9
    Location:
    UK

    I have another problem.

    When I look into Fidller I see that cookie isn't passed.

    I used your example for other site and webbrowser passing cookie, but HTTP doesn't. However I can't see in code set for cookies (there is only req.cookies = cookies, but no cookies = resp.cookies), is it automatically set in this class or am I something missing ?
     
  19. xxf8xx

    xxf8xx Supreme Member

    Joined:
    Nov 30, 2009
    Messages:
    1,321
    Likes Received:
    596
    Occupation:
    IM
    You can try doing what they did in this thread:

    Code:
    http://stackoverflow.com/questions/17982131/cookies-dropped-lost-on-httpwebrequest-getreponse