1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

make webbrowser control load pages faster

Discussion in 'General Programming Chat' started by coolcurrent, Oct 2, 2012.

  1. coolcurrent

    coolcurrent Newbie

    Joined:
    Feb 15, 2011
    Messages:
    25
    Likes Received:
    0
    is there any way i could make the webbrowser control load large textual pages faster. its taking about 5-10mins to loag blogs with about 500 comments
     
  2. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    use webrequest rather. its fast.

    if you are destined to use webbrowser control, and searching for tags and stuff then immediately after navigating to the page. you should start looking for tags and don't need to wait for the whole page to load. The idea is to only wait till you get your desired element.
     
  3. HostStage

    HostStage Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    May 20, 2010
    Messages:
    1,773
    Likes Received:
    1,730
    Occupation:
    BHW - CEO of Webhosting Company
    Location:
    BWH from France
    Home Page:
    If you are using wordpress, you can set the comment to be spread on several pages. And that would be the end of your troubles.
     
  4. coolcurrent

    coolcurrent Newbie

    Joined:
    Feb 15, 2011
    Messages:
    25
    Likes Received:
    0
    blackcatavican

    you post a sample code, HttpWebRequest does not maintain cookies and hidden input fields nor even javascript

    abysseam

    posting to several wordpress website which are not mine, so i cant do that
     
  5. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    hey,

    webrequest can have cookies. lemme see if i can post code here, else i will inbox you.

    HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(url);
    Request.CookieContainer = loginCookies;
    var Response = Request.GetResponse();
    loginCookies.Add(Response.Cookies);

    For hidden fields, are you working on, post or get methods, you can always extract hidden fields from response and then pass it as parameters.
     
  6. coolcurrent

    coolcurrent Newbie

    Joined:
    Feb 15, 2011
    Messages:
    25
    Likes Received:
    0
    thanks, but how do extract the hidden fields knowing that i will then have to write a parser for that?
     
  7. judson

    judson Power Member

    Joined:
    Nov 29, 2009
    Messages:
    530
    Likes Received:
    319
    Occupation:
    Fulltime Newbie IM
    Location:
    Sub Ubi
    Do a google search for AJAX Lazy Loading.

    You can lazily load content. i.e. The page loads, and 'finishes'. As the user scrolls down the page, it loads the additional content as the user gets to it. If they never scroll down to that bit of the content, it does not load.

    Used mainly for images, although you can tweak it to lazily load text too.
     
  8. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,884
    Likes Received:
    1,932
    Easy parsing function:

    PHP:
    Public Function GetBetween(ByVal haystack As StringByVal needle As StringByVal needle_two As String) As String
            Dim istart 
    As Integer InStr(haystackneedle)
            If 
    istart 0 Then
                
    ' Dim istop As Integer = InStr(istart, haystack, needle_two)
                Dim istop As Integer = InStr(istart + Len(needle), haystack, needle_two)
                If istop > 0 Then
                    Try
                        Dim value As String = haystack.Substring(istart + Len(needle) - 1, istop - istart - Len(needle))
                        Return value
                    Catch ex As Exception
                        Return ""
                    End Try
                End If
            End If
            Return ""
        End Function
    Dim myStr as String = "Hello Cool World"
    Dim WhatAmI As String = GetBetween(myStr, "Hello ", " World")
     
  9. cgimaster

    cgimaster Power Member

    Joined:
    Jun 30, 2012
    Messages:
    525
    Likes Received:
    311
    Gender:
    Male
    You can use HtmlAgilityPack to read the httprequest result, here is a sample to get the input fields:

    Code:
    HtmlAgilityPack.HtmlNode.ElementsFlags.Remove("form");
    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    doc.LoadHtml(content);
    
    var nodes = doc.DocumentNode.SelectNodes("//input");
    if (nodes != null && nodes.Count > 0)
    {
        foreach (var item in nodes)
        {
            MessageBox.Show("Found input field: " + item.Name);
        }
    }
    else
    {
        MessageBox.Show("Found not input fields");
    }

    Keep in mind if there is more than one form in the page you might want to narrow the xpath looking for input fields.

    You could also save the fields on a dictionary or other way to later reuse it by setting your default values.
     
  10. CodingAndStuff

    CodingAndStuff Regular Member

    Joined:
    May 6, 2012
    Messages:
    236
    Likes Received:
    84
    Occupation:
    Swagstronaut
    Location:
    You can't have my bots. Sorry :'(
    Since the WebBrowser control is merely an Internet Explorer frame (with some slight differences), chances are it's not going to run faster than what IE is throttled to on your machine based on your settings.

    You could try disabling javascript (IE's JS engine blows), or even prevent the loading of assets (images, css files, etc).

    If those aren't an option for you, look into WebKit. You can't statically link it, though, so you would need to distribute the software with WebKit.dll which shouldn't be an issue.

    If you're decent with C/C++, check out QT by Nokia. It's cross platform, fast, and has a nice little form editor for the widgets (QT Creator).
     
  11. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    webkit or gecko browser is a very good alternative for webbrowser control. The only restriction is you cannot click programmatically by invokemember. There is no method available as of now like invokemember.
     
  12. CodingAndStuff

    CodingAndStuff Regular Member

    Joined:
    May 6, 2012
    Messages:
    236
    Likes Received:
    84
    Occupation:
    Swagstronaut
    Location:
    You can't have my bots. Sorry :'(
    With WebKit there most definitely is. You can easily bridge the DOM on the page to the native code very easily, and even eval raw JS if required. It's a complete browser, therefore you've got full control over it. :\
     
  13. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    hey there,

    can you provide me with a sample code or something of that sort, for webkit, with calling eval raw JS. I am getting, what you meant there. My problem is somewhat like, sometimes i need to click on a div element (which essentially doesnt have the onclick property).

    Can you show it to me, how can i accomplish that, with a webkit. thanks alot.
     
  14. coolcurrent

    coolcurrent Newbie

    Joined:
    Feb 15, 2011
    Messages:
    25
    Likes Received:
    0
    when using WB control, I noticed that it slows done after processing some urls, also there is a constant spike in the memory usage, reach about 60MB, for just a single form navigating.i considered webkit, but i cant click through code, so don't know how to submit forms right now.
     
  15. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    coolcurrent:

    you can throw, js for form.submit. There is a specific webkit method for that. check that out.
     
  16. CodingAndStuff

    CodingAndStuff Regular Member

    Joined:
    May 6, 2012
    Messages:
    236
    Likes Received:
    84
    Occupation:
    Swagstronaut
    Location:
    You can't have my bots. Sorry :'(
    http://sourceforge.net/projects/webkitdotnet/

    Then: http://webkitdotnet.sourceforge.net/basics.php?p=3 if you want to be lazy with it (just append a script tag to it). I'm not really sure what members were added to the library, so you'll just have to look.

    If you're using Python/C++ with QT, there is a delegation class for QWebKit and it's worth checking out.
     
  17. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    hey i have already gone through all the documentation on webkitdotnet.

    i will see the OWebKit, though.