make webbrowser control load pages faster

coolcurrent · Oct 2, 2012

is there any way i could make the webbrowser control load large textual pages faster. its taking about 5-10mins to loag blogs with about 500 comments

blackcatavi · Oct 2, 2012

use webrequest rather. its fast.

if you are destined to use webbrowser control, and searching for tags and stuff then immediately after navigating to the page. you should start looking for tags and don't need to wait for the whole page to load. The idea is to only wait till you get your desired element.

HostStage · Oct 2, 2012

If you are using wordpress, you can set the comment to be spread on several pages. And that would be the end of your troubles.

coolcurrent · Oct 2, 2012

blackcatavican

you post a sample code, HttpWebRequest does not maintain cookies and hidden input fields nor even javascript

abysseam

posting to several wordpress website which are not mine, so i cant do that

blackcatavi · Oct 3, 2012

hey,

webrequest can have cookies. lemme see if i can post code here, else i will inbox you.

HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(url);
Request.CookieContainer = loginCookies;
var Response = Request.GetResponse();
loginCookies.Add(Response.Cookies);

For hidden fields, are you working on, post or get methods, you can always extract hidden fields from response and then pass it as parameters.

coolcurrent · Oct 4, 2012

thanks, but how do extract the hidden fields knowing that i will then have to write a parser for that?

judson · Oct 4, 2012

Do a google search for AJAX Lazy Loading.

You can lazily load content. i.e. The page loads, and 'finishes'. As the user scrolls down the page, it loads the additional content as the user gets to it. If they never scroll down to that bit of the content, it does not load.

Used mainly for images, although you can tweak it to lazily load text too.

gimme4free · Oct 4, 2012

Easy parsing function:

PHP:

Public Function GetBetween(ByVal haystack As String, ByVal needle As String, ByVal needle_two As String) As String
        Dim istart As Integer = InStr(haystack, needle)
        If istart > 0 Then
            ' Dim istop As Integer = InStr(istart, haystack, needle_two)
            Dim istop As Integer = InStr(istart + Len(needle), haystack, needle_two)
            If istop > 0 Then
                Try
                    Dim value As String = haystack.Substring(istart + Len(needle) - 1, istop - istart - Len(needle))
                    Return value
                Catch ex As Exception
                    Return ""
                End Try
            End If
        End If
        Return ""
    End Function

Dim myStr as String = "Hello Cool World"
Dim WhatAmI As String = GetBetween(myStr, "Hello ", " World")

cgimaster · Oct 4, 2012

coolcurrent said:
thanks, but how do extract the hidden fields knowing that i will then have to write a parser for that?

You can use HtmlAgilityPack to read the httprequest result, here is a sample to get the input fields:

Code:

HtmlAgilityPack.HtmlNode.ElementsFlags.Remove("form");
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(content);

var nodes = doc.DocumentNode.SelectNodes("//input");
if (nodes != null && nodes.Count > 0)
{
    foreach (var item in nodes)
    {
        MessageBox.Show("Found input field: " + item.Name);
    }
}
else
{
    MessageBox.Show("Found not input fields");
}

Keep in mind if there is more than one form in the page you might want to narrow the xpath looking for input fields.

You could also save the fields on a dictionary or other way to later reuse it by setting your default values.

CodingAndStuff · Oct 5, 2012

Since the WebBrowser control is merely an Internet Explorer frame (with some slight differences), chances are it's not going to run faster than what IE is throttled to on your machine based on your settings.

You could try disabling javascript (IE's JS engine blows), or even prevent the loading of assets (images, css files, etc).

If those aren't an option for you, look into WebKit. You can't statically link it, though, so you would need to distribute the software with WebKit.dll which shouldn't be an issue.

If you're decent with C/C++, check out QT by Nokia. It's cross platform, fast, and has a nice little form editor for the widgets (QT Creator).

blackcatavi · Oct 5, 2012

webkit or gecko browser is a very good alternative for webbrowser control. The only restriction is you cannot click programmatically by invokemember. There is no method available as of now like invokemember.

CodingAndStuff · Oct 5, 2012

blackcatavi said:
webkit or gecko browser is a very good alternative for webbrowser control. The only restriction is you cannot click programmatically by invokemember. There is no method available as of now like invokemember.

With WebKit there most definitely is. You can easily bridge the DOM on the page to the native code very easily, and even eval raw JS if required. It's a complete browser, therefore you've got full control over it. :\

blackcatavi · Oct 7, 2012

CodingAndStuff said:
With WebKit there most definitely is. You can easily bridge the DOM on the page to the native code very easily, and even eval raw JS if required. It's a complete browser, therefore you've got full control over it. :\

hey there,

can you provide me with a sample code or something of that sort, for webkit, with calling eval raw JS. I am getting, what you meant there. My problem is somewhat like, sometimes i need to click on a div element (which essentially doesnt have the onclick property).

Can you show it to me, how can i accomplish that, with a webkit. thanks alot.

coolcurrent · Oct 8, 2012

when using WB control, I noticed that it slows done after processing some urls, also there is a constant spike in the memory usage, reach about 60MB, for just a single form navigating.i considered webkit, but i cant click through code, so don't know how to submit forms right now.

blackcatavi · Oct 8, 2012

coolcurrent:

you can throw, js for form.submit. There is a specific webkit method for that. check that out.

CodingAndStuff · Oct 10, 2012

blackcatavi said:
hey there,

can you provide me with a sample code or something of that sort, for webkit, with calling eval raw JS. I am getting, what you meant there. My problem is somewhat like, sometimes i need to click on a div element (which essentially doesnt have the onclick property).

Can you show it to me, how can i accomplish that, with a webkit. thanks alot.

http://sourceforge.net/projects/webkitdotnet/

Then: http://webkitdotnet.sourceforge.net/basics.php?p=3 if you want to be lazy with it (just append a script tag to it). I'm not really sure what members were added to the library, so you'll just have to look.

If you're using Python/C++ with QT, there is a delegation class for QWebKit and it's worth checking out.

blackcatavi · Oct 17, 2012

hey i have already gone through all the documentation on webkitdotnet.

i will see the OWebKit, though.

make webbrowser control load pages faster

coolcurrent

Newbie

blackcatavi

Registered Member

HostStage

Elite Member

coolcurrent

Newbie

blackcatavi

Registered Member

coolcurrent

Newbie

judson

Power Member

gimme4free

Elite Member

cgimaster

Power Member

CodingAndStuff

Regular Member

blackcatavi

Registered Member

CodingAndStuff

Regular Member

blackcatavi

Registered Member

coolcurrent

Newbie

blackcatavi

Registered Member

CodingAndStuff

Regular Member

blackcatavi

Registered Member

Main Menu

Marketplace

Making Money

BlackHat World