A script to fetch HTML

1337python · Sep 23, 2013

Hello,

I would like to learn if there's a way for you to fetch a certain part of a website, for example, I want to know if my competitors use the same type of a script, line of code, etc. in their website. You type a part of code you want to check in the website and it searches for it in different URL's. The URL's would be checked from a text file which contains a fair amount of websites? I tried messing around with "curl" in bash, but I just can't seem to get it right. If you have some insight on how you can do it in python, korn shell, bash, c++, java, please share. Thanks

Panther28 · Sep 23, 2013

Here is some php script to do a curl scrape for a site. You will need to add a bit more to it, but this will get you going in the right direction.

<?php

$url = "ticketmaster.com";$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;

?>

kboing · Sep 23, 2013

use the code from Panther28 to fetch html and after preg_match_all to get all info you need !

roach · Sep 23, 2013

I would just use a simple webrequest to download the html, then parse it for the keywords you are looking for. I could code this in 15 mins tops. Just do a google on simple webrequests and another on regex... Best of luck to you.

1337python · Sep 23, 2013

Panther28 said:
Here is some php script to do a curl scrape for a site. You will need to add a bit more to it, but this will get you going in the right direction.

Thanks Panther, I'm going to try this tomorrow. You're helpful as ever.

roach said:
I would just use a simple webrequest to download the html, then parse it for the keywords you are looking for. I could code this in 15 mins tops. Just do a google on simple webrequests and another on regex... Best of luck to you.

I thought of that already, but it would take too much resources to complete that.

bytzu · Sep 23, 2013

You can use simple DOM parser http://simplehtmldom.sourceforge.net/. Extremely easy to use.

If you need any help let me know

roach · Sep 23, 2013

Sir it takes no resources at all to use a simple webrequest.. a general rule of thumb in life... If you dont know what you are talking about then keep your mouth shut.. Just makes you sound stupid when you make statements like that....

roach · Sep 23, 2013

Code to Download Source Code in VB NET:

Code:

 Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(TextBox2.Text)
    Dim response As System.Net.HttpWebResponse = request.GetResponse()

    Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream())

    Dim sourcecode As String = sr.ReadToEnd()
    TextBox1.Text = sourcecode

innosoft · Sep 23, 2013

if ur not good with programming and if u have less sites, u can do it manually by viewing source and hit on ctrl + F and type what u wanna search.. sometimes manual work is timesaver..

Edit: While i was typing roach gave a nice code..

small addition to it..

Code:

If SourceCode.Contains("<Text to Search>") then

MsgBox("Found....")

End If

roach · Sep 23, 2013

Here is a complete function I just got for you. It is VB NET and does exactly what you want:

Code:

Dim request As WebRequest = WebRequest.Create("http://pagewantingtouse.com") 'create a web request to the html file
    Using response As WebResponse = request.GetResponse() 'get the response back from the request
        Using Reader As New StreamReader(response.GetResponseStream) 'identify how you want to read the response and assign it to streamreader
            Dim webtext As String = Reader.ReadToEnd() 'read the response (web page) as a string to the end of the file

 Dim wbc As WebBrowser = New WebBrowser() 'create a web browser to handle the data. this will help sift through it

            wbc.DocumentText = ""

            With wbc.Document
                .OpenNew(True)
                .Write(webtext) 'write the web page response into the web browser control


                Dim itemlist As HtmlElementCollection = .GetElementsByTagName("DIV")

                    For Each item In itemlist 'look at each item in the collection
                    If item.classname = "description" Then 
                        msgbox item.innertext 'this would msgbox your description
                    Exit For 'exit once found
                    End If



                Next 'do this for every item in the collection until we find it

            End With 

        wbc.dispose()

        End Using

    End Using

1337python · Sep 24, 2013

roach said:

Here is a complete function I just got for you. It is VB NET and does exactly what you want:

Code:

Dim request As WebRequest = WebRequest.Create("http://pagewantingtouse.com") 'create a web request to the html file
    Using response As WebResponse = request.GetResponse() 'get the response back from the request
        Using Reader As New StreamReader(response.GetResponseStream) 'identify how you want to read the response and assign it to streamreader
            Dim webtext As String = Reader.ReadToEnd() 'read the response (web page) as a string to the end of the file

 Dim wbc As WebBrowser = New WebBrowser() 'create a web browser to handle the data. this will help sift through it

            wbc.DocumentText = ""

            With wbc.Document
                .OpenNew(True)
                .Write(webtext) 'write the web page response into the web browser control


                Dim itemlist As HtmlElementCollection = .GetElementsByTagName("DIV")

                    For Each item In itemlist 'look at each item in the collection
                    If item.classname = "description" Then 
                        msgbox item.innertext 'this would msgbox your description
                    Exit For 'exit once found
                    End If



                Next 'do this for every item in the collection until we find it

            End With 

        wbc.dispose()

        End Using

    End Using

I can't seem to get it working with MS Visual Studio 2012. Which version did you use when you were writing? A lot of your code was being changed by the compiler and it went all bad. Sorry if you misunderstood me in my previous post. I meant to say "Wouldn't it take too much resources" Lol. English isn't my 1st language, I know it isn't an excuse.

roach · Sep 24, 2013

Yes I am sorry I use Visual studio 2010. I have not moved to 2012 yet. I suppose I should have told you that. My bad

Good Luck.

Four Seasons · Sep 24, 2013

What you need is something called scraping and "parsing". Can be done in PHP too.

roach · Sep 24, 2013

BeautifulSoup and Python will also do that......

1337python · Sep 24, 2013

Thanks guys!

TZ2011 · Sep 24, 2013

Four Seasons said:
What you need is something called scraping and "parsing". Can be done in PHP too.

It can be done in php, but working just with classic .php it's not recommended since php can be pretty heavy on server resources. I have done various projects with fetching and content scraping with it, but to optimize it and make it to work stable, in timeout limits, can be major pain in the ass.

roach · Sep 24, 2013

Since you have sent me a PM explaining that you can not get this code to work. I am going to build you a very simple tool. This tool is just to show you a working function. I will not add features or update this tool. This tool is just to help you learn. Give me a few minutes to make this and then to virus scan it for you. I will upload the source code and comment it the best I can. I do not mind helping. Please take this simple tool and learn from it.

1337python · Sep 24, 2013

roach said:
Since you have sent me a PM explaining that you can not get this code to work. I am going to build you a very simple tool. This tool is just to show you a working function. I will not add features or update this tool. This tool is just to help you learn. Give me a few minutes to make this and then to virus scan it for you. I will upload the source code and comment it the best I can. I do not mind helping. Please take this simple tool and learn from it.

I appreciate that you do this for me. When I get more successful in coding and in IM I will definitely remember you!

roach · Sep 24, 2013

Here is the VERY SIMPLE APP. This is just a proof of concept to show you how to use the code I gave you. This does exactly what you asked for. http://puu.sh/4zchg.rar I uploaded using puush.me I love puush

Virus Total: https://www.virustotal.com/en/file/4b60dadbf3dd238f80d56e995cfb81902ba44c4300e9c8a56ac01ae07dde3a3a/analysis/1380043664/ This download has full source as well as exe. Good Luck learning to code

1337python · Sep 24, 2013

roach said:
Here is the VERY SIMPLE APP. This is just a proof of concept to show you how to use the code I gave you. This does exactly what you asked for. http://puu.sh/4zchg.rar I uploaded using puush.me I love puush Virus Total: https://www.virustotal.com/en/file/...00e9c8a56ac01ae07dde3a3a/analysis/1380043664/ This download has full source as well as exe. Good Luck learning to code

Thank you!!

A script to fetch HTML

Regular Member

Elite Member

Newbie

BANNED

Regular Member

Junior Member

BANNED

BANNED

Elite Member

BANNED

Regular Member

BANNED

Regular Member

BANNED

Regular Member

Senior Member

BANNED

Regular Member

BANNED

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World