Custom Scrape Tool PHP

[fatboy]

Ah spot on - just started coding my own to scrape member lists from forums
Lets me get on to part 2, the PM spammer!

 

[neta1o]

fatboy, glad I could save you some time. I programmed this a while back to test making new scraping tools. It just made more sense to use this instead. Let me know if you encounter a page your having trouble scraping and I'll take a look at it with you.

Also keep in mind that this only works on public pages. If you have to log into the page to see the content this scraper will not work. I've actually tried a few different $$$ tools that have trouble scraping private pages also.

 

[fatboy]

Well between us I think we have it cracked - I have a script that logs into things like forums, your script will grab the bits I need and then I have to write the PM spammer bit

Will slap it up here once its done - won't be soon as time isn't on my side lately!!

 

[neta1o]

That's great I'd love to enhance this for other BHW members. This is one of those tools that has thousands of implications.

 

[neta1o]

Per some requests in another thread for a copy and past code segment to automatically scrape a given page. Here was an example of someone wanting to scrape the now playing movie titles from IMDB, using this code you get the following result.
http://answerexpert.net/movies.php

<?php
$url="http://www.imdb.com/nowplaying/";
$beg="<td class="movie ">";
$end="</a>";

$data = file_get_contents($url);
	$regex = '/'.$beg.'(.+?)'.$end.'/';
	$count=1;
	preg_match_all($regex,$data,$match,PREG_SET_ORDER);
	foreach ($match as $result) {
		$link = $result[1];
		$link=strip_tags($link);
		echo $link . '<br>';
	}
?>

Just put that anywhere on a php page and it will automatically scrape the given page every time the page is loaded.

To use this for other pages just change the first 3 lines url, beg, and end

 

[kojakfull]

Per some requests in another thread for a copy and past code segment to automatically scrape a given page. Here was an example of someone wanting to scrape the now playing movie titles from IMDB, using this code you get the following result.
http://answerexpert.net/movies.php

<?php
$url="http://www.imdb.com/nowplaying/";
$beg="<td class="movie ">";
$end="</a>";

$data = file_get_contents($url);
	$regex = '/'.$beg.'(.+?)'.$end.'/';
	$count=1;
	preg_match_all($regex,$data,$match,PREG_SET_ORDER);
	foreach ($match as $result) {
		$link = $result[1];
		$link=strip_tags($link);
		echo $link . '<br>';
	}
?>

Just put that anywhere on a php page and it will automatically scrape the given page every time the page is loaded.

To use this for other pages just change the first 3 lines url, beg, and end

Thanks for this. But it is getting me error i tried it here http://kojakdirectory.com/scrape.php

 

[neta1o]

Thanks for this. But it is getting me error i tried it here http://kojakdirectory.com/scrape.php

That error is usually if your missing a ; at the end of the line, if you want I can look at the code or file for you

 

[kojakfull]

I just copied the one you provided which is this

<?php
$url="http://www.imdb.com/nowplaying/";
$beg="<td class="movie ">";
$end="</a>";

$data = file_get_contents($url);
	$regex = '/'.$beg.'(.+?)'.$end.'/';
	$count=1;
	preg_match_all($regex,$data,$match,PREG_SET_ORDER);
	foreach ($match as $result) {
		$link = $result[1];
		$link=strip_tags($link);
		echo $link . '<br>';
	}
?>

thanks

 

[fatboy]

You need to add a backslash to this line:

$beg="<td class="movie ">";

after the word movie, then the space, put a backslash before the next quote marks

 

[neta1o]

Yep fatboy is right, I just checked the code, for whatever reason the forum is modifying it when I copy/paste here no matter which tag I use html, code, quote etc...it's removing the back slashes...Anyways I attached the file here for download.

 

[fatboy]

Glad it wasn't only me having problems posting backslashes to the forum!!!

 

[chucklechuck]

ok...think i'm getting there.

Is there a rule of thumb on where backslashes need to go? In the example given if i wanted to scrape the opening paragraph how would i need to tweak the code below to get the desired result?
$end="<div id="nowplaying-media-strip">";

Many variations i've tried have returned a blank screen.

 

[fatboy]

if the thing you are looking for includes quotes stick a slash in there and you should be happy. In that bot of code you need one before the quote to the left of nowplaying and also to the quote just after strip.

 

[neta1o]

$end="<div id="nowplaying-media-strip">";

You'll need to put back slashes before any inner quotes, this forum doesn't allow me to post back slashes so I'll use the pipe instead to illustrate where the backslashes would be

$end="<div id=|"nowplaying-media-strip|">";

 

[chucklechuck]

Thats what i thought but kept getting a blank page. I'll have to keep playing with it tomorrow. Thanks for your help

 

[neta1o]

Thats what i thought but kept getting a blank page. I'll have to keep playing with it tomorrow. Thanks for your help

Let me know if you need help, just pm me a link and tell me what your wanting to scrape and I'll see if I can help

 

[Whookidive]

It would be cool to have a tool where you can enter a movie name and that it fetches some details like plot, year and rating of IMDB and puts it in a database. That would be pwnage.

 

[undeterminederror]

Great info NETA10.
do you think its possible to have more than 1 variable?
like this:
<string1>...<string11>{text I}<string22>...<string111>{text II}<string222>...<string2>
-to define the strings and to output the text I and II in a template.

 

[neta1o]

Great info NETA10.
do you think its possible to have more than 1 variable?
like this:
<string1>...<string11>{text I}<string22>...<string111>{text II}<string222>...<string2>
-to define the strings and to output the text I and II in a template.

I could work on something like that for you, could you post here or pm me a page/uRL and an example of what you'd like to scrape so I can test?

Thanks

 

[undeterminederror]

I could work on something like that for you, could you post here or pm me a page/uRL and an example of what you'd like to scrape so I can test?

Thanks

Wait. could you combine with a image grabber? i'd pay for that. send you a pm.