mainceaft
Senior Member
- Apr 10, 2013
- 884
- 185
Hi . I'm doing some work o simple PHP pages crawler . I'm still test it ad not even reach step to make it automatically crawl ad store pages in DB .
First thing I faced after success crawling first page . is detecting Contents div .
I can do this manually and define contents div Id or class . but this will take more time for each new site I add . as I'm Thinking to hundreds of sites to it .
in short this is my code
^^ I reach that codes after long search to put best suitable codes .. but it's not finished yet and I'll show you why .
This example of X site have unknown dive contents Id .
The script in first loop will strip html codes and return pain text for dive 01 and all remaining div's .
Second loop will return div02 plain-text . third loop will return div-03 and div-03-1 texts .
here Draw for the same code
http://s16.postimg.org/c2ixl33np/rect3399.png
That was simple example in real test it return me with +50 Div from all page contents .
I cant post this question on stackoverflow as they close every BH related questions . and I don't have many account with others web forums .
First thing I faced after success crawling first page . is detecting Contents div .
I can do this manually and define contents div Id or class . but this will take more time for each new site I add . as I'm Thinking to hundreds of sites to it .
in short this is my code
Code:
$html = new simple_html_dom();
$html->load($target_url);
$divContent =$html->find('div');
foreach($divContent as $e) {
$Co=(utf8(preg_replace('#<[^>]+>#','',$e->outertext )));
$Co = preg_replace(array('/\s{2,}/', '/[\t\n]/'), ' ', $Co);
if ( !rep($Co) AND strlen($Co)>90)
{ { if (strlen($Co)>90)
{ echo '<h1 class="h" > '.$h++.'</h1>';
if ($e->id) echo 'ID Is :- '.(utf8($e->id)).'<br/>'.PHP_EOL;
if ($e->class) echo'<br/>Class Is :- '.(utf8($e->class)).'<br/>'.PHP_EOL;
echo'<div class="con"><h2>Content is</h2>'.$Co.'</div>';
}
} }^^ I reach that codes after long search to put best suitable codes .. but it's not finished yet and I'll show you why .
This example of X site have unknown dive contents Id .
Code:
<div id="01">
<h3>The standard Lorem Ipsum passage, used since the 1500s</h3>
<div class="02">
<p>"Lorem ipsum dolor sit amet, consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. . "</p>
</div>
Section 1.10.32 of "de Finibus Bonorum et Malorum", written by Cicero in 45 BC
<div id="03">
<div id="03-1">
ads
</div>
"Sed ut perspiciatis unde omnis iste natus error sit voluptatem
accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo
</div>
</div>Second loop will return div02 plain-text . third loop will return div-03 and div-03-1 texts .
here Draw for the same code
http://s16.postimg.org/c2ixl33np/rect3399.png
That was simple example in real test it return me with +50 Div from all page contents .
I cant post this question on stackoverflow as they close every BH related questions . and I don't have many account with others web forums .
Last edited:
