crawling website with ocasional redirects

[Mutikasa]

I am trying to scrape website which have popups or whatever. So sometimes my parser "simple_html_dom" for php gets the html of a popup instead of the page I need.
How can I get around this?

 

[bartosimpsonio]

Filter out pages outside $target_domain

if (!preg_match("mydomain.com",$that_shit)) redirect("matt cutts home");

 

[Mutikasa]

cool, so I just keep on reloading until i get the page i want

 

[Mutikasa]

It doesn't work. It keeps bringing me back to the popup for eternity until maximum execution time is exceeded

 

[Mutikasa]

I had to add cookie to the crawler because the popup is only one per session

 

[bartosimpsonio]

It doesn't work. It keeps bringing me back to the popup for eternity until maximum execution time is exceeded

You may be doing something wrong. If in doubt, hire a freelancer to do this for you.