crawling website with ocasional redirects

Mutikasa

Power Member

May 23, 2011

589

216

• Sep 14, 2015

• #1

I am trying to scrape website which have popups or whatever. So sometimes my parser "simple_html_dom" for php gets the html of a popup instead of the page I need.
How can I get around this?

 

bartosimpsonio

Elite Member

Executive VIP

Jr. VIP

Mar 21, 2013

24,770

31,555

• Sep 14, 2015

• #2

Filter out pages outside $target_domain

if (!preg_match("mydomain.com",$that_shit)) redirect("matt cutts home");

 

Mutikasa

Power Member

May 23, 2011

589

216

• Sep 14, 2015
• Thread Starter Thread Starter

• #3

cool, so I just keep on reloading until i get the page i want

 

Mutikasa

Power Member

May 23, 2011

589

216

• Sep 14, 2015
• Thread Starter Thread Starter

• #4

It doesn't work. It keeps bringing me back to the popup for eternity until maximum execution time is exceeded

 

Mutikasa

Power Member

May 23, 2011

589

216

• Sep 14, 2015
• Thread Starter Thread Starter

• #5

I had to add cookie to the crawler because the popup is only one per session

 

bartosimpsonio

Elite Member

Executive VIP

Jr. VIP

Mar 21, 2013

24,770

31,555

• Sep 14, 2015

• #6

Mutikasa said:

It doesn't work. It keeps bringing me back to the popup for eternity until maximum execution time is exceeded

Click to expand...

You may be doing something wrong. If in doubt, hire a freelancer to do this for you.