Can someone help me with this script?

istuff

Junior Member
Joined
Mar 2, 2009
Messages
138
Reaction score
48
I'm trying to build a scraper for CL, I have got the majority of it down, but i need help. The script is made up of two components.

first component gets the links for all the postings. second components finds emails in the page in which you provide the url. Can someone tell me how to intergrate the $url (which spews out the links) into the scraper which gets the url from a form. I tried outputting the $url into the form and clicking submit however that doesnt output all the links into the form. I'd appreciate it if someone can help.
 
Why not have the first part collect the links and add them into a file or db table, then have the second component just loop through the list?

If you are asking for specific instructions on integrating two scripts, it is hard for me to explain without seeing the scripts, but from what I gather the second script is executed by way of form post so you would need to find the section of the code that is executed when the form is submitted. From there you could create a function from that code then run a loop through all your urls calling the function on each url.

Hope this helps.
 
I'll tell you what I did.

First I saves all countries of USA.
then saved all categories.
After then, got each country and category in loop and fetched all links one by one. It took 5 hours but fetched over 300k listings. and stored all in notepad.

Finally grabbed each listing and fetch emails from it. so leeched whole CL US section :D
 
Why not just just cURL and save the data it retreives to a variable...then use a regular expression to search the data for emails and output them into another variable or print them.
 
Back
Top