Duplicate pages (not urls) in craigslist scrape

drifterzz

Newbie
Joined
Feb 21, 2014
Messages
1
Reaction score
0
I've scraped craigslist to email potential clients to my directory, but I find people have several pages with the same content in different cities. How can I take my list of craigslist urls and delete the duplicate content urls so I'm not sending these people multiple emails?
 
Back
Top