Duplicate pages (not urls) in craigslist scrape

Discussion in 'Black Hat SEO' started by drifterzz, Aug 14, 2015.

  1. drifterzz

    drifterzz Newbie

    Joined:
    Feb 21, 2014
    Messages:
    1
    Likes Received:
    0
    I've scraped craigslist to email potential clients to my directory, but I find people have several pages with the same content in different cities. How can I take my list of craigslist urls and delete the duplicate content urls so I'm not sending these people multiple emails?