Filtering data

Is there any good way to filter previously posted comments for example on BE or WP with newly harvested lists?

It's not about simply removing duplicates in SB but comparing two/multiple .txt files to remove the already posted entries.

Thanks in advance.