we're not doing anything too unusual. looking for journalist emails. sometimes they are small journalists and maybe a scraper would be easy since we could just scrape their blog for any emails mentioned and that might be useful to get emails to the owner of the blog.
however, if the journalist is not an owner of the site / URL then their email may or may not be buried somewhere in one of the articles or the author page or a Contact Us page or something.
so we're just targeting specific people / journalists. many of the journalists we are trying to find are just freelancers and aren't the blog-owners. ???
i could easily compile a list of URL's that publish works of the journalists we are looking for but that won't be terribly targeted information.
?
To your first question, having an API isn't going to help you with this, not that i can see. I mean the only thing I can see an API doing in this case is over complicating things.
As for how to get your end results, I don't have a great solution. You need a human brain to process that sort of data typically. I mean any scraper can scrape, but unless the email has Email - this is the owner. Information with it, then you have no way of really knowing that right?
I mean think of it opposite of probably how you are now. This is how I approach it anyway.
1 - How can I do what I want to do manually.
2 - How can I then automate that.
Most people start with number 2 and skip 1 and then run into a wall. So how do you get the data you want now? figure that out, then look at, can I automate this?
If its repetative and you get the data you want based on X qualifiers, then you can try and automate that.
If its different in every article and every case and sometimes you have to dig around for your authors mails and try and associate things, then its going to be hard to automate because its not consistent. So in that case you may be able to take it part way and aggregate data faster and filter, but may still need a human involved.
