How do I target names on imprint sites?

Bonvi said:

I struggle to consistently find names in imprint sites. Imprints are a big deal in my country, as they are mandatory for websites and a great source for finding business contact info.

But they slightly differ all the time and I have no clue how I can filter names? I am using python and anything selenium, scrapy, beautifulsoup etc is okay to work with but I posted in general because I have no idea how to even attack the issue.

I tried looking for clues like "Geschäftsführer" (director) on the same line as names, but I got so much BS from extracting those lines and rarely ever found names. Either they were on a different line or I found everything after etc.

Click to expand...

Ashk881 said:

You don't use a hard-coded scraping system for this kind of stuff.

What's usually done is you scrape the entire html inside the body, preprocess (remove html entities/tags, strip whitespace etc) so that you get a bunch of clean text paragraphs in the end and then use an NLP pipeline to find out names, addresses, phone numbers etc from the paragraph.

It's called Named Entity Recognition, look up Spacy and NLTK in python.

LLMs are also really good for this kind of stuff. So you can give chatGPT an example of a impressum and the fields you need (name, address, phone etc) and it can give you nicely formatted json from unstructured paragraphs.

If you are not parsing millions of words and are willing to pay like $5-10, it would be the quickest route I would choose.

Click to expand...

Bonvi said:

5-10 dollars for how much? I am looking to use it on 650 websites.

Click to expand...

Bonvi said:

But do why can't I just take the text from the page instead?

Click to expand...

Bonvi said:

use regex or beautifulsoup or similar to filter through it until I find my keywords and scrape everything behind it on the same line. But I am sitting here for 3 hours and it doesn't even work

Click to expand...

Ashk881 said:

Considering each imprint would be about 500 -750 words or roughly 1k token for the system prompt, input and output, 5-10$ should be enough for 650 documents given the current pricing at $0.0015/1k tokens for gpt-3.5-instruct api



Usually it isn't enough. There's still whitespaces and html entities left which you would need to clean for better results.



Right regexes are hard and hardcoding values from them can indeed by hair-pullingly difficult. This is a more nuanced and automated way compared way to those manual methods and is what search engines use.

Click to expand...