How do I target names on imprint sites?

[Bonvi]

I struggle to consistently find names in imprint sites. Imprints are a big deal in my country, as they are mandatory for websites and a great source for finding business contact info.

But they slightly differ all the time and I have no clue how I can filter names? I am using python and anything selenium, scrapy, beautifulsoup etc is okay to work with but I posted in general because I have no idea how to even attack the issue.

I tried looking for clues like "Geschäftsführer" (director) on the same line as names, but I got so much BS from extracting those lines and rarely ever found names. Either they were on a different line or I found everything after etc.

 

[Smackhead]

Can you share some examples please either here or on pm

 

[Ashk881]

I struggle to consistently find names in imprint sites. Imprints are a big deal in my country, as they are mandatory for websites and a great source for finding business contact info.

But they slightly differ all the time and I have no clue how I can filter names? I am using python and anything selenium, scrapy, beautifulsoup etc is okay to work with but I posted in general because I have no idea how to even attack the issue.

I tried looking for clues like "Geschäftsführer" (director) on the same line as names, but I got so much BS from extracting those lines and rarely ever found names. Either they were on a different line or I found everything after etc.

You don't use a hard-coded scraping system for this kind of stuff.

What's usually done is you scrape the entire html inside the body, preprocess (remove html entities/tags, strip whitespace etc) so that you get a bunch of clean text paragraphs in the end and then use an NLP pipeline to find out names, addresses, phone numbers etc from the paragraph.

It's called Named Entity Recognition, look up Spacy and NLTK in python.

LLMs are also really good for this kind of stuff. So you can give chatGPT an example of a impressum and the fields you need (name, address, phone etc) and it can give you nicely formatted json from unstructured paragraphs.

If you are not parsing millions of words and are willing to pay like $5-10, it would be the quickest route I would choose.

 

[Bonvi]

You don't use a hard-coded scraping system for this kind of stuff.

What's usually done is you scrape the entire html inside the body, preprocess (remove html entities/tags, strip whitespace etc) so that you get a bunch of clean text paragraphs in the end and then use an NLP pipeline to find out names, addresses, phone numbers etc from the paragraph.

It's called Named Entity Recognition, look up Spacy and NLTK in python.

LLMs are also really good for this kind of stuff. So you can give chatGPT an example of a impressum and the fields you need (name, address, phone etc) and it can give you nicely formatted json from unstructured paragraphs.

If you are not parsing millions of words and are willing to pay like $5-10, it would be the quickest route I would choose.

5-10 dollars for how much? I am looking to use it on 650 websites.

Your advice is a really interesting procedure though, because I tried to get the page text and then use regex or beautifulsoup or similar to filter through it until I find my keywords and scrape everything behind it on the same line. But I am sitting here for 3 hours and it doesn't even work with my direct example no matter how specific I make it. I am pulling hairs out here.

But do why can't I just take the text from the page instead? Wouldn't that eliminate the need to clear out all html tags etc? Seems unnecessary. Or maybe I don't understand it right.

 

[Ashk881]

5-10 dollars for how much? I am looking to use it on 650 websites.

Considering each imprint would be about 500 -750 words or roughly 1k token for the system prompt, input and output, 5-10$ should be enough for 650 documents given the current pricing at $0.0015/1k tokens for gpt-3.5-instruct api

But do why can't I just take the text from the page instead?

Usually it isn't enough. There's still whitespaces and html entities left which you would need to clean for better results.

use regex or beautifulsoup or similar to filter through it until I find my keywords and scrape everything behind it on the same line. But I am sitting here for 3 hours and it doesn't even work

Right regexes are hard and hardcoding values from them can indeed by hair-pullingly difficult. This is a more nuanced and automated way compared way to those manual methods and is what search engines use.

 

[Bonvi]

Considering each imprint would be about 500 -750 words or roughly 1k token for the system prompt, input and output, 5-10$ should be enough for 650 documents given the current pricing at $0.0015/1k tokens for gpt-3.5-instruct api



Usually it isn't enough. There's still whitespaces and html entities left which you would need to clean for better results.



Right regexes are hard and hardcoding values from them can indeed by hair-pullingly difficult. This is a more nuanced and automated way compared way to those manual methods and is what search engines use.

It's all new to me, but I will look into it. I will, however, need to find a solution that is free as i have other plans with it as well. But your reply was massively helpful

 

[Bonvi]

Soo.. it worked a lot better now although I still get stuff that's definitely not names, but for a first try it's so much better. Not quite sure how it all works yet, as I used chatgpt to help me coding it, but I will definitely study this waaaaay more in depth. Didn't expect it to work so well right off the bat. Very pleased with the method.