How to search through 10 000 domains ?

[Balouffe]

Hello,
I develop a private interface in order to know in which websites I can post an article (with my backlinks of course). I have more than 10 000 differents domains. All website have the same language.
1/ I know that I can pay SEMRush more than $4800/year in order to extract with API all keywords of all websites and create a personnal little search engine in my interface.
Do you know cheaper service that have API and can do that please ?

2/ I also could extract all the anchors of each domain with Majestic. The cost is almost the same as SEMRush. It's really to expensive for me.
3/ I could scrape dozens of webpage of each domain and try to find some keywords in it. it's really cheaper but I don't know how can I identified keywords and not stop words like "the", "moreover" etc.

Do you have ideas, tips or anything else that could help me please ?
Thank you very much

 

[bossofthebosses]

Are you trying to find out which sites suit for your niche? You can use scrapbox to grab their meta title and description - put the results in an excel file and can easily search from there.

 

[Balouffe]

Hello bossofthebossess ^^ Thank you
Yep with the 10 000 domains, when I need to have a backlink on a e-cigarette website, I need to find it quickly.
Same thing, if I need a backlink on a website that is dog niche, I need to find it when I search in my interface the keywords "dog", "wouf" or other kind of keywords relevant to the dog niche
Your idea is very nice with Scrapebox and the meta title & description ! =) I will check how can I do it with Scrapebox. Hopefully I've bought this soft years ago
If you have any other idea, please let me know

 

[Balouffe]

Do you think is possible with Scrapebox to scrape all the H1, H2, H3 & H4 of 10 000 webpages ?

 

[bossofthebosses]

Do you think is possible with Scrapebox to scrape all the H1, H2, H3 & H4 of 10 000 webpages ?

Yes it is possible. But the software can only scrap those sites which have H1,2,3,4 in the HTML codes. If they don't have, then the software will not be able to do that.

 

[bossofthebosses]

scrapebox.com/webpage-meta-scraper

 

[Balouffe]

Yes I did this job with scrapebox.com/webpage-meta-scraper to scrape META. No problem with that.
But I don't know how to scrape the H1, H2, H3 of each page. Do you really think it's possible with scrapebox ?

 

[GILGAME$H]

You can do this if you're running Linux/BSD.

Simple bash script:
1. Download every single page from a domain using wget
2. Grep recursively to search every single page for a word e.g "dog" or even use regex to search for multiple words e.g "dog|e-cigarette|panties"
3. Output results to a text file and there you go

For H1/H2/H3 it's simple grep regex, too.

Don't know if this is allowed by the rules, but I'd be willing to do it for you if you're willing to do business.

 

[Crashtest]

Yeah I was going to say learning a bit regex +notepad++ search and replace bar could do it but learning regex seems like a pain in the arse!

 

[Zapdron]

I would make a python script that scrapes the text of each website, then I would implement TextRank to find the main keywords of the texts as shown here: https://towardsdatascience.com/textrank-for-keyword-extraction-by-python-c0bae21bcec0

 

[GILGAME$H]

Lol, you have absolutely no idea

Back when I was a freelance programmer, I had to make a script to automatically fetch all tweets from a single user -- only using regex.

Here's a small snippet (total 500 lines)

sed "s/<\/p>/<\/p><\/content><\/entry>/g" |\
        sed "s/$Z.\{0,$MAX_HASHTAG_CHAR\}$Y<s>\#<\/s>/\#/g" |\
        sed -r "s/<.{0,1}(b|s|a)>/ /g" | \
        sed "s/$Z.\{0,$MAX_HASHTAG_CHAR\}$W/<a href=\"/g" |\
        sed "s/$V.*span>/\"\/>/g" |\
        sed "s/$Z.\{0,$MAX_HASHTAG_CHAR\}$U.* >//g" |\
        sed -r "s/$Z\https?\:\/\/t\.co\/.{10}//g" |\
        sed "s/$V.*\" >/<img src=\"https\:\/\//g" |\
        sed "s/\&\#39\;/\'/g" |\
        sed "s/\&quot\;/\"/g" |\
        sed "s/  / /g" |\
        sed "s/ </</g" > "PAGE2";

What OP wants is so much easier than that.