Please help with Google scrapping

[santis]

Hi!
I need to scrap domain names URLs from Google search like this: I will provide a keyword, ex."physiocare" and it will give me domain name results containing this keyword, ex: physiocaretz.com, physiocarehealthsolutions.com etc.
I have attached 2 screenshots.

Please suggest me tools and ways to do that.

 

[redwatermelon]

Use scrapebox to help you with that.

 

[Wilson Grant Fisk]

If you don't want anything fancy, then Simple SERP Scraper should be OK
https://urlprofiler.com/free-tools/

 

[amberlawan37]

You can scrape Google search results using Beautiful Soup.

 

[santis]

Thank you for your messages!

I've checked those tools, unfortunately, they don't do what I wanted

I hope I can find someone who can make a tool like this...
Is there a thread for custom tools?

Thank you!

 

[Wilson Grant Fisk]

Thank you for your messages!

I've checked those tools, unfortunately, they don't do what I wanted

I hope I can find someone who can make a tool like this...
Is there a thread for custom tools?

Thank you!

Use the WtB or HaF sections, somebody should be able to write you a script/app/program.

https://www.blackhatworld.com/forums/hire-a-freelancer.76/
https://www.blackhatworld.com/forums/want-to-buy.92/

 

[Wilson Grant Fisk]

i have a tool for this, kindly send me a pm

Share the tool in the download section, you can't sell via PM unless you upgrade.

 

[loopline]

Your just asking about filtering it sounds like. If you only want domain names that contain your keyword, then you either need to do 1 of 2 things.

1 - scrape google for inurl:keyword

and that should get you pretty close, you could try some other variants.

2 - filtering. So if you want to scrape a list of urls just based on your keywords - because that gets your ips banned less - then you could simply just "remove urls not containing" and give it a list of your keywords and it will remove anything that doesn't have your keyword in it.



Scrapebox can definitely do all of this.

If you use another tool you could just take the results and feed them to chat gpt and ask it to do the filtering.

Else probably there is some online tool that would allow you to do filtering of scraped urls.

 

[santis]

Your just asking about filtering it sounds like. If you only want domain names that contain your keyword, then you either need to do 1 of 2 things.

1 - scrape google for inurl:keyword

and that should get you pretty close, you could try some other variants.

2 - filtering. So if you want to scrape a list of urls just based on your keywords - because that gets your ips banned less - then you could simply just "remove urls not containing" and give it a list of your keywords and it will remove anything that doesn't have your keyword in it.



Scrapebox can definitely do all of this.

If you use another tool you could just take the results and feed them to chat gpt and ask it to do the filtering.

Else probably there is some online tool that would allow you to do filtering of scraped urls.

Loopline, thank you for your reply!

1. inurl doesn't work here, because, in my case, keywords from URL is different from keywords from domain names,
Good example of "physiocare" keyword in domain name: physiocarepro.com
Bad example of "physiocare" keyword in URL: nature.com/physiocare-services

Basically, I need to scrap domain names URLs, rather than just URLs.

I hope you understand what I'm trying to say, and maybe you have an idea how to do it.

Thanks again!

 

[gamball]

Loopline, thank you for your reply!

1. inurl doesn't work here, because, in my case, keywords from URL is different from keywords from domain names,
Good example of "physiocare" keyword in domain name: physiocarepro.com
Bad example of "physiocare" keyword in URL: nature.com/physiocare-services

Basically, I need to scrap domain names URLs, rather than just URLs.

I hope you understand what I'm trying to say, and maybe you have an idea how to do it.

Thanks again!

you can try "keyword here" site:hysiocarepro.com

 

[loopline]

Loopline, thank you for your reply!

1. inurl doesn't work here, because, in my case, keywords from URL is different from keywords from domain names,
Good example of "physiocare" keyword in domain name: physiocarepro.com
Bad example of "physiocare" keyword in URL: nature.com/physiocare-services

Basically, I need to scrap domain names URLs, rather than just URLs.

I hope you understand what I'm trying to say, and maybe you have an idea how to do it.

Thanks again!

Ok, sure, same concept applies to my 2 options.

option 1 - do like @gamball said and just do site: operator. You can test this in a browser and see if you get the results you want.

Option 2 - just filter. you can just trim urls to domain level and then remove urls that don't contain your keywords and your done.

Scrapebox does all of this.

Other tools may do both.

Or you can combo some tools. Find a tool that scrapes and then feed an AI the results and let it filter or just do the same steps with different tools.

 

[PhantomRoot]

Have you tried using a custom Google search engine and then scraping those results? Might be easier than trying to parse the standard results page.