Bot or Script for Scraping Search Results?

[The Curator]

Wanted something that will scrape X number of search results for a given list of keyword(s) and put it into an excel file, would that be a script or bot? Which would be cheaper to get designed?
Appreciate any guidance.

 

[SpoonFeeder]

That's what ScrapeBox does or do you want to scrape results not from search engines?

If you can post your requirements, I'll see if I can code one real quick and give it away for free for BHW members.

 

[The Curator]

That's what ScrapeBox does or do you want to scrape results not from search engines?

If you can post your requirements, I'll see if I can code one real quick and give it away for free for BHW members.

I didn't find that scrapebox could be so surgical with harvesting, usually limiting number of results still didn't work, and the results weren't in order.

Requirements:

• Scrape X number of search results from Google (X being a variable number)
• Scrape results for X number of keywords (for projects that have multiple keywords)
• Take out duplicate url's

I use this for content development, to put together for our researchers who distill the info down for our writers. The variability of both the number of keywords and the number of search results can allow people to research one keyword/topic lightly or heavily - even allowing for silo/skyscraper content research.

Appreciate it the help with this!

 

[SpoonFeeder]

Scrapebox is the right tool for this and it costs just $67 : http://www.scrapebox.com/bhw

Or do you want a custom coded bot/script?

P.S I'm not applying for the job as it's against BHW TOS. It's just a general discussion in open public.

 

[Warrenway]

If you don't mind using Bing search results, they have a Search API which would negate the need for scraping search results entirely.

 

[theRevolt]

If you don't mind using Bing search results, they have a Search API which would negate the need for scraping search results entirely.

I think his point is he wants the data from Google to see what is ranking there for top keywords, vs Bing which we know is still less about quality than Google (mostly)

 

[The Curator]

Scrapebox is the right tool for this and it costs just $67 : http://www.scrapebox.com/bhw

Or do you want a custom coded bot/script?

P.S I'm not applying for the job as it's against BHW TOS. It's just a general discussion in open public.

I have Scrapebox, but didn't find it surgical enough, tried to limit the harvesting options and noticed it kept going over what I needed...

 

[loopline]

I didn't find that scrapebox could be so surgical with harvesting, usually limiting number of results still didn't work, and the results weren't in order.

Requirements:

• Scrape X number of search results from Google (X being a variable number)
• Scrape results for X number of keywords (for projects that have multiple keywords)
• Take out duplicate url's

I use this for content development, to put together for our researchers who distill the info down for our writers. The variability of both the number of keywords and the number of search results can allow people to research one keyword/topic lightly or heavily - even allowing for silo/skyscraper content research.

Appreciate it the help with this!

Scrapebox can do this. The detailed harvester will keep the results in order they are scraped and then also you can save off the keyword with the url, if you like.

 

[MKSKS]

http://urlprofiler.com/free-tools/

This is a free tool from url profiler but doesn't have much options other than scraping serp results.

 

[The Curator]

http://urlprofiler.com/free-tools/

This is a free tool from url profiler but doesn't have much options other than scraping serp results.

Nice thank you for posting that!

 

[The Curator]

http://urlprofiler.com/free-tools/

This is a free tool from url profiler but doesn't have much options other than scraping serp results.

They had a blog post that showed how to use Linkclump to do this manually, pretty easy, free, and without the use of proxies...

 

[The Curator]

It took the free scraper 9 and half minutes to scrape 50 search results for 9 keywords using random delay. The tool also has the option to choose which country for Google you scrape. Compared to the plugin Linkclump, it's slower, but it's a nice set it and forget it option. Again when I used scrapebox I would tell it not to scrape past 50 results for a given keyword, and I would get random numbers, sometimes more 25%+ more than I asked - so I think I will use this free scraper...
Also, if you're scraping a list of keywords, order them first in order of greater search volume descending, so that when you take out duplicates in excel, it takes out the duplicate results from the lesser keywords - preserving the natural order of domain authority for the URL in relation to the entire keyword group.

Kuddos @akssiv2007 for finding this for me, saved me some money!

 

[loopline]

It took the free scraper 9 and half minutes to scrape 50 search results for 9 keywords using random delay. The tool also has the option to choose which country for Google you scrape. Compared to the plugin Linkclump, it's slower, but it's a nice set it and forget it option. Again when I used scrapebox I would tell it not to scrape past 50 results for a given keyword, and I would get random numbers, sometimes more 25%+ more than I asked - so I think I will use this free scraper...
Also, if you're scraping a list of keywords, order them first in order of greater search volume descending, so that when you take out duplicates in excel, it takes out the duplicate results from the lesser keywords - preserving the natural order of domain authority for the URL in relation to the entire keyword group.

Kuddos @akssiv2007 for finding this for me, saved me some money!

If you want to give a screenshot I can probably help you as there must be a setting wrong. Scrapebox granularly respects the scraping limit. If I put 37, I might get less then 37 results if there are less, but never more.


Your on mac or windows?

 

[MKSKS]

It took the free scraper 9 and half minutes to scrape 50 search results for 9 keywords using random delay. The tool also has the option to choose which country for Google you scrape. Compared to the plugin Linkclump, it's slower, but it's a nice set it and forget it option. Again when I used scrapebox I would tell it not to scrape past 50 results for a given keyword, and I would get random numbers, sometimes more 25%+ more than I asked - so I think I will use this free scraper...
Also, if you're scraping a list of keywords, order them first in order of greater search volume descending, so that when you take out duplicates in excel, it takes out the duplicate results from the lesser keywords - preserving the natural order of domain authority for the URL in relation to the entire keyword group.

Kuddos @akssiv2007 for finding this for me, saved me some money!

Nice to see that helped you. Didn't know about Linkclump, will check that now.

 

[kahuna74]

If you don't mind using Bing search results, they have a Search API which would negate the need for scraping search results entirely.

Not free, but then again neither is scrapebox:
https://azure.microsoft.com/en-us/pricing/details/cognitive-services/search-api/web/