Bot or Script for Scraping Search Results?

SpoonFeeder said:

That's what ScrapeBox does or do you want to scrape results not from search engines?

If you can post your requirements, I'll see if I can code one real quick and give it away for free for BHW members.

Click to expand...

• Scrape X number of search results from Google (X being a variable number)
• Scrape results for X number of keywords (for projects that have multiple keywords)
• Take out duplicate url's

Warrenway said:

If you don't mind using Bing search results, they have a Search API which would negate the need for scraping search results entirely.

Click to expand...

SpoonFeeder said:

Scrapebox is the right tool for this and it costs just $67 : http://www.scrapebox.com/bhw

Or do you want a custom coded bot/script?

P.S I'm not applying for the job as it's against BHW TOS. It's just a general discussion in open public.

Click to expand...

The Curator said:

I didn't find that scrapebox could be so surgical with harvesting, usually limiting number of results still didn't work, and the results weren't in order.

Requirements:

• Scrape X number of search results from Google (X being a variable number)
• Scrape results for X number of keywords (for projects that have multiple keywords)
• Take out duplicate url's

I use this for content development, to put together for our researchers who distill the info down for our writers. The variability of both the number of keywords and the number of search results can allow people to research one keyword/topic lightly or heavily - even allowing for silo/skyscraper content research.

Appreciate it the help with this!

Click to expand...

akssiv2007 said:

http://urlprofiler.com/free-tools/

This is a free tool from url profiler but doesn't have much options other than scraping serp results.

Click to expand...

akssiv2007 said:

http://urlprofiler.com/free-tools/

This is a free tool from url profiler but doesn't have much options other than scraping serp results.

Click to expand...

The Curator said:

It took the free scraper 9 and half minutes to scrape 50 search results for 9 keywords using random delay. The tool also has the option to choose which country for Google you scrape. Compared to the plugin Linkclump, it's slower, but it's a nice set it and forget it option. Again when I used scrapebox I would tell it not to scrape past 50 results for a given keyword, and I would get random numbers, sometimes more 25%+ more than I asked - so I think I will use this free scraper...
Also, if you're scraping a list of keywords, order them first in order of greater search volume descending, so that when you take out duplicates in excel, it takes out the duplicate results from the lesser keywords - preserving the natural order of domain authority for the URL in relation to the entire keyword group.

Kuddos @akssiv2007 for finding this for me, saved me some money!

Click to expand...

The Curator said:

It took the free scraper 9 and half minutes to scrape 50 search results for 9 keywords using random delay. The tool also has the option to choose which country for Google you scrape. Compared to the plugin Linkclump, it's slower, but it's a nice set it and forget it option. Again when I used scrapebox I would tell it not to scrape past 50 results for a given keyword, and I would get random numbers, sometimes more 25%+ more than I asked - so I think I will use this free scraper...
Also, if you're scraping a list of keywords, order them first in order of greater search volume descending, so that when you take out duplicates in excel, it takes out the duplicate results from the lesser keywords - preserving the natural order of domain authority for the URL in relation to the entire keyword group.

Kuddos @akssiv2007 for finding this for me, saved me some money!

Click to expand...

Warrenway said:

If you don't mind using Bing search results, they have a Search API which would negate the need for scraping search results entirely.

Click to expand...