want to make tool that scrape 10 first results from google

forvenz · Jan 6, 2014

Hi ,
I prefer to do that with javascript or with java , but if there is a better language that suit for this job so no problem .
I want to make a tool that get a input (X numbers of keywords ) and return the 10 first url in google to each of these keywords .
how can I do that? Do I need to use google api?

virtualc08 · Jan 6, 2014

Why don't you just buy Scrapebox and use that?

DiamonMike · Jan 6, 2014

Or try Market Samurai....it does exactly what you want

divok · Jan 6, 2014

Install Scraper for Google Chrome if you are planning to use it for personal use .
Python has better libraries for scraping & you can convert it to windows program easily or run on linux .
also try zenno and ubot .

Shirko · Jan 6, 2014

Any language is good for doing this task... this is way too simple.

You just need to know programming or hire someone to do it for you.

forvenz · Jan 6, 2014

I know little bit programming but I need a direction how to do that ,
and with scrape box it won't scrape the 10 first results

forvenz · Jan 6, 2014

someone ???

?

mypmmail · Jan 8, 2014

What you can do is to look at the google result html code.

e.g.
when using Firefox, the google result has the

Code:

<div id="ires" data-async-context="query:your_query_string">
<ol id="rso" eid="lkdjhaaifubiuad">
<li class="g">
<div data-hveid="44" class="rc"><span class="altcts"></span><h3 class="r"><a href="......">1st result title</a></h3>
</li>

// ......  the <li> will repeat for each result

</ol>
</div>

With this known format, you can sieve through the source by looking for the string data-async-context and extract the result as you wish from there on.

hth

forvenz · Jan 8, 2014

yes but how can I access the data?
what function give me access to the data of a web page?

Asif A Khan LONDON · Jan 8, 2014

I mustn't understand your post because most web scrapers should be able to do this or you could use iMacros if it is a small list.
https://chrome.google.com/webstore/detail/imacros-for-chrome/cplklnmnlbnpmjogncfgfijoopmnlemp?hl=en
https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd

garthor · Jan 8, 2014

This task can be done in many different languages. If i were you I would use Visual Basic,
You can take advantage of the webbrowser component. Simply make it navigate to the page you'd like
and then it's all about parsing the HTML document in the webbrowser.

mypmmail · Jan 8, 2014

forvenz said:
yes but how can I access the data?
what function give me access to the data of a web page?

If you are using PHP, then you can use cURL.

If you are using java, then URLConnection. Or, if you are using a library, use HttpClient

If you are using javascript, then you need to use an ajax call using XMLHttpRequest

hth

tratata · Jan 13, 2014

Better to use specialized software. Because after simple script development would be needed proxy support, captcha filling and etc

termseo · Jan 13, 2014

it can be done by most languages, you want to code it from scrach ? or you are searching for available softwares which do the task ? cause the most of tools do it...

bytzu · Jan 14, 2014

forvenz said:
I know little bit programming but I need a direction how to do that ,
and with scrape box it won't scrape the 10 first results

You can harvest the first 10 results from G with scrapebox. You can see it here:

http://www.youtube.com/watch?v=LyCLfL_ffqQ

Go to min 2.46

Good luck

justone · Mar 18, 2014

You will find a full featured PHP source for scraping Google at http://google-rank-checker.squabbel.com
To only scrape the 10 first results you need to set it to 1 result page and the number of results per page to 10.

It is open source so you can use the code for your stuff but converting that one to javascript or java would be difficult, but after all PHP is a very nice language and easy to understand.

jamb0ss · Mar 30, 2014

Python solution http://incolumitas.com/2013/01/06/googlesearch-a-rapid-python-class-to-get-search-results/

hamd01 · May 20, 2014

W130SN said:
I mustn't understand your post because most web scrapers should be able to do this or you could use iMacros if it is a small list.
https://chrome.google.com/webstore/detail/imacros-for-chrome/cplklnmnlbnpmjogncfgfijoopmnlemp?hl=en
https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd

Is that how you would do it?

FJX · May 21, 2014

Gogol · May 21, 2014

Not sure how you would scrape google with just javascript, but I did make a keyword tracker project using PHP which can be useful here. You will need to modify it to return the urls instead of the position of your domain. The basic fetch algo will remain the same.
Check that thread here:
http://www.blackhatworld.com/blackh...serp-position-checker-script-written-php.html

Let me know if you are unsure about something there.

want to make tool that scrape 10 first results from google

Regular Member

Supreme Member

Regular Member

Senior Member

Regular Member

Regular Member

Regular Member

Junior Member

Regular Member

Elite Member

Newbie

Junior Member

Newbie

Junior Member

Junior Member

BANNED

Junior Member

Jr. VIP

Regular Member

Jr. VIP

Main Menu

Marketplace

Making Money

BlackHatWorld

Other