want to make tool that scrape 10 first results from google

forvenz

forvenz

Regular Member
Joined
Jan 25, 2013
Messages
471
Reaction score
77
Hi ,
I prefer to do that with javascript or with java , but if there is a better language that suit for this job so no problem .
I want to make a tool that get a input (X numbers of keywords ) and return the 10 first url in google to each of these keywords .
how can I do that? Do I need to use google api?
 
Advertise on BHW
Or try Market Samurai....it does exactly what you want
 
Install Scraper for Google Chrome if you are planning to use it for personal use .
Python has better libraries for scraping & you can convert it to windows program easily or run on linux .
also try zenno and ubot .
 
Any language is good for doing this task... this is way too simple.

You just need to know programming or hire someone to do it for you.
 
I know little bit programming but I need a direction how to do that ,
and with scrape box it won't scrape the 10 first results
 
someone ???:confused:?
 
What you can do is to look at the google result html code.

e.g.
when using Firefox, the google result has the
Code: 
<div id="ires" data-async-context="query:your_query_string">
<ol id="rso" eid="lkdjhaaifubiuad">
<li class="g">
<div data-hveid="44" class="rc"><span class="altcts"></span><h3 class="r"><a href="......">1st result title</a></h3>
</li>

// ......  the <li> will repeat for each result

</ol>
</div>

With this known format, you can sieve through the source by looking for the string data-async-context and extract the result as you wish from there on.


hth
 
yes but how can I access the data?
what function give me access to the data of a web page?
 
This task can be done in many different languages. If i were you I would use Visual Basic,
You can take advantage of the webbrowser component. Simply make it navigate to the page you'd like
and then it's all about parsing the HTML document in the webbrowser.
 
forvenz said:
yes but how can I access the data?
what function give me access to the data of a web page?
Click to expand...

If you are using PHP, then you can use cURL.

If you are using java, then URLConnection. Or, if you are using a library, use HttpClient

If you are using javascript, then you need to use an ajax call using XMLHttpRequest

hth
 
Last edited:
Better to use specialized software. Because after simple script development would be needed proxy support, captcha filling and etc
 
it can be done by most languages, you want to code it from scrach ? or you are searching for available softwares which do the task ? cause the most of tools do it...
 
You will find a full featured PHP source for scraping Google at http://google-rank-checker.squabbel.com
To only scrape the 10 first results you need to set it to 1 result page and the number of results per page to 10.

It is open source so you can use the code for your stuff but converting that one to javascript or java would be difficult, but after all PHP is a very nice language and easy to understand.
 
Not sure how you would scrape google with just javascript, but I did make a keyword tracker project using PHP which can be useful here. You will need to modify it to return the urls instead of the position of your domain. The basic fetch algo will remain the same.
Check that thread here:
http://www.blackhatworld.com/blackh...serp-position-checker-script-written-php.html

Let me know if you are unsure about something there. :)
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top