[CHRISTMAS GIFT] Simple Google Scraper

bimbimo · Feb 23, 2012

kokoloko75, thx bro! for ur usefull tool!

There is one important item Miss! Can u add plz checkbox, for saving "full url" from google. For example, i search "inurl:********.php?pid= site:*.ua" and i wanna get urls like "site.ua/page.php?pid=34&lang=1","site.ua/page.php?pid=1278".

WE(security guards

) rly need this

Plz add this

Thx.

kokoloko75 · Feb 24, 2012

bimbimo said:
kokoloko75, thx bro! for ur usefull tool!

There is one important item Miss! Can u add plz checkbox, for saving "full url" from google. For example, i search "inurl:********.php?pid= site:*.ua" and i wanna get urls like "site.ua/page.php?pid=34&lang=1","site.ua/page.php?pid=1278".

WE(security guards ) rly need this Plz add this

Thx.

Understood nothing, sorry

EDIT : What you ask is a basic feature, present since the version... 1.0.

Beny

royserpa · Feb 24, 2012

merry christmas and happy new year 2013!!
(too early, i know) but ftw

bimbimo · Feb 24, 2012

kokoloko75 said:
Understood nothing, sorry

EDIT : What you ask is a basic feature, present since the version... 1.0.

Beny

sry for my albanian english.

but... look...
keywords:

Code:

inurl:order.php?id= site:*.ua

result:
***.ua/order.php
***.ua/order.php
***.ua/order.php

BUT NEEDED THIS:
***.ua/order.php?id=3
***.ua/order.php?id=1
***.ua/order.php?id=371

Need full link, with parametrs...

Make it plz.....

kokoloko75 · Feb 24, 2012

bimbimo said:
sry for my albanian english.

but... look...
keywords:

Code:

inurl:order.php?id= site:*.ua

result:
***.ua/order.php
***.ua/order.php
***.ua/order.php

BUT NEEDED THIS:
***.ua/order.php?id=3
***.ua/order.php?id=1
***.ua/order.php?id=371

Need full link, with parametrs... Make it plz.....

Ah, I understood

It's a bug due to Google changes.
Before, the software scraped full links (otherwise it's useless, you're right).

I'll fix that soon.

Beny

bimbimo · Feb 24, 2012

kokoloko75 said:
Ah, I understood
It's a bug due to Google changes.
Before, the software scraped full links (otherwise it's useless, you're right).

I'll fix that soon.

Beny

thx bro! we are waiting

kokoloko75 · Feb 24, 2012

Update (1.4.1) :
- Bug fix : Google changes

I changed the engine scraping process.
Now, it should be bulletproof for a lot of future Google changes (or anyway easy to update).

Download (unchanged) :

Code:

http://bit.ly/SimpleGoogleScraper

Beny

bimbimo · Feb 24, 2012

Ur best!
But now one more thing...

need to exlude noneeded url. regular expression cant help when there are keywords with diff parametrs.

try to scrap with keyword: inurl:"order.php?id=" site:*.ua
u will get urls also with no parametrs and even url "schema d.o.t org s.l.a.s.h WebPage".

seems smthng going wrong
thx.

kokoloko75 · Feb 24, 2012

bimbimo said:
Ur best!
But now one more thing...

need to exlude noneeded url. regular expression cant help when there are keywords with diff parametrs.

try to scrap with keyword: inurl:"order.php?id=" site:*.ua
u will get urls also with no parametrs and even url "schema d.o.t org s.l.a.s.h WebPage".

seems smthng going wrong
thx.

You can use the function "Remove URLs Not Containing" with as filter : order.php?id

Beny

haleluo · Feb 24, 2012

Really thank you, I am newbie,
Can you please tell me how to use it?

templaries · Feb 24, 2012

who_was_phone · Feb 24, 2012

Thanks it is very nice

bimbimo · Feb 24, 2012

kokoloko75 said:
You can use the function "Remove URLs Not Containing" with as filter : order.php?id

Beny

try to read FULL text.
i said

need to exlude noneeded url. regular expression cant help when there are keywords with diff parametrs.

for example, i have 1k keywords
part of them:
inurl:List.php?CatID= site:*.ua,
inurl:listcategoriesandproducts.php?idCategory= site:*.ua,
inurl:modline.php?id= site:*.ua,
inurl:myaccount.php?catid= site:*.ua,
inurl:news.php?id= site:*.ua,
inurl

rder.php?BookID= site:*.ua,
inurl

rder.php?id= site:*.ua,

so how u will use ur "Remove URLs Not Containing"?? ha???
thats why i ask to make clear parse.

kokoloko75 · Feb 24, 2012

bimbimo said:
try to read FULL text.
i said

...

so how u will use ur "Remove URLs Not Containing"?? ha???
thats why i ask to make clear parse.

Haaa ?!

I'll try to improve the parsing engine.

Thanks for your feedback.

Beny

bimbimo · Feb 24, 2012

Sry Beny =)
thought it was other man. didnt read nick

thats why little "agression" :1:

zenglihunter · Feb 25, 2012

It's really good toolï¼Thks

jpgaga · Feb 25, 2012

You are a genius. Thanks.

tiuz89 · Feb 25, 2012

Thanks. This is a really great tool.
But I have one problem. I have 10 working proxies, but it seems as if the program just scrapes without proxies.
When I google myself from the local ip i get the google captcha (IP blocked), when I google with proxy i get no captcha.

I'm using version 1.4.1

I think it would be helpfull if there would be something like a small debug-box where u see what the programm is currently trying (scraping "keyword" with ip 127.0.0.1).

I made a changelog for you. cos i had to search through all posts, maybe this helps other people to know what the software is able to do.

Update (1.4.1) :

Code:

- Bug fix : Google changes

Update (Version 1.4) :

Code:

- Feature added : Ability to load URLs from a file
- Feature added : Remove entries not containing...
- Minor bugs fixed
- Stability and performance improved...

Update (Version 1.3) :

Code:

- Addon : PageRank checker
- Addon : Google Results Number Scraper
- Addon : Regular Expression Extractor
- Feature added : Remove entries containing...
- Feature added : Support all Google locations
- Bug fix : Regex extractor no results
- Bug fix : latest Google update
- Bug fix : Google Results Number Scraper no results
- Minor bugs fixed
- Stability and performance improved...

Update (Version 1.2.1) :

Code:

- Win XP Button fix

Update (Version 1.2) :

Code:

- Google Results Number Scraper (scrape the number of results of a keyword)
- Regular Expression Extractor (extract any element of a page)
- Several improvements...

Update V1.1 :

Code:

- Proxy rotation
- Google period (anytime or past 24h)
- Choose number pages to parse (1-100)
- Improved performances...

Features 1.0:

Code:

- Fast!
- Full threaded
- Support connection via a proxy
- Support delay between queries
- Support advanced search terms (inurl, intitle, site, ...)
- Intelligent scraper engine to avoid Google ban
- Portable application (single executable)
- Quick start help file included (and tooltips on fields/buttons)

kokoloko75 · Feb 25, 2012

Thank you Tiuz for the changelog

The debug box is a good idea but it's a bit too much work, and for the moment I'm barely busy.
But I keep the idea for future.

What I plan to add in the next version will be :
- Feature added : Updates checker
- Feature added : Google ban detection (warning)
- Addon : Proxy harvester
- Addon : Google proxy checker
- Addon : Links extractor (in/out)
- Bug fix : Remove Selected
- Bug fix : Scraping multiple pages
- Bug fix : Scraping with proxies

Beny

randyya · Mar 1, 2012

Great tool Beny,, it works great for me...and i give my first rep on BHW to you..

[CHRISTMAS GIFT] Simple Google Scraper

Newbie

Elite Member

BANNED

Newbie

Elite Member

Newbie

Elite Member

Newbie

Elite Member

Newbie

Newbie

Newbie

Newbie

Elite Member

Newbie

Newbie

BANNED

Newbie

Elite Member

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World