Google search scraper help (Python)

chrispliakos · Sep 18, 2017

Well, I'm struggling to find and write a Python script that will let me scrape some results from Google
Especially I need the script to search ex: uk travel agents "gmail.com"
And then filter and save all the emails that it can find.
I can do it manually with chrome plugins that will show me all the emails from this search but its a pain in the ass.
If someone can help me develop this or if you have something ready (for free) please reply.

Khakhat · Sep 18, 2017

what you exactly need do PM me. I will try to finish this bot by today

DrPorn · Sep 19, 2017

Depends on how you want to tackle this.

Do you want to use their API? Are you aware there are tools to do this already?

https://github.com/jarun/googler

You shouldn't reinvent the wheel

chrispliakos · Sep 19, 2017

DrPorn said:
Depends on how you want to tackle this.

Do you want to use their API? Are you aware there are tools to do this already?

https://github.com/jarun/googler

You shouldn't reinvent the wheel

I think that doesnt work for windows.

Staz · Sep 19, 2017

You can find with PHP on google easily.

DrPorn · Sep 19, 2017

chrispliakos said:
I think that doesnt work for windows.

Ah, you're right. Apologies I'm not a windows user.

Are you wanting to use the Google Search API or do you want to perform searches without it? Depend on your needs the solution will differ but I'm happy to help you out.

chrispliakos · Sep 19, 2017

DrPorn said:
Ah, you're right. Apologies I'm not a windows user.

Are you wanting to use the Google Search API or do you want to perform searches without it? Depend on your needs the solution will differ but I'm happy to help you out.

Its okey mate,thanks anyway.

Well I think that I read somewhere that Google's API gives a small amount of results for each search. Is it true?
I have a pretty basic Python knowledge,especially with internet/http requests etc. I can read a code but I need a lot of practice/studying to write one for this use.

I need the script to perform searches and return me some specific results.Right now I have two ways in my mind:

Perform several google searches with some google search symbols like I mentioned. Ex: Keyword 'UK Travel Agents ".co.uk" ' . With this keyword the results that will come up its likely that they have an amount of emails at their description without the need to open the urls. The script will grab those emails,I assume with the "re" library and some html searching.
Second option that I have in my mind is this spider crawl thing. I'll enter the same keyword and it will go through websites for some fixed depth or not,it will just keep searching if it finds any emails. If it doesnt it will come back to google search and it'll open another url etc etc

What do you think?

DrPorn · Sep 19, 2017

You're already using Python, so I'd suggest taking a look at Scrapy if you're wanting to create a spider/crawler. That's perfect for the job.

Regarding the first, leave it with me tonight and I will get something put together for you. I've been meaning to do a few tutorials on scraping anyway so I might use Google scraping as an example.

chrispliakos · Sep 19, 2017

DrPorn said:
You're already using Python, so I'd suggest taking a look at Scrapy if you're wanting to create a spider/crawler. That's perfect for the job.

Regarding the first, leave it with me tonight and I will get something put together for you. I've been meaning to do a few tutorials on scraping anyway so I might use Google scraping as an example.

Alright I'll check that out.
Thank you very much.

LostLife · Sep 21, 2017

For scrapping you should use scrapy. Just look at some tutorial on web and you can start using scrapy.

Google search scraper help (Python)

chrispliakos

Regular Member

Khakhat

Elite Member

DrPorn

Junior Member

chrispliakos

Regular Member

Staz

Regular Member

DrPorn

Junior Member

chrispliakos

Regular Member

DrPorn

Junior Member

chrispliakos

Regular Member

LostLife

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World