How to gather URL to documents from a google search

Hi all,

For the last few weeks I have tried to develop a Python based tool to retrieve URLs to documents from the search engines by requesting queries like: "site:http://www.example.com filetypedf". Then parse the HTML and extract the link. There is a point where I get an error "429: too many requests".

When I use https://www.elevenpaths.com/labstools/foca/index.html to retrieve the same documents, it can get 1000s of documents URLs in very short time. This tools has been mentioned on the forum on this https://www.blackhatworld.com/seo/tool-for-finding-peoples-email-adress.765012/#post-7954049.

I have few questions you may can help me to find a solution....

• Do you know any command line tool to automate custom queries inquiries?
• Is it possible to automate that search using Python to get the URLs as quick as FOCA?

Thanks

You must associate each request with a unique proxy to bypass Google antibot detection.
For example, if you have 5k proxies. You should rorate over them and pick a random proxy to associate with the next request.

Not only that, eventually you'll have to start solving their captchas.