Parsing google search by predicting Url on Get queries.

[FiveForwardSlash]

Good afternoon. Recently I needed to collect all the sites from a google search on a list of keywords. I already had a project on Zennoposter which used the browser but ran into a problem with stability and speed. Spent a lot of time on page load especially if you use proxy or vpn and often does not find items on the page to click. But if you look carefully at Url request you can see the solution to the problem in the parameter "start".

To get the first page it is enough to query

https://www.google.com/search?q=Bitcoin

and in the url we will see a link like this

https://www.google.com/search?q=Bitcoin&ei=l9_aYcXFEK-srgSK8YCwDg&ved=0ahUKEwiF7efx36T1AhUvlosKHYo4AOYQ4dUDCA4&uact=5&oq=Bitcoin&gs_lcp=Cgdnd3Mtd2l6EAMyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAESgQIQRgASgQIRhgAUABYAGCqBWgAcAJ4AIAB0AGIAdABkgEDMi0xmAEAoAECoAEBwAEB

But if we go to second page we will see parameter "start" with value 10.

https://www.google.com/search?q=Bitcoin&ei=b-PaYdieLYigptQPivyjkAQ&start=10&sa=N&ved=2ahUKEwiYkp_H46T1AhUIkIkEHQr-CEIQ8tMDegQIAxA8&cshid=1641735038153052&biw=1366&bih=625&dpr=1

In fact, the sites in the search results are ranking, where on the first page of the top 10 sites have numbers from 0 to 9. On the second page 11-20 sites will have ordinal numbers 10-19. Parameter "start" says from which place in the ranking to show you the search results. The important point is that you should change the parameter to +-10 (switch to a new page) to display search results correctly. The second page start=10, the third start=20, the fourth start=30. Other parameters were not necessary, so request for next pages will look like this.

https://www.google.com/search?q=Bitcoin&start=10

The query itself in the template

https://google.com{-Variable.search_key-}&start={-Variable.start-}

Then we use regex to collect the links we need from the received answer.

This part of the template took me the longest, and I hope it helped save someone's time.

 

[Castamir]

Let me see your pretend I understand the code you wrote. But I know it's useful.

 

[FiveForwardSlash]

Let me see your pretend I understand the code you wrote. But I know it's useful.

I understand that this post will not be equally relevant to most. But for those who make bots to automate their work or to order it can save a lot of time. Even in one project there can be a lot of such little things, and they are not complicated in nature. The main difficulty is to pay attention to details and see new things in the already clear things.

 

[Deleted member 731332]

Good afternoon. Recently I needed to collect all the sites from a google search on a list of keywords. I already had a project on Zennoposter which used the browser but ran into a problem with stability and speed. Spent a lot of time on page load especially if you use proxy or vpn and often does not find items on the page to click. But if you look carefully at Url request you can see the solution to the problem in the parameter "start".

To get the first page it is enough to query

https://www.google.com/search?q=Bitcoin

and in the url we will see a link like this

https://www.google.com/search?q=Bitcoin&ei=l9_aYcXFEK-srgSK8YCwDg&ved=0ahUKEwiF7efx36T1AhUvlosKHYo4AOYQ4dUDCA4&uact=5&oq=Bitcoin&gs_lcp=Cgdnd3Mtd2l6EAMyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAESgQIQRgASgQIRhgAUABYAGCqBWgAcAJ4AIAB0AGIAdABkgEDMi0xmAEAoAECoAEBwAEB

But if we go to second page we will see parameter "start" with value 10.

https://www.google.com/search?q=Bitcoin&ei=b-PaYdieLYigptQPivyjkAQ&start=10&sa=N&ved=2ahUKEwiYkp_H46T1AhUIkIkEHQr-CEIQ8tMDegQIAxA8&cshid=1641735038153052&biw=1366&bih=625&dpr=1

In fact, the sites in the search results are ranking, where on the first page of the top 10 sites have numbers from 0 to 9. On the second page 11-20 sites will have ordinal numbers 10-19. Parameter "start" says from which place in the ranking to show you the search results. The important point is that you should change the parameter to +-10 (switch to a new page) to display search results correctly. The second page start=10, the third start=20, the fourth start=30. Other parameters were not necessary, so request for next pages will look like this.

https://www.google.com/search?q=Bitcoin&start=10

The query itself in the template

https://google.com{-Variable.search_key-}&start={-Variable.start-}

Then we use regex to collect the links we need from the received answer.

This part of the template took me the longest, and I hope it helped save someone's time.

Please share your regex for scraping the results.

 

[FiveForwardSlash]

Please share your regex for scraping the results.

I replied to you in a personal message with additions.I have a regular expression list working in my project.

 

[Deleted member 731332]

I replied to you in a personal message with additions.I have a regular expression list working in my project.

Yes, thank you very much. Obviously I haven’t implemented it yet, but I will be shortly. Im sure your template will help me in a few other projects as well, so thanks again.