FiveForwardSlash
Registered Member
- Dec 16, 2020
- 80
- 106
Good afternoon. Recently I needed to collect all the sites from a google search on a list of keywords. I already had a project on Zennoposter which used the browser but ran into a problem with stability and speed. Spent a lot of time on page load especially if you use proxy or vpn and often does not find items on the page to click. 
But if you look carefully at Url request you can see the solution to the problem in the parameter "start".
To get the first page it is enough to query
and in the url we will see a link like this
But if we go to second page we will see parameter "start" with value 10.
In fact, the sites in the search results are ranking, where on the first page of the top 10 sites have numbers from 0 to 9. On the second page 11-20 sites will have ordinal numbers 10-19. Parameter "start" says from which place in the ranking to show you the search results. The important point is that you should change the parameter to +-10 (switch to a new page) to display search results correctly. The second page start=10, the third start=20, the fourth start=30. Other parameters were not necessary, so request for next pages will look like this.
The query itself in the template
Then we use regex to collect the links we need from the received answer.
This part of the template took me the longest, and I hope it helped save someone's time.
But if you look carefully at Url request you can see the solution to the problem in the parameter "start".To get the first page it is enough to query
Code:
https://www.google.com/search?q=Bitcoinand in the url we will see a link like this
Code:
https://www.google.com/search?q=Bitcoin&ei=l9_aYcXFEK-srgSK8YCwDg&ved=0ahUKEwiF7efx36T1AhUvlosKHYo4AOYQ4dUDCA4&uact=5&oq=Bitcoin&gs_lcp=Cgdnd3Mtd2l6EAMyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAESgQIQRgASgQIRhgAUABYAGCqBWgAcAJ4AIAB0AGIAdABkgEDMi0xmAEAoAECoAEBwAEBBut if we go to second page we will see parameter "start" with value 10.
Code:
https://www.google.com/search?q=Bitcoin&ei=b-PaYdieLYigptQPivyjkAQ&start=10&sa=N&ved=2ahUKEwiYkp_H46T1AhUIkIkEHQr-CEIQ8tMDegQIAxA8&cshid=1641735038153052&biw=1366&bih=625&dpr=1In fact, the sites in the search results are ranking, where on the first page of the top 10 sites have numbers from 0 to 9. On the second page 11-20 sites will have ordinal numbers 10-19. Parameter "start" says from which place in the ranking to show you the search results. The important point is that you should change the parameter to +-10 (switch to a new page) to display search results correctly. The second page start=10, the third start=20, the fourth start=30. Other parameters were not necessary, so request for next pages will look like this.
Code:
https://www.google.com/search?q=Bitcoin&start=10The query itself in the template
Code:
https://google.com{-Variable.search_key-}&start={-Variable.start-}Then we use regex to collect the links we need from the received answer.
This part of the template took me the longest, and I hope it helped save someone's time.
