Scraping help!!!

I am making a scraper, and I almost finished it, but I am struggling with a very simple thing:
I have a list of keywords and the bot must:

1. open google and input one of the keywords

2. select "search"

3. extract the link of the first result

4. save extracted data in the same file as: "keyword" : "url"

5. ultra loop to make it several thousends times

I know that it's a very simple task, but I tried with imacros, Browser automation studio and webharvy and I wasn't able to do that... Any advice?

I would suggest python + selenium for browser automation. Code will need following functions which each is easy to find in google:
1. read file by line for input (-> List of keywords)
2. iterate thru the list
3. run scraper with keyword
4. write scrape result to file

Example for Python and selenium with Google:

Marcel Freiberg said:

I would suggest python + selenium for browser automation. Code will need following functions which each is easy to find in google:
1. read file by line for input (-> List of keywords)
2. iterate thru the list
3. run scraper with keyword
4. write scrape result to file

Example for Python and selenium with Google:

Code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://www.google.com")
input_element = driver.find_element_by_name("q")
input_element.send_keys("python")
input_element.submit()


RESULTS_LOCATOR = "//div/h3/a"

WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.XPATH, RESULTS_LOCATOR)))

page1_results = driver.find_elements(By.XPATH, RESULTS_LOCATOR)

for item in page1_results:
    print(item.text)

Click to expand...

Amazing!!! Thank you very much, you saved my life!!! haha

You will need Python and install (command pip install selenium) to run the example. You will need firefox either. Depending on the goal it Could be worth to Google „selenium headless chrome“ so the scraper will be able to run on a server without visual output.
It may also occur that you are a bit annoying to Google (to much request), so you will need a rotating proxy (e.g. Tor) or add some waits in your Code.

Happy coding
Mf