Best Method to Scrape Images From a Site Using Scrapebox?

[BlackHatInTraining]

Okay, I'm familiar with a variety of free stock photo websites. Problem is, I want to scrape photos from these sites.

I'm also familiar with using Scrapebox to scrape these images (even though they may not be on google, per se). Why not scrape Google? Google is messy and you get branded images from there (which I want to avoid) and it's too much work to curate, which must be done manually.

How do I scrape images from these stock photo sites quickly with a keyword to target groups of image URLs (without having to click, click, click and right click to save all day long)?

 

[Roger Marquez]

Crawl them using Screaming Frog and then mass/bulk upload those images using a third-party app or build a custom one yourself.

I am not a programmer but I am fairly sure you can build something in Python that is NOT too complicated.
Look for this on Google.

EDIT: I run a search on Google and found this free app that is supposed to download images in bulk from an URL list...

https://www.wfdownloader.xyz/
...Check this tutorial...

https://www.wfdownloader.xyz/blog/how-to-batch-download-a-list-of-urls
DISCLAIMER: I haven´t tested it the app so, I don´t know if it works or not.

Let me know how it goes!

 

[MKSKS]

Unsplash & pexels do have an api which you can use.
Or get a scraper build.
I use both api and scraper based on my use case.
I have recently built one that scrapes images from blogs & then uploads to Imgur then uses it as blog & featured image on my autoblog.

 

[BlackHatInTraining]

Crawl them using Screaming Frog and then mass/bulk upload those images using a third-party app or build a custom one yourself.

I am not a programmer but I am fairly sure you can build something in Python that is NOT too complicated.
Look for this on Google.

EDIT: I run a search on Google and found this free app that is supposed to download images in bulk from an URL list...

https://www.wfdownloader.xyz/
...Check this tutorial...

https://www.wfdownloader.xyz/blog/how-to-batch-download-a-list-of-urls
DISCLAIMER: I haven´t tested it the app so, I don´t know if it works or not.

Let me know how it goes!

Thanks, Roger! I want to bulk download 500-1000 images or more at once. That's far too much work in those apps - there is a limit of 20.

 

[akaseo]

# First Section: Importing Libraries
import os
import requests
from bs4 import BeautifulSoup

# Second Section: Declare important variables
google_image = "https://www.google.com/search?site=&tbm=isch&source=hp&biw=1873&bih=990&"

user_agent = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
}

# Third Section: Build the main function
saved_folder = 'images'


def main():
    if not os.path.exists(saved_folder):
        os.mkdir(saved_folder)
    download_images()


# Fourth Section: Build the download function
def download_images():
    data = input('What are you looking for? ')
    n_images = int(input('How many images do you want? '))

    print('searching...')

    search_url = google_image + 'q=' + data

    response = requests.get(search_url, headers=user_agent)

    html = response.text

    soup = BeautifulSoup(html, 'html.parser')

    results = soup.findAll('img', {'class': 'rg_i Q4LuWd'})

    count = 1
    links = []
    for result in results:
        try:
            link = result['data-src']
            links.append(link)
            count += 1
            if(count > n_images):
                break

        except KeyError:
            continue

    print(f"Downloading {len(links)} images...")

    for i, link in enumerate(links):
        response = requests.get(link)

        image_name = saved_folder + '/' + data + str(i+1) + '.jpg'

        with open(image_name, 'wb') as fh:
            fh.write(response.content)


# Fifth Section: Run your code
if __name__ == "__main__":
    main()

credit - https://python.plainenglish.io/how-to-automatically-download-bulk-images-for-your-dataset-using-python-f1efffba7a03

Read this it has your solution if you have doubts let me know

 

[BlackHatInTraining]

# First Section: Importing Libraries
import os
import requests
from bs4 import BeautifulSoup

# Second Section: Declare important variables
google_image = "https://www.google.com/search?site=&tbm=isch&source=hp&biw=1873&bih=990&"

user_agent = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
}

# Third Section: Build the main function
saved_folder = 'images'


def main():
    if not os.path.exists(saved_folder):
        os.mkdir(saved_folder)
    download_images()


# Fourth Section: Build the download function
def download_images():
    data = input('What are you looking for? ')
    n_images = int(input('How many images do you want? '))

    print('searching...')

    search_url = google_image + 'q=' + data

    response = requests.get(search_url, headers=user_agent)

    html = response.text

    soup = BeautifulSoup(html, 'html.parser')

    results = soup.findAll('img', {'class': 'rg_i Q4LuWd'})

    count = 1
    links = []
    for result in results:
        try:
            link = result['data-src']
            links.append(link)
            count += 1
            if(count > n_images):
                break

        except KeyError:
            continue

    print(f"Downloading {len(links)} images...")

    for i, link in enumerate(links):
        response = requests.get(link)

        image_name = saved_folder + '/' + data + str(i+1) + '.jpg'

        with open(image_name, 'wb') as fh:
            fh.write(response.content)


# Fifth Section: Run your code
if __name__ == "__main__":
    main()

credit - https://python.plainenglish.io/how-to-automatically-download-bulk-images-for-your-dataset-using-python-f1efffba7a03

Read this it has your solution if you have doubts let me know

Thanks, akaseo. I don't know Python.

 

[akaseo]

No problem but if you don't find any solution just search on youtube for how to install python and when you do it successfully then search how to install python modules. After that just follow the blog post step by step.

Thanks, akaseo. I don't know Python.

 

[BlackHatInTraining]

Okay, so I have the Screaming Frog method working. Scrapebox is ideal because of the volume of images you can download.

How do I target just the main image on the page to download as opposed to all other images on that page as well that Screaming Frog pulls through the crawl?

 

[Akension]

Okay, I'm familiar with a variety of free stock photo websites. Problem is, I want to scrape photos from these sites.

I'm also familiar with using Scrapebox to scrape these images (even though they may not be on google, per se). Why not scrape Google? Google is messy and you get branded images from there (which I want to avoid) and it's too much work to curate, which must be done manually.

How do I scrape images from these stock photo sites quickly with a keyword to target groups of image URLs (without having to click, click, click and right click to save all day long)?

You should be able to find a working bot on GitHub.

 

[arcadiafire]

I was looking for the same thing and found a program called Looksy here: hxxp(s)://w3 . sparkbyte.ca/products/looksy-smart-web-scraper. Not free, but I think it does what you want. It worked well for my purposes. I had also tried Beautiful Soup and Python code, but I'm not a coder so it was just a mess. Looksy was money well spent.