How to Scrape Goodreads

[Ecodor]

I would be very glad if someone tell me the tool to scrape Goodreads i am trying with python but i can't do it since i dont know the language. There was a method with "Inspect element" function in browser but i dont remember it.
Thank you guys
Here is the code i got from internet.

import requests
import csv
from bs4 import BeautifulSoup as bs
import urllib
import os
def scrape_and_run(genre):
    # scrape on goodreads.com using desire genre type or keyword
    # and save the titles and autors in a csv file
    page = requests.get("https://www.goodreads.com/shelf/show/" + genre)
    soup = bs(page.content, 'html.parser')
    titles = soup.find_all('a', class_='bookTitle')
    authors = soup.find_all('a', class_='authorName')


    image_dir = os.getcwd() + "/images/" + genre

    ## check if the desire genre path exists
    ## create a new one if it doesnt
    if not os.path.exists(image_dir):
        os.makedirs(image_dir)

    with open(genre + '.csv', 'w') as csvfile:
        fieldnames = ['title', 'author']
        csv_write = csv.DictWriter(csvfile, fieldnames=fieldnames)
        books_save = 0

        for title, author in zip(titles, authors):

            try:
                ## single book page
                book_page = requests.get("https://www.goodreads.com" + title['href'])
                soup = bs(book_page.content, 'html.parser')
                # get image id
                image = soup.find('img', id='coverImage')

                title_name = title.get_text()

                save_dir = image_dir + "/" + title_name
                urllib.request.urlretrieve(image['src'], save_dir)

                csv_write.writerow({'title': title_name, 'author': author.get_text()})
                books_save += 1
                ## error handelling for long file names
            except OSError as exc:
                if exc.errno == 36:
                    print(exc)

        print("%d %s books saved." % (books_save, genre)) # books count feedback



if __name__ == '__main__':

    ## run ifinite till user tells you to stop
    ## to avoid having to compile again and again
    while True:
        genre = input("Enter the genre (or quit to stop): ").lower() # input case lowered
        if(genre == "quit"):
            break
        else:
            print("ELSE FIRED!");
scrape_and_run(genre)

 

[GrandScraperG]

You did wrong by choosing python. Would be much easier to emulate all the requests.

 

[Ecodor]

Feels good to get back to old days of using python
So far i did this and i scraped the fuck out of GoodReads

import bs4 as bs
import urllib.request

page = 1

while page != 6:

    url = 'https://www.goodreads.com/list/show/567.Best_Chapter_Books_to_Read_Out_Loud?page=' + str(++page)

    grURL = urllib.request.urlopen(url)
    grHTML = grURL.read()
    grURL.close()

    soup = bs.BeautifulSoup(grHTML, 'html.parser')
    #grAllSpans = soup.find_all("span")
    spans = soup.find_all('span', {'itemprop' : 'name'})
    for links in spans:
        print (links.get_text())

    if page == 6:
        break
    page += 1
    #print ('pageNo:' + str(page))

I need little help tho... this code gets all book titles but it scrapes the authors also because they are written in span tags
<span itemprop="name">author name</span>

I need to escape this and just get whats inside here

<a class="bookTitle" itemprop="url" href="/book/show/803171.Skippyjon_Jones">
     <span itemprop="name">Harry Potter</span>
</a>

If anyone know how to do that i would be very glad
It would save me ton of time

 

[SpoonFeeder]

Feels good to get back to old days of using python
So far i did this and i scraped the fuck out of GoodReads

import bs4 as bs
import urllib.request

page = 1

while page != 6:

    url = 'https://www.goodreads.com/list/show/567.Best_Chapter_Books_to_Read_Out_Loud?page=' + str(++page)

    grURL = urllib.request.urlopen(url)
    grHTML = grURL.read()
    grURL.close()

    soup = bs.BeautifulSoup(grHTML, 'html.parser')
    #grAllSpans = soup.find_all("span")
    spans = soup.find_all('span', {'itemprop' : 'name'})
    for links in spans:
        print (links.get_text())

    if page == 6:
        break
    page += 1
    #print ('pageNo:' + str(page))

I need little help tho... this code gets all book titles but it scrapes the authors also because they are written in span tags
<span itemprop="name">author name</span>

I need to escape this and just get whats inside here

<a class="bookTitle" itemprop="url" href="/book/show/803171.Skippyjon_Jones">
     <span itemprop="name">Harry Potter</span>
</a>

If anyone know how to do that i would be very glad
It would save me ton of time

Hope I saved you a ton of time.

import bs4 as bs
import requests

page = 1

while page != 6:

    url = 'https://www.goodreads.com/list/show/567.Best_Chapter_Books_to_Read_Out_Loud?page=' + str(++page)

    grURL = requests.get(url)
    grHTML = grURL.text

    soup = bs.BeautifulSoup(grHTML, 'html.parser')
    booklink = soup.find_all('a', attrs={'class': 'bookTitle'})
    for links in booklink:
        bookname = links.get_text()
        booklink = links['href']
        print bookname
        print booklink
    if page == 6:
        break
    page += 1
    print ('pageNo:' + str(page))

 

[Deleted member 657006]

Hope I saved you a ton of time.

import bs4 as bs
import requests

page = 1

while page != 6:

    url = 'https://www.goodreads.com/list/show/567.Best_Chapter_Books_to_Read_Out_Loud?page=' + str(++page)

    grURL = requests.get(url)
    grHTML = grURL.text

    soup = bs.BeautifulSoup(grHTML, 'html.parser')
    booklink = soup.find_all('a', attrs={'class': 'bookTitle'})
    for links in booklink:
        bookname = links.get_text()
        booklink = links['href']
        print bookname
        print booklink
    if page == 6:
        break
    page += 1
    print ('pageNo:' + str(page))

Are you a programmer or just scrapeing?

 

[mhtro]

Why would you scrape GoodReads?

 

[SpoonFeeder]

Are you a programmer or just scrapeing?

I do both.

 

[Ecodor]

Thank you very much @SpoonFeeder lol i really didnt figure out that... i need to buy u a beer if we see in rl
Now i am upgrding the code to scarpe the categories also and maybe get the images

Why would you scrape GoodReads?

Its fun to scrape mate