Best method to scrape HTML headers from blogs/pages?

hreyhrey

Power Member
Jr. VIP
Joined
Jan 9, 2019
Messages
527
Reaction score
240
I'm looking for a way to take a completed blog/page and be able to scrape the header tags in order.

So one output could be

Title Tag: Best Candy Bars 2022
H1:Candy Bars 2022
H2:Twix Bar
H3:Is the vanilla Twix Better Than Chocolate Twix?
H2:Snickers Bar
H2:Oreo Bar

Something like that. Does such a service exist? It doesn't seem too difficult so i'm hoping there's a tool that can easily do this!
 
Code:
soup = BeautifulSoup(driver.page_source, 'lxml')

try:
    h1tag = driver.find_element_by_tag_name('h1')
    h1 = h1tag.text
    print("H1:  ",h1)

    allh1 = soup.find('h1',text=h1)
    for pp in allh1.find_next_siblings():
        if pp.name == 'h1':
            break
        if hasattr(pp,"text"):
            print("H1 Paragraph:  ",pp.text)
except Exception as e:
    print(e)
    
try:
    h2tag = driver.find_elements_by_tag_name('h2')
    h2 = []
    for rez in h2tag:
        h2t = rez.text
        print("H2:  ",h2t)
        h2.append(h2t)
        allh2 = soup.find('h2',text=h2t)

        for elt in allh2.find_next_siblings():
            if elt.name == "h2":
                break
            if hasattr(elt, "text"):
                print("H2 Paragraph:  ",elt.text)
            
except Exception as e:
    print(e)

Python, this is how I do most of the scraping. You have to edit it to make it in order, but you get the idea
 
Back
Top