Regex to capture content or text after H1?

The Curator · Jan 4, 2017

I am using a cool scraping software, webharvy that will scrape title, meta, url, but I am having a tough time scraping the content of the page and I was thinking of doing it by identifying via regex any text/content found on the page after the H1 tag. I just don't understand regex to formulate this myself. Appreciate any help with it!

Sam Green · Jan 4, 2017

it's generally considered bad form to regex html. I think the first answer here does a good job of explaining why:

http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

tldr: Use a parser actually made for parsing html

The Curator · Jan 5, 2017

Sam Green said:
it's generally considered bad form to regex html. I think the first answer here does a good job of explaining why:

http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

tldr: Use a parser actually made for parsing html

Yeah I saw someone recommending python to parse the html. I dont have those skills and webharvy doesnt utilize that option unfortunately.

Sam Green · Jan 5, 2017

depending on how challenging the site is you trying to scrape i might write something up. post a link or pm me.

mynameisfrankenstein · Jan 5, 2017

If there a tag that contains what you want to scrape?

You could use <div>(.*?)</div>

Jomasdf · Jan 5, 2017

The Curator said:
Yeah I saw someone recommending python to parse the html. I dont have those skills and webharvy doesnt utilize that option unfortunately.

What he said. Are you making it yourself? I have no idea what webharvy is, but going with a HTML parser + xpaths is a great way to do it.

The Curator · Jan 5, 2017

Jomasdf said:
What he said. Are you making it yourself? I have no idea what webharvy is, but going with a HTML parser + xpaths is a great way to do it.

Unfortunately that's not in my bag of skill sets. Here is webharvy https://webharvy.com/

Jomasdf · Jan 5, 2017

The Curator said:
Unfortunately that's not in my bag of skill sets. Here is webharvy https://webharvy.com/

Ok, well maybe test this one then - <h1>(.*?)<\/h1>
https://regex101.com/r/bOEnwp/1

fastlinks · Jan 10, 2017

xpath will change over time, but regex will always get what you need exactly

try this:

(?<=\<h1\>).*?(?=\<\/h1\>)

mynameisfrankenstein · Jan 10, 2017

How is your content scraping and posting bot coming along?

I'm quite interested in how it all works out for you.

frenchboy · Jan 10, 2017

(.+) means everything so <h1>(.+)</h1> will capture everything in the title tag. if you want after you can do </h1>(.+)

fastlinks · Jan 25, 2017

when you cant make own regex, try search on regexlib.com for existing regex

seogibbon · Jan 28, 2017

The Curator said:
I am using a cool scraping software, webharvy that will scrape title, meta, url, but I am having a tough time scraping the content of the page and I was thinking of doing it by identifying via regex any text/content found on the page after the H1 tag. I just don't understand regex to formulate this myself. Appreciate any help with it!

You can easily find id of tag in HTML and parse text by Id.

Can share this tool with bhw members - just send me send me request in PM.

Crawlie · Feb 5, 2017

Everything after the h1 tag : (?s)(?<=</h1>).* (python, java)
https://regex101.com/r/8yAinc/1

Regex to capture content or text after H1?

The Curator

Elite Member

Sam Green

Junior Member

The Curator

Elite Member

Sam Green

Junior Member

mynameisfrankenstein

Regular Member

Jomasdf

Power Member

The Curator

Elite Member

Jomasdf

Power Member

fastlinks

Power Member

mynameisfrankenstein

Regular Member

frenchboy

Power Member

fastlinks

Power Member

seogibbon

Newbie

Crawlie

Registered Member

Main Menu

Marketplace

Making Money

BlackHat World