Regex to capture content or text after H1?

[The Curator]

I am using a cool scraping software, webharvy that will scrape title, meta, url, but I am having a tough time scraping the content of the page and I was thinking of doing it by identifying via regex any text/content found on the page after the H1 tag. I just don't understand regex to formulate this myself. Appreciate any help with it!

 

[Sam Green]

it's generally considered bad form to regex html. I think the first answer here does a good job of explaining why:

http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

tldr: Use a parser actually made for parsing html

 

[The Curator]

it's generally considered bad form to regex html. I think the first answer here does a good job of explaining why:

http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

tldr: Use a parser actually made for parsing html

Yeah I saw someone recommending python to parse the html. I dont have those skills and webharvy doesnt utilize that option unfortunately.

 

[Sam Green]

depending on how challenging the site is you trying to scrape i might write something up. post a link or pm me.

 

[mynameisfrankenstein]

If there a tag that contains what you want to scrape?

You could use <div>(.*?)</div>

 

[Jomasdf]

Yeah I saw someone recommending python to parse the html. I dont have those skills and webharvy doesnt utilize that option unfortunately.

What he said. Are you making it yourself? I have no idea what webharvy is, but going with a HTML parser + xpaths is a great way to do it.

 

[The Curator]

What he said. Are you making it yourself? I have no idea what webharvy is, but going with a HTML parser + xpaths is a great way to do it.

Unfortunately that's not in my bag of skill sets. Here is webharvy https://webharvy.com/

 

[Jomasdf]

Unfortunately that's not in my bag of skill sets. Here is webharvy https://webharvy.com/

Ok, well maybe test this one then - <h1>(.*?)<\/h1>
https://regex101.com/r/bOEnwp/1

 

[fastlinks]

xpath will change over time, but regex will always get what you need exactly

try this:

(?<=\<h1\>).*?(?=\<\/h1\>)

 

[mynameisfrankenstein]

How is your content scraping and posting bot coming along?

I'm quite interested in how it all works out for you.

 

[frenchboy]

(.+) means everything so <h1>(.+)</h1> will capture everything in the title tag. if you want after you can do </h1>(.+)

 

[fastlinks]

when you cant make own regex, try search on regexlib.com for existing regex

 

[seogibbon]

I am using a cool scraping software, webharvy that will scrape title, meta, url, but I am having a tough time scraping the content of the page and I was thinking of doing it by identifying via regex any text/content found on the page after the H1 tag. I just don't understand regex to formulate this myself. Appreciate any help with it!

You can easily find id of tag in HTML and parse text by Id.

Can share this tool with bhw members - just send me send me request in PM.

 

[Crawlie]

Everything after the h1 tag : (?s)(?<=</h1>).* (python, java)
https://regex101.com/r/8yAinc/1