How do I scrape a tag forum directory ?

[stallion45d]

Hi , guys. I would like to scrape the tags directory from a xenforo forum. To be more specific , I will use BHW itself as an example:

https://www.blackhatworld.com/tags
Well, you can see that there are already many existing tags , such as

https://www.blackhatworld.com/tags/clickbank/https://www.blackhatworld.com/tags/affiliate/https://www.blackhatworld.com/tags/seo
and thousands more.

Is there any way to scrape all the existing tags from a forum ? Would this be like scrapping the full directory https://www. forumIwanttoscrape .com/tags ? I have a scrapebox license , but honestly , I haven´t used much since I bought 10 years ago , so I am not really skilled when It comes to use this software

I would really appreciate any help . Thank you so much in advance

 

[WebGenix]

Hi,

The easiest way is to find the sitemaps of the website you want to scrape,
You will need to get all the URLs from that website, and you can filter the URLs based on /tags/
I think that's the best way to do it in your case.

 

[stallion45d]

Hi,

The easiest way is to find the sitemaps of the website you want to scrape,
You will need to get all the URLs from that website, and you can filter the URLs based on /tags/
I think that's the best way to do it in your case.

thank you for the quick reply , WebGenix.

Any suggestion on how to get the sitemap of that forum ?

 

[WebGenix]

thank you for the quick reply , WebGenix.

Any suggestion on how to get the sitemap of that forum ?

Check www.website.com/sitemap.xml

 

[stallion45d]

Check www.website.com/sitemap.xml

I checked it

"This XML file does not appear to have any style information associated with it. The document tree is shown below."

the tree just contains the basic structure of the subforums.

 

[WebGenix]

I checked it

"This XML file does not appear to have any style information associated with it. The document tree is shown below."

the tree just contains the basic structure of the subforums.

PM me the link to the website and I will try to help you find the links you want.

 

[stallion45d]

pm sent !

 

[hideath]

I checked it

"This XML file does not appear to have any style information associated with it. The document tree is shown below."

the tree just contains the basic structure of the subforums.

check robots.txt,
Xenforo uses domain.com/sitemap.php btw

 

[stallion45d]

Hi, just found That scrapebox can crawl a whole site. Then filter the results as " delete urls not containing " /tags and you are done.

Thanks for all the help given !!