Website Scraper/Crawler

[YoloSwagLord]

Hello Good Folks of Black Hat,

Im searching for a tool that scrapes all URLS on a specific domain.
I have been searching for weeks but so far no good. There is a tricky thing because the site requires a login.

how does this site look like?
it has the following layers

1. The main site which contains a lot of images with clickable links
2. if you click on one of those links you are in the sub category
3. in this sub category there are up to 10 links which direct u to a sub sub category
4. Now I want all the urls in the last sub sub category

I tried httrack ,Teleport Pro & multiple link extraction tool add ons for chrome.

if any of you guys knows a tool or script or what so ever please let me know.

 

[arpitagarwal82]

Use scrapebox....
Right now my scrapebox is lying idle. So if you can PM me the domain and I will scrape ll internal URLs for you.. provided its not something as big as wikipedia or bbc.

P.S. I love @Sweetfunny

 

[YoloSwagLord]

Use scrapebox....
Right now my scrapebox is lying idle. So if you can PM me the domain and I will scrape ll internal URLs for you.. provided its not something as big as wikipedia or bbc.

P.S. I love @Sweetfunny

yeah hook me up on discord YoloSwagLord#7376

 

[thetrustedzone]

Try Xenu’s Link Sleuth

 

[loopline]

Unfortunately Scrapebox won't work. Because scrapebox does not support authentication so it can not login.

 

[arpitagarwal82]

Unfortunately Scrapebox won't work. Because scrapebox does not support authentication so it can not login.

Yup.. I tried the OP's website. Can scrape the URLs behind login area.

 

[YoloSwagLord]

maybe I have to make a custom python script on selium to scrape this, not sure how but

 

[nycdude]

maybe I have to make a custom python script on selium to scrape this, not sure how but

I use Python Scrapy library for scraping content and I did a quick check and there is login functionality that can be added with more libraries. Could probably make a simple middleware to do it but Scrapy for Python is your answer.