Need Link Checking Tool

Hello all,
I have a HUGE list of url's, and looking for a tool (winblows or linux) that I can take this list to check for valid urls from a site. This is a site that requires a user/pass to see these urls.

I have a site I need to scrape, but the site creates the urls with a random number. I create 10M urls (yes this big, as I've seen them in the high 9M area), but need to create a list of ONLY valid urls, so I can then scrape the data I need, from this smaller list (thinking it will only be around 8,000 valid links in the end)

Invalid urls are redirected to my 'home' page while logged in.

Any help would be great.

Any one have any suggestions?

valid urls usually will return 200 http response , non valid will return 404 http response , you don't need login details to check that ... i think you could check with SB live checker ...

Whatthe? said:

Any one have any suggestions?

Click to expand...

thetrustedzone said:

valid urls usually will return 200 http response , non valid will return 404 http response , you don't need login details to check that ... i think you could check with SB live checker ...

Click to expand...

I was able to find a script in python, but as the site doesn't redirect (page populates, but all datasets are blank), it's considered valid.

I do indeed need to be logged in as user to validate url. If not logged in, is redirected to sign-in page. If logged in... populates the page, but shows no data.

Have you tried https://www.screamingfrog.co.uk/seo-spider/