Software that checks if websites are up and running

Adivertising

Regular Member

Aug 25, 2019

216

165

• Apr 29, 2023

• #1

I have about 3 million business data with their website in a column and want to filter out the domains that are offline, and preferably if they haven't been updated in x amount of years. Any ides how to achieve that?

 

SeasonedCode

Senior Member

Jr. Executive VIP

Jr. VIP

Oct 24, 2022

1,112

1,347

• Apr 29, 2023

• #2

Python is the answer

Use pandas lib to deal with data
Use requests to see if the website is returning 200 response

Then you are good

 

tazarbm

Elite Member

Executive VIP

Jr. VIP

Oct 28, 2020

10,370

12,331

• Apr 29, 2023

• #3

scrapebox has an addon for this:
https://www.scrapebox.com/mass-alive-checker

 

Gogol

Moderator

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,658

26,171

• Apr 29, 2023

• #4

Adivertising said:

I have about 3 million business data with their website in a column and want to filter out the domains that are offline, and preferably if they haven't been updated in x amount of years. Any ides how to achieve that?

Click to expand...

Make http calls, and see the last modification header, if returned (catch the exceptions with a retry count and add them to offline domains after a few tries). This is the most straightforward way I can think of, but there could be more nuances.

 

Adivertising

Regular Member

Aug 25, 2019

216

165

• Apr 29, 2023
• Thread Starter Thread Starter

• #5

courses_retailer , tazarbm , BlurryBit​

Thank you for your replies. I ended up doing this using Screaming Frog. Plain and simple response codes and export filtered list. Broken links are a gold mine for getting people who already started a business and failed, to ignite their will of starting a new business and buying my services (web dev)