1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Verify domain names have a real website up and running

Discussion in 'Black Hat SEO Tools' started by Trident50, Dec 9, 2013.

  1. Trident50

    Trident50 Newbie

    Joined:
    Sep 5, 2012
    Messages:
    11
    Likes Received:
    4
    Hey!

    Does anyone know of way to take a list of 30,000 domain names and verify that each domain has real, live, up and running website on it?

    I guess it would like a mail verifier but instead of just checking for an mx record it would actually hit the domain and report back if a page loaded
    or if there was an error like "page not found".

    Using an email verification program won't help because it's easy enough to register a domain, setup email for it, and not have a website.

    Any thoughts????

    Thanks!
     
  2. Methuselah

    Methuselah Regular Member

    Joined:
    Nov 2, 2013
    Messages:
    218
    Likes Received:
    118
    Location:
    Tropical Paradise
    Scrapebox can do this I think. There are most likely other programs that could too, but I can't think of any at the moment.
     
  3. crazyflx

    crazyflx Elite Member

    Joined:
    Nov 9, 2009
    Messages:
    1,674
    Likes Received:
    4,825
    Location:
    http://CRAZYFLX.COM
    Home Page:
    Scrapebox has an "alive check" function where you set up certain HTTP response codes to report the URL as alive/dead.

    It returns anything that does return an HTTP response of 301 or 200 as "dead" and the others as alive. You can customize this though.

    That however is only going to be one step of the process, as plenty of your domains are going to be "parked" domains.

    After performing the above function and getting a list of domains that return as "alive" (200 response code) you'd need to perform another function to determine a live website...this is tricky though. A parked domain will return a response code of 200 even though there is not actually a website there. There isn't a form of AI sufficient enough to "look" at a website and tell you "yes, this is a real website" or "no, this isn't a real website".

    So you have to do something kind of "hacky" like either look for the word "parked" on the page or, another thought that ran through my head, check to see if there is a scrollbar present or not (thinking that if there is a scrollbar, it's a real website, if not, no website). Neither of those would return as 100% accurate though of course...just a few thoughts I had.
     
    • Thanks Thanks x 2
  4. Trident50

    Trident50 Newbie

    Joined:
    Sep 5, 2012
    Messages:
    11
    Likes Received:
    4
    That's awesome - I thought for sure I was stuck with spending a month viewing all these websites!

    Thank you both - looking Scrapebox over and it will definitely help - and has a ton of other uses too.

    Thanks!
     
  5. Methuselah

    Methuselah Regular Member

    Joined:
    Nov 2, 2013
    Messages:
    218
    Likes Received:
    118
    Location:
    Tropical Paradise
    SB is a very useful tool, lots of functionality. There is a thread here where you can get it for half price. The link is at the bottom of the post.
     
  6. Trident50

    Trident50 Newbie

    Joined:
    Sep 5, 2012
    Messages:
    11
    Likes Received:
    4
    Are there any free versions floating around out there? ;)
     
  7. Methuselah

    Methuselah Regular Member

    Joined:
    Nov 2, 2013
    Messages:
    218
    Likes Received:
    118
    Location:
    Tropical Paradise
    Unfortunately not, but it is well worth the price for all the features that it comes with.
     
  8. linnet

    linnet Junior Member

    Joined:
    Jan 4, 2013
    Messages:
    147
    Likes Received:
    15
    Occupation:
    SEO and Development
    Location:
    Brighton
    You can get SB for free man, its one of the top Tools in BH SEO .