how can I bulk-check domains?

[tazarbm]

Hey guys!

Probably there are no free bulk checkers out there... I mean, that would feel almost like we would be living in a normal society were this to ever happen

But are there at least cheap ($10-20) ones?

And when I say "checkers" I don't mean availability checkers, I am referring to checking the legitimacy / quality of a domain, such as performing checks for...

- google penalties
- spam inclusion on lists like Spamhaus
- quality of backlinks, RDs, and anchors
- automatic checks of their ENTIRE history on Wayback Machine
- number of drops / ownership changes / redirects
- etc...

This sort of checks I am interested in, and it needs to be for cheap too (up to $20 per month for example, or up to $20 for 100k domains, this sort of limits as I will be checking possibly 50k-100k domains each day, so yeah... I need POWA... UNLIMITED POWAAAA!

Anyway, please let me know what my options are for this type of "arrogant" demand...

Thanks in advance for any help, and may you live and prosper... along with me, of course, I don't want to be left out

Cheers!

 

[Shifted]

Never found a provider for that cheap and that offers that many checks for different things.

 

[Steptoe]

I don't think you'll be able to get that amount of data for $20 unfortunately. I would have a look around rapidapi and see what service offers the majority of the info you want, or maybe combine a couple: https://rapidapi.com/search?term=domain&sortBy=ByRelevance

 

[tazarbm]

Never found a provider for that cheap and that offers that many checks for different things.

I'm not surprised, I knew that I would be asking for too much before I posted, but I wanted to try my luck

I don't think you'll be able to get that amount of data for $20 unfortunately. I would have a look around rapidapi and see what service offers the majority of the info you want, or maybe combine a couple: https://rapidapi.com/search?term=domain&sortBy=ByRelevance

I'll check it out soon, thanks!

And yeah, I probably won't find such a good offer, but I tried poking the fate

 

[RoyRoyan]

- automatic checks of their ENTIRE history on Wayback Machine

As someone who has actually built a system like this, I can say — it’s definitely not a simple task.

Even just parsing the Wayback Machine alone is a serious challenge. Their servers are often slow, sometimes completely unresponsive, frequently throw errors or return broken data, and there are strict rate limits on how many requests you can make.

For example, checking the entire history of a single domain with 100 snapshots — and comparing content changes between them (which has to be done sequentially, not in parallel) — usually takes around 5–10 minutes and requires 30–50 rotating proxies.

In my case, my server parses around 1,000 full domain reports per hour, capped to the most recent 100 snapshots (which is enough for 80–90% of domains). Given that, your task would take around 50–100 hours of full load server.

So when we’re talking about processing 50–100k domains per day for user order, that’s actually a huge load for any public-facing tool. This kind of job really only makes sense as a self-hosted solution on dedicated infrastructure.

 

[tazarbm]

As someone who has actually built a system like this, I can say — it’s definitely not a simple task.

Even just parsing the Wayback Machine alone is a serious challenge. Their servers are often slow, sometimes completely unresponsive, frequently throw errors or return broken data, and there are strict rate limits on how many requests you can make.

For example, checking the entire history of a single domain with 100 snapshots — and comparing content changes between them (which has to be done sequentially, not in parallel) — usually takes around 5–10 minutes and requires 30–50 rotating proxies.

In my case, my server parses around 1,000 full domain reports per hour, capped to the most recent 100 snapshots (which is enough for 80–90% of domains). Given that, your task would take around 50–100 hours of full load server.

So when we’re talking about processing 50–100k domains per day for user order, that’s actually a huge load for any public-facing tool. This kind of job really only makes sense as a self-hosted solution on dedicated infrastructure.

dang! I expected this to be challenging, but this is beyond what I expected. I think that if I told someone who holds this kind of power that I'll give them $20 per month to let me use it they'd throw me out in a heartbeat

Ok, guys! I guess I'll just have to either do this manually (not for 50k domains, of course, I'll probably pick random ones that call out to me for whatever reason), or... I don't know... find other solution if it exists.

God damn it! Why is everything fighting me so hard? I swear...

 

[RoyRoyan]

Ok, guys! I guess I'll just have to either do this manually (not for 50k domains, of course, I'll probably pick random ones that call out to me for whatever reason), or... I don't know... find other solution if it exists.

What kind of domains are you checking? Where are you getting them from, and what’s the goal?

50–100k per day is honestly a lot.

 

[tazarbm]

What kind of domains are you checking? Where are you getting them from, and what’s the goal?

50–100k per day is honestly a lot.

Scrapebox's TDNAM scraper addon finds about 50k-100k domains per day that are closeouts or their auctioning ends on that day, and I want to go through these so I can find a few good ones that I can use for myself...

https://www.scrapebox.com/tdnam-closeouts-scraper

 

[HR RANKING]

Best to filter domains first, then run deeper checks only on the good ones.

 

[tazarbm]

Best to filter domains first, then run deeper checks only on the good ones.

I know, but how am I going to filter the good ones since I don't know which the good ones are? That's why I need to perform those checks

 

[HR RANKING]

Handling 1k reports per hour is impressive, that’s solid infrastructure

 

[RoyRoyan]

Scrapebox's TDNAM scraper addon finds about 50k-100k domains per day that are closeouts or their auctioning ends on that day, and I want to go through these so I can find a few good ones that I can use for myself...

I’m processing domains from GoDaddy.

Drop a couple of domains here — I’ll check them on my side.

 

[tazarbm]

I’m processing domains from GoDaddy.

Drop a couple of domains here — I’ll check them on my side.

you mean for yourself, or for me?

Because I don't need them right now as I can't buy them anyway until next week (at the earliest) and by then new domains will have popped up, so I'll probably have to check those ones instead

But if you need them for yourself I can download the list and send it to you...

 

[RoyRoyan]

you mean for yourself, or for me?

I meant — share a couple of domains you’d like to check. I’ll look them up in my database.

Since I process GoDaddy lists, I might already have info on their history.

 

[tazarbm]

I meant — share a couple of domains you’d like to check. I’ll look them up in my database.

Since I process GoDaddy lists, I might already have info on their history.

I know, I understood you from the first time

But even if you are going to check them for me I still won't be able to buy them until next week, so there's no point in checking them at the moment as I'll most likely skipped them for next week's ones.

But thanks for the offer

 

[RoyRoyan]

But even if you are going to check them for me I still won't be able to buy them until next week, so there's no point in checking them at the moment as I'll most likely skipped them for next week's ones.

I just downloaded and checked the GoDaddy Closeouts list. I already have those domains in my database with full reports, but since the auctions are marked as ended, they’re currently hidden from my interface.

I’ll work on this — and add support for searching Closeouts as well. Once it’s ready, I’ll let you test it for free.

Interested?

 

[tazarbm]

Once it’s ready, I’ll let you test it for free.

Interested?

sure

 

[Seonetworkuk]

Hey guys!

Probably there are no free bulk checkers out there... I mean, that would feel almost like we would be living in a normal society were this to ever happen

But are there at least cheap ($10-20) ones?

And when I say "checkers" I don't mean availability checkers, I am referring to checking the legitimacy / quality of a domain, such as performing checks for...

- google penalties
- spam inclusion on lists like Spamhaus
- quality of backlinks, RDs, and anchors
- automatic checks of their ENTIRE history on Wayback Machine
- number of drops / ownership changes / redirects
- etc...

This sort of checks I am interested in, and it needs to be for cheap too (up to $20 per month for example, or up to $20 for 100k domains, this sort of limits as I will be checking possibly 50k-100k domains each day, so yeah... I need POWA... UNLIMITED POWAAAA!

Anyway, please let me know what my options are for this type of "arrogant" demand...

Thanks in advance for any help, and may you live and prosper... along with me, of course, I don't want to be left out

Cheers!

At that volume, nothing solid in the $10–20 range, you’d need APIs + DIY setup, or a bigger budget.

 

[tazarbm]

At that volume, nothing solid in the $10–20 range, you’d need APIs + DIY setup, or a bigger budget.

I know, man! I mean, I expected it, I didn't necessarily know it as I've been out of the game for 5 years, so I was hoping that in the meantime something magical had come up

Oh well, I'll figure something out eventually...

 

[nicenic]

I hear you. Digging that deep usually isn’t free, but finding the right balance can make it worth it.