Scrapebox Expired Domain Finder

[UnDerDoG81]

Hi Loopline,

somehow the plugin skips domains. I guess it happens when I load a lot of domains into the list. Like more than 1000. I only scrape with crawl depth of 1 when I use such a huge list. For example I just loaded 27k domains and the loading took almost a minute and the scrape took 2 seconds. It starts slowly but then it skips after about 100 or so. I don´t use Proxies and the setting was at 30 threads. Maybe I do something wrong? As much as I know I don´t need proxies?

EDIT: I tried with 10 threads but it still happening?

 

[casperl]

Hi Loopline,

somehow the plugin skips domains. I guess it happens when I load a lot of domains into the list. Like more than 1000. I only scrape with crawl depth of 1 when I use such a huge list. For example I just loaded 27k domains and the loading took almost a minute and the scrape took 2 seconds. It starts slowly but then it skips after about 100 or so. I don´t use Proxies and the setting was at 30 threads. Maybe I do something wrong? As much as I know I don´t need proxies?

EDIT: I tried with 10 threads but it still happening?

A few weeks ago, I did a similar experiment. I put some expired domains, set crawl depth to 1 without proxies and it returned 0 expired domains. I think that it should report these as expired domains with these settings.

I see similar instabilities. For example, at every session it returns many live websites as expired domains.

It looks like the plugin is not stable enough. It is good to have such a tool for this price but if you are serious with searching for expired domains, I guess you should check other tools.
For my current state, it is enough but I plan to test other tools in the future.

And I have a question. I would like to filter all subdomains. What is the best way of doing this?

 

[Quality Copywriting]

can i have reviews copy please sample

 

[Mr Dingle]

can i have reviews copy please sample

Review copy of Scrapebox???
You should on the stage

 

[Sweetfunny]

Hi Loopline,

somehow the plugin skips domains. I guess it happens when I load a lot of domains into the list. Like more than 1000. I only scrape with crawl depth of 1 when I use such a huge list. For example I just loaded 27k domains and the loading took almost a minute and the scrape took 2 seconds. It starts slowly but then it skips after about 100 or so. I don´t use Proxies and the setting was at 30 threads. Maybe I do something wrong? As much as I know I don´t need proxies?

EDIT: I tried with 10 threads but it still happening?

It sounds like you might be using the wrong option and using the load list to check, rather than load list to crawl. So if you are loading a list of seed domains to crawl, make sure you use the option "Load URL's to crawl".

I see similar instabilities. For example, at every session it returns many live websites as expired domains.

It does a DNS lookup to first check if a domain resolves to a live website, and if not then does a whois lookup. So it should not be returning any live site with 2 separate checks in place. Please post just one single live website that it's returning and i'll make a webpage with the domain and see.

can i have reviews copy please sample

Haha yes, there's definitely not enough reviews.

 

[Mr Dingle]

Gonna give this plugin a go. I've had Scrapebox a few years. Only really used it for scraping keywords

 

[casperl]

Sweetfunny, thanks for the answer. I will check what you said about check/crawl. And after finishing this run, I will PM you alive domains that are reported as expired.

I would like to filter all subdomains. What is the best way of doing this? If not could you add an option to filter subdomains? Although I filter common subomains, I am still getting many subdomains. We can't register subdomains.

 

[UnDerDoG81]

It sounds like you might be using the wrong option and using the load list to check, rather than load list to crawl. So if you are loading a list of seed domains to crawl, make sure you use the option "Load URL's to crawl".

Nope. What I do is extract broken links from ahrefs, copy them to scrapebox -> trim to root -> remove duplicate domains -> start expired domain finder -> load urls to deepcrawl from scrapebox -> scrape on depth of 1

But somehow it works today like it should. Yesterday my internet connection was probably the problem.

Any tipps for scraping? Is it even possible to find good expireds? I always hear that it is impossible because all the good ones are catched up by pros seconds after expiring.

 

[loopline]

Nope. What I do is extract broken links from ahrefs, copy them to scrapebox -> trim to root -> remove duplicate domains -> start expired domain finder -> load urls to deepcrawl from scrapebox -> scrape on depth of 1

But somehow it works today like it should. Yesterday my internet connection was probably the problem.

Any tipps for scraping? Is it even possible to find good expireds? I always hear that it is impossible because all the good ones are catched up by pros seconds after expiring.

Depends on what you call "good".

what I do is take my keywords I want to rank for, scrape the top 15 (give or take depending) results from google for those keywords, then run the expired domain finder on them. I mean what better links to get for a domain you want to rank then links from the top urls that are already ranking. Of course from there I filter and pick the best options, but it depends on your niche as well as what your approach is.

 

[Deleted member 986121]

It does a DNS lookup to first check if a domain resolves to a live website, and if not then does a whois lookup. So it should not be returning any live site with 2 separate checks in place. Please post just one single live website that it's returning and i'll make a webpage with the domain and see.

I also get many domains which are alive in the expired domain list often for example .ch domains. Next time i save the domains.

 

[rankjuice]

I have been having the same problem with it starting slow, checking about 1000 (sometimes more, sometimes less), and then flying right though to the end of the list with no more domains identified.

I have also check using lists of expired domains. I imported 100 domains that I made sure was available using namecheaps bulk domain search, and it gave back a list of under 40 domains (no filters used on this test, filters used on the first test)

I have also checked with and without proxies

 

[rankjuice]

Ok, so I have ran another test and here is what I found:

I imported a seed list of 2779 urls to be crawled

Result: 22 expired domains found

Next, I took the seed list of the 2779 urls and imported it into gscraper, scraped the OBL for those urls, deduped/trimmed and exported the results (resulting in 26689 unique domains)

I imported these domains into expired domain finder as "list of domains to check and get metrics"

Result: 424 expired domains found

In this test, I used 5 threads, no filters, no proxies on both lists. Then I tried again with dedicated proxies and got the same results

 

[Deleted member 986121]

I also get many domains which are alive in the expired domain list often for example .ch domains. Next time i save the domains.

today the expired domain finder found this domain "winserion.org" which is already registered by someone else.

 

[loopline]

Ok, so I have ran another test and here is what I found:

I imported a seed list of 2779 urls to be crawled

Result: 22 expired domains found

Next, I took the seed list of the 2779 urls and imported it into gscraper, scraped the OBL for those urls, deduped/trimmed and exported the results (resulting in 26689 unique domains)

I imported these domains into expired domain finder as "list of domains to check and get metrics"

Result: 424 expired domains found

In this test, I used 5 threads, no filters, no proxies on both lists. Then I tried again with dedicated proxies and got the same results

It sounds like your hitting a wall of quantity of domains scraped before it stops. So if a security software has a threshold and you reach X conections total and then it shuts it down, that could cause some of your behavior. Also read my reply below this to foppaGG.

The expired finder does not behave like this for me, its consistent, and it also works thru till the end. So make sure you set the expired finder as trusted/whitelisted in all secuirty software such as anti-virus, malware checkers and firewalls.

But ultimately you can send a small sample list that consistently produces adverse results to support and see if they can replicate the issue.

In my experience thus far with the expired finder Ive heard your story many times over and 9.8 times out of 10 its an issue on the user end with DNS, proxies, security software etc... and .2 times its been a bug with scrapebox.

Also 5 connections may be too many for some domains, they may be blocking your ip. I guess it also depends on how deep you are crawling with it.

today the expired domain finder found this domain "winserion.org" which is already registered by someone else.

It does not show that as expired for me.

The thing about this, and the user above is that if you are using proxies and the proxy fouls up the response, it could cause something to show as expired when its not. Also security software interfering could cause an issue.

Manually run that domain, does it show as expired?

Also if your ISP is hijacking the connections and redirecting to their own ad pages, it can skew results.

 

[UnDerDoG81]

It happens again. 7K list, crawl depth 1, checking 700-800 domains and then jumps instantly to 7000.

 

[loopline]

It happens again. 7K list, crawl depth 1, checking 700-800 domains and then jumps instantly to 7000.

Something is hijacknig the threads. So you need to find the offending program, whitelist the expired domian finder in that program, or uninstall that program. Note that disabling security software still allows it to fire existing rules, so you have to whitelist. It could also be your router, so you can try bypassing it as a test, or using a mobile dongle etc..

 

[UnDerDoG81]

Something is hijacknig the threads. So you need to find the offending program, whitelist the expired domian finder in that program, or uninstall that program. Note that disabling security software still allows it to fire existing rules, so you have to whitelist. It could also be your router, so you can try bypassing it as a test, or using a mobile dongle etc..

Thanks loopline. Weird, I don´t use any antivirus program or security software. Only Windows 10 integrated stuff. This issue started to happen for 1 week or so now. Or maybe I was never watching it so I never realized. I have no idea how to bypass the router since I don´t know that stuff. I guess that is not even possible here in Germany as much as I know. Mobile dongle you mean use WIFI instead of a cable connection to the modem?

 

[loopline]

Thanks loopline. Weird, I don´t use any antivirus program or security software. Only Windows 10 integrated stuff. This issue started to happen for 1 week or so now. Or maybe I was never watching it so I never realized. I have no idea how to bypass the router since I don´t know that stuff. I guess that is not even possible here in Germany as much as I know. Mobile dongle you mean use WIFI instead of a cable connection to the modem?

So windows defender/security essentials will do this, disable real time scanning and whitelist in the programs or just shut them down. For windows 10 for me I had to hack the registry to get windows defender to shut off. You could try installing something else as a test too, like comodo, but you will need to whitelist scrapebox in comodo as well.

I assume you use a reouter, it is likely what gives you wifi, but it may be integrated into your modem. If thats the case, then you can't bypass it, you would just need to try disabling the router firewall.

Mobile dongle = cell phone or similar, nothing to do with wifi. Or you can take your pc to a friends house/internet cafe etc... if its a portable machine, just some way of getting around your router, or disable the router firewall.

 

[blackbeans]

Given the problems you guys reported, is there an update forthcoming that fixes these issues?

 

[UnDerDoG81]

So windows defender/security essentials will do this, disable real time scanning and whitelist in the programs or just shut them down. For windows 10 for me I had to hack the registry to get windows defender to shut off. You could try installing something else as a test too, like comodo, but you will need to whitelist scrapebox in comodo as well.

I assume you use a reouter, it is likely what gives you wifi, but it may be integrated into your modem. If thats the case, then you can't bypass it, you would just need to try disabling the router firewall.

Mobile dongle = cell phone or similar, nothing to do with wifi. Or you can take your pc to a friends house/internet cafe etc... if its a portable machine, just some way of getting around your router, or disable the router firewall.

Disabled Defender and whitelisted scrapebox but no chance, it skips the whole list after checking a few sites. It skips it even when I reduce threads to 1. Strange thing is, it does this only on crawl depth 1. I just tried it with crawl depth 5 and in the 5th depth it checks 12k pages and does not skip anything.

Heres the list, maybe somebody want to try it on crawl depth 1 and see if he gets the same error -> http://filehorst.de/d/bcdHybEm

EDIT: Is it possible to implement "copy to clipboard" feature when I do left click on the expired domain in the list. I have to manually type in the domain name in ahrefs etc. or have to click "open in default browser" and then copy the url to clipboard. Would be a nice shortcut