Scrapebox Expired Domain Finder

[loopline]

HI. Thanks for replying. Yes, I understand the concept of deep-crawling.

But the problem is that when I load lets say 50 URL as seed urls and set 2 levels deep, I would expect that 50 threads are created and deep crawling the 50 seed links separately.

But like I said, if I set level2 crawling on 50 URL it only uses 1 thread and crawls the URLs in cascade mode. It does not use several threads to crawl multiple seed urls at the same time. But If I use 1 level deep crawling the 50 URLs are crawled quickly by 50 threads at one time like they should.

So what seems to be the issue here?

Thank you for your response.

Thats just not how it works. It starts with 1 seed url and works thru it and then moves to the next. There are many reasons why this is best, which I won't go into.

There is no issue. When your only looking at initial urls it can work with 50 threads, but as soon as you start crawling that uses a different segment of code and thus each url is worked 1 at a time in succession.

So in summary the expired finder is working as its supposed to. You can't change it, thats just how it is.

What you can do is run multiple instances of Scrapebox at once and thusly run multiple instance of the expired domain finder at once. There is no limit, you can run as many on the same machine as you want.

 

[Sweetfunny]

HI. Thanks for replying. Yes, I understand the concept of deep-crawling.

But the problem is that when I load lets say 50 URL as seed urls and set 2 levels deep, I would expect that 50 threads are created and deep crawling the 50 seed links separately.

But like I said, if I set level2 crawling on 50 URL it only uses 1 thread and crawls the URLs in cascade mode. It does not use several threads to crawl multiple seed urls at the same time. But If I use 1 level deep crawling the 50 URLs are crawled quickly by 50 threads at one time like they should.

So what seems to be the issue here?

Thank you for your response.

It does use 50 threads. If you load 50 urls to crawl XX levels deep, then it will load the first URL and use 50 threads and process it until the first URL has been completed. Then it will move on to the second URL, process the levels using 50 threads and so on. If the site only has 2 pages then it's not going to get up to 50 threads on that particular site.

If you process all 50 urls at once and use 50 threads, you will have 50x50 = 2,500 threads running which is not what people want when setting the connections to 50.

 

[mtj85]

Ok, thanks for explaining.

But this is what puzzles me.
Last time I entered 716 urls and selected 1 level deep crawling (meaning just the initial urls get crawled) and I got a nice output of 0/716 in the status bar (along with 50 thread and 50 grabber indicated), and the tool would crawl thru them nicely. It also showed me 4 dropped domains. See image below - the upper section of it.

But then I enter 3 levels deep (which I suppose would uncover even more dropped domains) and I never get an indication in the status bar (it always shows 0/1 or 0/0 in progress) and there is always just 1 thread and 1 grabber. See image above - bottom part of it. It just looks like there's no links on subsequential pages of the seed url (or maybe I am understanding this the wrong way).

So how come level1 crawling finds 4 dropped domains, but level3 crawling finds none? This is the reason why I am confused.

Sorry again to hijack this thread, but I figured this would be the best way to get help fast and this can help others understand the tool as well.

Thank you.

 

[Sweetfunny]

But then I enter 3 levels deep (which I suppose would uncover even more dropped domains) and I never get an indication in the status bar (it always shows 0/1 or 0/0 in progress) and there is always just 1 thread and 1 grabber. See image above - bottom part of it. It just looks like there's no links on subsequential pages of the seed url (or maybe I am understanding this the wrong way).
.

It doesnt do this for me, i just loaded a random list of domains and set it to crawl 3 levels deep with 50 connections and it used them it doesn't just sit on one connection. See below:

So perhaps there's something not right with your list or config, you may need to contact support with further info and a sample list to figure out what's wrong.

 

[mtj85]

HI. I did contact support and I'm yet to receive a reply.
@loopline
Could you please expedite this for me as at this time, only level1 crawling is working for me.
Paypal id: xxxxxxxxxxx51131B
Hope I get a reply soon.

Thanks for assisting me.

 

[Ambitious12]

Have a look here - http://www.scrapebox.com/expired-domain-finder

Umm yes it is good.I never used this but now I am thinking to move with this.

 

[efastvps]

In the list generated from Domain finder am i the only one thats seeing many already registerd domains? Becuase each time i add the domains to majestic to see the metrics and i look at the ones with high metrics they are already registered.

 

[loopline]

In the list generated from Domain finder am i the only one thats seeing many already registerd domains? Becuase each time i add the domains to majestic to see the metrics and i look at the ones with high metrics they are already registered.

Its working for me. Can you give some examples ? I mean if you can give some examples of domains and the original page where the domain was found, that way someone can try and reproduce it.

 

[efastvps]

I have it scanning at the moment, when i have the list ill get back to you.

 

[efastvps]

Its working for me. Can you give some examples ? I mean if you can give some examples of domains and the original page where the domain was found, that way someone can try and reproduce it.

I found the issue, it was with the proxies, dont ask me how this happens but when i changed them it suddenly started working.

 

[loopline]

I found the issue, it was with the proxies, dont ask me how this happens but when i changed them it suddenly started working.

Sometimes things can't be explained, but as long as its working thats all that matters.

 

[UnDerDoG81]

Lets say I want to search wikipedia for expired domains. Do I just put in www.wikipedia.org (wich depth?)? Or do I do a scrape like site:www.wikipedia.org and then scan the results? I know Wiki is a bad example but you get what I mean.

And is there a need to use proxies when it scans wikipedia.org for expireds?

EDIT: One more question, I am a noob in expired domains, so, this is a serious question. How does the Scrapebox expired domain finder have an advantage over sites like expireddomains.net? I mean, they list all the expired domains out there, don´t they?

I am not so sure if it is really possible to find good (no spam) expired domains in the web? The big players crawl and reserve them even before we even know they will expire. So how on earth would it be possible to find good expired domains?

 

[loopline]

Lets say I want to search wikipedia for expired domains. Do I just put in www.wikipedia.org (wich depth?)? Or do I do a scrape like site:www.wikipedia.org and then scan the results? I know Wiki is a bad example but you get what I mean.

And is there a need to use proxies when it scans wikipedia.org for expireds?

EDIT: One more question, I am a noob in expired domains, so, this is a serious question. How does the Scrapebox expired domain finder have an advantage over sites like expireddomains.net? I mean, they list all the expired domains out there, don´t they?

I am not so sure if it is really possible to find good (no spam) expired domains in the web? The big players crawl and reserve them even before we even know they will expire. So how on earth would it be possible to find good expired domains?

The expired domain finder will go thru and scan them all to see if they are available. As for depth that depends. I mean if its a small site maybe 4-6 levels will pretty much have the whole site, but with a site like wikipedia you could max out the depth, but bear in mind as you get 6 -8 levels deep you are likley to start getting into millions of urls. So thats gonna take a minute. lol

Wikipedia is good about not blocking so if you keep connections low your not likley to need proxies, but it really depends on the end domain your scraping and how many connections your using.

What is a "good" expired domain? You need more distinctions on what your after. So you first need to figure out what you consider a good expired domain.

But yes you can find them, or what most people consider good anyway. If your "Good" unreasonable, then thats different. So what is good?

As for the site sure it lists some, but its not possible for it to crawl the entire internet like google, they only have to crawl enough to make a profit.

Really it comes back to what you call good, I mean for you good might be the perfect niche relevant site with links from sites that will drive real traffic but have no high ranking value, and no one else is in your niche thinking this way so they don't want it cause its low authority. But say you get the site and funnle hundreds of traffic to your site a day, its not good its great!

So you must define good before you can really proceed.

 

[UnDerDoG81]

Thanks Matt for explaining. What I am not sure about now is, when you set "deep" to lets say 5 on wikipedia (or any other domain), will SB EDF only follow internal links and search the internal for expireds? Or will it follow the external links and search the external links for expireds too? Because with a deepnes of 20 or so I bet you will get a lot of spam sites if it follows external sites?

As for "good" domain, I did some searches and only found spammy stuff. I tried expired domains 3-4 years ago but never with much success. A good domain for me is a domain with no spammy links (or at least any links) and some authority. I don´t have an Majestic account so probably I am not really able to find the right metrics. I use Moz only actually. But I think I don´t really know how to search. Maybe you have some tipps? I watched your video but I still don´t know for what I should search? Wich footprints? I am back to online marketing after a 3-4 years of break and am starting with scrapebox again

 

[ciociombo]

I have a problem with Majestic. I get some 0s, email and password are correct. Any idea whats wrong?

 

[loopline]

Thanks Matt for explaining. What I am not sure about now is, when you set "deep" to lets say 5 on wikipedia (or any other domain), will SB EDF only follow internal links and search the internal for expireds? Or will it follow the external links and search the external links for expireds too? Because with a deepnes of 20 or so I bet you will get a lot of spam sites if it follows external sites?

As for "good" domain, I did some searches and only found spammy stuff. I tried expired domains 3-4 years ago but never with much success. A good domain for me is a domain with no spammy links (or at least any links) and some authority. I don´t have an Majestic account so probably I am not really able to find the right metrics. I use Moz only actually. But I think I don´t really know how to search. Maybe you have some tipps? I watched your video but I still don´t know for what I should search? Wich footprints? I am back to online marketing after a 3-4 years of break and am starting with scrapebox again

Yes it will crawl internal expired only. It will not leave the wikipedia site or whatever site you set it on .

I don't really have any set tips on finding them. I mean if you have domains you want links from start there, else you can do a date range search in google or bing etc.. and search your desired keywords or big sites you want links from etc..

Then crawl, then use moz to filter down to the most ideal domains then you can use a free majestic account to get some data and limited checks.

I have a problem with Majestic. I get some 0s, email and password are correct. Any idea whats wrong?

So some work but some are 0s?

Did you click the majestic metrics button?

Did you check those same domains in majestic, they may actually be 0.

 

[ciociombo]

So some work but some are 0s?

Did you click the majestic metrics button?

Did you check those same domains in majestic, they may actually be 0.

Sorry for my bad english - I had only 0s, nothing else. I installed scrapebox again and its working after clicking majestic metrics It doesnt check majestic automically like moz metrics, only after I click the button majestic metrics? I ask because majestic after 100-200 checs makes me wait (too many checks in short time)

BTW Scrapebox is the best SEO tool, great job

 

[loopline]

Sorry for my bad english - I had only 0s, nothing else. I installed scrapebox again and its working after clicking majestic metrics It doesnt check majestic automically like moz metrics, only after I click the button majestic metrics? I ask because majestic after 100-200 checs makes me wait (too many checks in short time)

BTW Scrapebox is the best SEO tool, great job

Yes you have to click the majestic button, its not automatic. Because majestic checks can get expensive, so people that are working with tens of thousands of expired domains - it lets them filter based on other criteria and then majestic check only the ones they are really interested in.

 

[bourrinfurtif]

Yes you have to click the majestic button, its not automatic. Because majestic checks can get expensive, so people that are working with tens of thousands of expired domains - it lets them filter based on other criteria and then majestic check only the ones they are really interested in.

Hi Loopline,
I've had 2 runs of 24h each with Scrapebox expired domains finder since I bought it, and so far it didn't find any domain....
First batch I inserted 15 big old sites, second batch I inserted top 100 worldwide websites in science.

So I think something is wrong with the tool or my configuration. Any idea from where the problem could come from?

Thanks in advance

 

[wishedwashed]

I think there are many people trying to find expired domains from popular sources and your chances of finding decent ones may be reduced if you put a popular domain as your seed one.