why doesn't scrapebox make a domain miner?

nonai · Feb 10, 2015

scrapebox is so close to being a domain miner.

can it scrape? yes
can it extract external domains? yes
can it check PA DA? I am not sure, but that's not hard to implement.
can it check domain availability? at this point no, but that's not hard to implement

I know of the 2 programs made for domain mining but I dont want to pay a monthly fee. I would pay like $20 for a SB add on.

why dont they make it? this should be so easy for someone with programming knowledge.

loopline · Feb 10, 2015

nonai said:
scrapebox is so close to being a domain miner.

can it scrape? yes
can it extract external domains? yes
can it check PA DA? I am not sure, but that's not hard to implement.
can it check domain availability? at this point no, but that's not hard to implement

I know of the 2 programs made for domain mining but I dont want to pay a monthly fee. I would pay like $20 for a SB add on.

why dont they make it? this should be so easy for someone with programming knowledge.

I actually have people routinely ask me how to do this, and there are even videos on it.

It does scrape
It does extract external domains
It does check PA/DA - Page Authority Addon - added maybe a year ago, or actually I think late 2013

can it domain availability - yes - they added that in early 2014 I believe.

A lot of people actually use SB to get expired domains

nonai · Feb 10, 2015

loopline said:
I actually have people routinely ask me how to do this, and there are even videos on it.

It does scrape
It does extract external domains
It does check PA/DA - Page Authority Addon - added maybe a year ago, or actually I think late 2013

can it domain availability - yes - they added that in early 2014 I believe.

A lot of people actually use SB to get expired domains

Actually, you are right, it does have domain availability checker. But last time I used it, it sucked. a lot of false positives and false negatives.

regicider · Feb 11, 2015

As loop line said, it can do all of the things you mentioned. However, I think what you're missing is that scrapebox is a web scraper - the Expired Doman Miner and DomRipper are web crawlers. While scrapers are very simple, web crawlers are significantly more difficult to implement, especially with the memory constraints of your typical VPS.

If you you ask anyone building PBNs at any scale, they'll tell you that crawling is absolutely necessary to stay competitive.

myownhero · Feb 11, 2015

I think you have to manually remove the https:// sites from the URL structure before you check whether the domain is registered or not in Scrapebox and you'll also want to remove the weird TLDs that you can't register since they'll often pop up as available to register.

nonai · Feb 11, 2015

regicider said:
As loop line said, it can do all of the things you mentioned. However, I think what you're missing is that scrapebox is a web scraper - the Expired Doman Miner and DomRipper are web crawlers. While scrapers are very simple, web crawlers are significantly more difficult to implement, especially with the memory constraints of your typical VPS.

If you you ask anyone building PBNs at any scale, they'll tell you that crawling is absolutely necessary to stay competitive.

What's the difference between scraping and crawling? I always thought they are the same thing.

nonai · Feb 11, 2015

myownhero said:
I think you have to manually remove the https:// sites from the URL structure before you check whether the domain is registered or not in Scrapebox and you'll also want to remove the weird TLDs that you can't register since they'll often pop up as available to register.

I don't think SB can check domain availability. I just checked all the addons and there is no such thing.
for example if you were to check the following domains, how would you check them in SB?
abc.com
hty.org
fhere.net

myownhero · Feb 11, 2015

nonai said:
I don't think SB can check domain availability. I just checked all the addons and there is no such thing.
for example if you were to check the following domains, how would you check them in SB?
abc.com
hty.org
fhere.net

It isn't an add-on. Import your URL list and click:

Grab / Check > Check for unregistered domains.

regicider · Feb 11, 2015

nonai said:
What's the difference between scraping and crawling? I always thought they are the same thing.

Scraping refers to processing a list of web pages to extract some data from In this case it would be individual domains for processing. Scrapers, like scrapebox, only process data from the Urls you input

Crawling processes data from the web page, similar to a scraper, but then it crawls to the links on the page also, and then processes them.

for example, if you wanted to process all the pages on bbc.com, a scraper would need you to have a list of all Urls on bbc.com. A crawled can simply take the input of the root domain, bbc.com and then will crawl all the pages for you.

Does that make sense?

nonai · Feb 11, 2015

regicider said:
Scraping refers to processing a list of web pages to extract some data from In this case it would be individual domains for processing. Scrapers, like scrapebox, only process data from the Urls you input

Crawling processes data from the web page, similar to a scraper, but then it crawls to the links on the page also, and then processes them.

for example, if you wanted to process all the pages on bbc.com, a scraper would need you to have a list of all Urls on bbc.com. A crawled can simply take the input of the root domain, bbc.com and then will crawl all the pages for you.

Does that make sense?

yes it does, thank you so much.

lilmasta · Feb 11, 2015

Its has the best broken link finder so far

RuthSam · Feb 11, 2015

It would be great if they somehoe built a Add On to get expired domains from Scrapebox.. I know there are Videos and great help to get from them, however, it's very complicated I find. DomRipper is OK, but so far I don't had luck to find a single domain with it.. on 900 different domains... maybe the quality is to low.. will keep trieing for a while.

regicider · Feb 11, 2015

RuthSam said:
It would be great if they somehoe built a Add On to get expired domains from Scrapebox.. I know there are Videos and great help to get from them, however, it's very complicated I find. DomRipper is OK, but so far I don't had luck to find a single domain with it.. on 900 different domains... maybe the quality is to low.. will keep trieing for a while.

PM me or add me on skype (in my bio), I can give you some tips to help you out.

loopline · Feb 11, 2015

nonai said:
Actually, you are right, it does have domain availability checker. But last time I used it, it sucked. a lot of false positives and false negatives.

nonai said:
I don't think SB can check domain availability. I just checked all the addons and there is no such thing.
for example if you were to check the following domains, how would you check them in SB?
abc.com
hty.org
fhere.net

So first you say it has it and it sucks, and then you say it doesn't have it? I think you are not really doing much research before you make statements.

Go to Grab/check >> check unregistered domains.

myownhero · Feb 15, 2015

loopline said:
So first you say it has it and it sucks, and then you say it doesn't have it? I think you are not really doing much research before you make statements.

Go to Grab/check >> check unregistered domains.

It does have some flaws. Copy and import this list into Scrapebox and then use the domain checker to see if they're available:

https://apple.com
https://google.com
https://yahoo.com

So long as it's https it'll always show up as available when you do a domain check. This is what is most likely causing the problems.

nonai · Feb 15, 2015

myownhero said:
It does have some flaws. Copy and import this list into Scrapebox and then use the domain checker to see if they're available:

https://apple.com
https://google.com
https://yahoo.com

So long as it's https it'll always show up as available when you do a domain check. This is what is most likely causing the problems.

actually, the problem is not with https.
I just tried a list of domains and 5 says they are available. I tried all 5 and none of them are actually available.

myownhero · Feb 15, 2015

nonai said:
actually, the problem is not with https.
I just tried a list of domains and 5 says they are available. I tried all 5 and none of them are actually available.

You need to start giving examples. So far, https domains are the only ones I know are bugged. If you post which ones gave you false positives they might be able to fix it.

laur.laurix · Feb 15, 2015

The price of Domripper is the guarantee that it will fail in the long run. I can think of 3 software combo, one being scrapebox, that will smash any "pro" expired domain crawler. 350 $ a month, what a joke.

Regarding scrapebox, I still think that is the best seo software out there. Well designed, affordable and very useful. If they will integrate a profi domain crawler or make it as a premium addon I will buy it.

regicider said:
PM me or add me on skype (in my bio), I can give you some tips to help you out.

loopline · Feb 15, 2015

myownhero said:
It does have some flaws. Copy and import this list into Scrapebox and then use the domain checker to see if they're available:

https://apple.com
https://google.com
https://yahoo.com

So long as it's https it'll always show up as available when you do a domain check. This is what is most likely causing the problems.

In 1.x I believe it does not have the dlls needed and/or support https in the domain checker and actually in multiple functions. Your examples show as not available in 2.x, which supports https just fine in every function I have tried.

nonai said:
actually, the problem is not with https.
I just tried a list of domains and 5 says they are available. I tried all 5 and none of them are actually available.

I agree with myownhero. We aren't posting in this thread to say that your wrong or disagree with you, I don't care if you are right or wrong and I don't have time to go around trying to prove a point of someone being right or wrong etc... I have a business to run.

We are posting here because we genuinely want to help. If you want help then you need to post a LOT more specific detail. If you just want a place to chat or ramble about Scrapeboxes failures without actually giving an example of something that could be fixed (Assuming something was actually wrong, because the dev will fix issues. I mail them bugs and they always fix them) then just say so and we will all leave and you can speak with your self.

If you genuinely want help then you are going to have to be the first person to help your self by helping us. Please give explicit detail, screenshots would be good as well.

myownhero said:
You need to start giving examples. So far, https domains are the only ones I know are bugged. If you post which ones gave you false positives they might be able to fix it.

Agreed. Thx.

domainingin · Feb 16, 2015

regicider said:
Scraping refers to processing a list of web pages to extract some data from In this case it would be individual domains for processing. Scrapers, like scrapebox, only process data from the Urls you input

Crawling processes data from the web page, similar to a scraper, but then it crawls to the links on the page also, and then processes them.

for example, if you wanted to process all the pages on bbc.com, a scraper would need you to have a list of all Urls on bbc.com. A crawled can simply take the input of the root domain, bbc.com and then will crawl all the pages for you.

Does that make sense?

I am not so sure, that you are completely right with the definitions of scraping and crawling. I think ?scraping" and "crawling? means the same in regard to domains, it only depends on the depth you adjust?

I mean, at our domain scraper I can tell the tool to scrape just the links of the seed URL (depth=1) or the links of the links of the seed URL (depth=2) or the links of the links of the links of the links of the seed URL (depth=3) and so on...

Correct me if I am wrong?

Cheers
Dan

why doesn't scrapebox make a domain miner?

BANNED

Elite Member

BANNED

Junior Member

Power Member

BANNED

BANNED

Power Member

Junior Member

BANNED

Elite Member

BANNED

Junior Member

Elite Member

Power Member

BANNED

Power Member

Elite Member

Elite Member

Regular Member

Main Menu

Marketplace

Making Money

BlackHatWorld

Other