1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

why doesn't scrapebox make a domain miner?

Discussion in 'Black Hat SEO' started by nonai, Feb 10, 2015.

  1. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    scrapebox is so close to being a domain miner.

    can it scrape? yes
    can it extract external domains? yes
    can it check PA DA? I am not sure, but that's not hard to implement.
    can it check domain availability? at this point no, but that's not hard to implement

    I know of the 2 programs made for domain mining but I dont want to pay a monthly fee. I would pay like $20 for a SB add on.

    why dont they make it? this should be so easy for someone with programming knowledge.
     
  2. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,726
    Likes Received:
    1,993
    Gender:
    Male
    Home Page:
    I actually have people routinely ask me how to do this, and there are even videos on it.

    It does scrape
    It does extract external domains
    It does check PA/DA - Page Authority Addon - added maybe a year ago, or actually I think late 2013

    can it domain availability - yes - they added that in early 2014 I believe.

    A lot of people actually use SB to get expired domains
     
    • Thanks Thanks x 1
  3. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    Actually, you are right, it does have domain availability checker. But last time I used it, it sucked. a lot of false positives and false negatives.
     
  4. regicider

    regicider Junior Member

    Joined:
    Apr 2, 2012
    Messages:
    108
    Likes Received:
    54
    As loop line said, it can do all of the things you mentioned. However, I think what you're missing is that scrapebox is a web scraper - the Expired Doman Miner and DomRipper are web crawlers. While scrapers are very simple, web crawlers are significantly more difficult to implement, especially with the memory constraints of your typical VPS.

    If you you ask anyone building PBNs at any scale, they'll tell you that crawling is absolutely necessary to stay competitive.
     
  5. myownhero

    myownhero Power Member Premium Member

    Joined:
    Mar 13, 2012
    Messages:
    763
    Likes Received:
    714
    Occupation:
    SEO Analyst / Link Builder
    Location:
    United States
    Home Page:
    I think you have to manually remove the https:// sites from the URL structure before you check whether the domain is registered or not in Scrapebox and you'll also want to remove the weird TLDs that you can't register since they'll often pop up as available to register.
     
  6. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    What's the difference between scraping and crawling? I always thought they are the same thing.
     
  7. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    I don't think SB can check domain availability. I just checked all the addons and there is no such thing.
    for example if you were to check the following domains, how would you check them in SB?
    abc.com
    hty.org
    fhere.net
     
  8. myownhero

    myownhero Power Member Premium Member

    Joined:
    Mar 13, 2012
    Messages:
    763
    Likes Received:
    714
    Occupation:
    SEO Analyst / Link Builder
    Location:
    United States
    Home Page:
    It isn't an add-on. Import your URL list and click:

    Grab / Check > Check for unregistered domains.
     
    • Thanks Thanks x 1
  9. regicider

    regicider Junior Member

    Joined:
    Apr 2, 2012
    Messages:
    108
    Likes Received:
    54
    Scraping refers to processing a list of web pages to extract some data from In this case it would be individual domains for processing. Scrapers, like scrapebox, only process data from the Urls you input

    Crawling processes data from the web page, similar to a scraper, but then it crawls to the links on the page also, and then processes them.

    for example, if you wanted to process all the pages on bbc.com, a scraper would need you to have a list of all Urls on bbc.com. A crawled can simply take the input of the root domain, bbc.com and then will crawl all the pages for you.

    Does that make sense?
     
    • Thanks Thanks x 1
  10. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    yes it does, thank you so much.
     
  11. lilmasta

    lilmasta Elite Member

    Joined:
    May 21, 2009
    Messages:
    2,467
    Likes Received:
    1,137
    Gender:
    Male
    Location:
    192.168.0.1
    Its has the best broken link finder so far
     
  12. RuthSam

    RuthSam Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 19, 2010
    Messages:
    3,812
    Likes Received:
    973
    Gender:
    Male
    Home Page:
    It would be great if they somehoe built a Add On to get expired domains from Scrapebox.. I know there are Videos and great help to get from them, however, it's very complicated I find. DomRipper is OK, but so far I don't had luck to find a single domain with it.. on 900 different domains... maybe the quality is to low.. will keep trieing for a while.
     
  13. regicider

    regicider Junior Member

    Joined:
    Apr 2, 2012
    Messages:
    108
    Likes Received:
    54
    PM me or add me on skype (in my bio), I can give you some tips to help you out.
     
  14. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,726
    Likes Received:
    1,993
    Gender:
    Male
    Home Page:


    So first you say it has it and it sucks, and then you say it doesn't have it? I think you are not really doing much research before you make statements.

    Go to Grab/check >> check unregistered domains.
     
  15. myownhero

    myownhero Power Member Premium Member

    Joined:
    Mar 13, 2012
    Messages:
    763
    Likes Received:
    714
    Occupation:
    SEO Analyst / Link Builder
    Location:
    United States
    Home Page:
    It does have some flaws. Copy and import this list into Scrapebox and then use the domain checker to see if they're available:

    https://apple.com
    https://google.com
    https://yahoo.com

    So long as it's https it'll always show up as available when you do a domain check. This is what is most likely causing the problems.
     
  16. nonai

    nonai Power Member

    Joined:
    Oct 10, 2013
    Messages:
    524
    Likes Received:
    64
    actually, the problem is not with https.
    I just tried a list of domains and 5 says they are available. I tried all 5 and none of them are actually available.
     
  17. myownhero

    myownhero Power Member Premium Member

    Joined:
    Mar 13, 2012
    Messages:
    763
    Likes Received:
    714
    Occupation:
    SEO Analyst / Link Builder
    Location:
    United States
    Home Page:
    You need to start giving examples. So far, https domains are the only ones I know are bugged. If you post which ones gave you false positives they might be able to fix it.
     
    • Thanks Thanks x 1
  18. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    592
    Likes Received:
    222
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    The price of Domripper is the guarantee that it will fail in the long run. I can think of 3 software combo, one being scrapebox, that will smash any "pro" expired domain crawler. 350 $ a month, what a joke.

    Regarding scrapebox, I still think that is the best seo software out there. Well designed, affordable and very useful. If they will integrate a profi domain crawler or make it as a premium addon I will buy it.

     
  19. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,726
    Likes Received:
    1,993
    Gender:
    Male
    Home Page:


    In 1.x I believe it does not have the dlls needed and/or support https in the domain checker and actually in multiple functions. Your examples show as not available in 2.x, which supports https just fine in every function I have tried.


    [​IMG]


    I agree with myownhero. We aren't posting in this thread to say that your wrong or disagree with you, I don't care if you are right or wrong and I don't have time to go around trying to prove a point of someone being right or wrong etc... I have a business to run.

    We are posting here because we genuinely want to help. If you want help then you need to post a LOT more specific detail. If you just want a place to chat or ramble about Scrapeboxes failures without actually giving an example of something that could be fixed (Assuming something was actually wrong, because the dev will fix issues. I mail them bugs and they always fix them) then just say so and we will all leave and you can speak with your self.

    If you genuinely want help then you are going to have to be the first person to help your self by helping us. Please give explicit detail, screenshots would be good as well.

    Agreed. Thx.
     
    Last edited: Feb 15, 2015
  20. domainingin

    domainingin Regular Member

    Joined:
    Oct 8, 2014
    Messages:
    429
    Likes Received:
    184
    Occupation:
    Domainer
    Location:
    Seychelles
    Home Page:
    I am not so sure, that you are completely right with the definitions of scraping and crawling. I think ?scraping" and "crawling? means the same in regard to domains, it only depends on the depth you adjust?

    I mean, at our domain scraper I can tell the tool to scrape just the links of the seed URL (depth=1) or the links of the links of the seed URL (depth=2) or the links of the links of the links of the links of the seed URL (depth=3) and so on...

    Correct me if I am wrong?

    Cheers
    Dan