1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Free Expired Domain Crawling with 80Legs

Discussion in 'Black Hat SEO' started by regicider, Nov 19, 2014.

  1. regicider

    regicider Junior Member

    Joined:
    Apr 2, 2012
    Messages:
    107
    Likes Received:
    54
    I feel like it?s high time that I start giving back to the community, so I thought I?d share a new method for expired domain crawling. We will utilize a cloud-based web crawling service, called 80Legs, and some software that I?ve written, which you can download for free here. If you?re more of a visual learner, there?s also a video tutorial on my site.

    Ok, let?s get started!

    1. Download the software package above.
    2. Go to 80Legs and register for a free account. You can run unlimited crawls of up to 10,000 pages with a free account.
    3. Sign in to your 80Legs account and click on the tab ?My 80apps?. An 80app is a javascript file that tells the crawler which links to crawl next, and which data it should return. The 80app I?ve included only crawls to pages on the domain of the URLs you enter. For example, if you enter ?http://bbc.com/technology? the crawler will crawl to ?http://bbc.com/news? but not ?http://nytimes.com/technology?
    4. Click the button ?Create an 80app.?
    5. Click ?Choose a File? and browse to the file named ?GetUniqueDomains80App.js? We now have the specialized 80app that we can use for crawling.
    6. Click the tab ?Create a Crawl?
    7. Give your crawl a name.
    8. Enter your URL list. To start, we?ll choose the tab ?Type in a URL List?.
    9. Enter the URLs you want to crawl. If you?re using a free account, I only recommend entering 1 URL at a time.
    10. Leave the crawl depth at 20. We want to make sure we crawl all 10,000 pages available to us.
    11. Select the 80app ?GetUniqueDomains80App? from the dropdown.
    12. Click Create crawl.

    Your crawl will now start running. Crawling 10,000 pages usually takes anywhere from 20 minutes to an hour, depending on the site you?re crawling. You can check the status of your crawl in the ?My Crawls? tab. Once your crawl has finished, click on it and download the JSON results file(s). Now, we can use the 80LegsDomainProcessor included in the download. Open the folder 80LegsDomainProcessor and run ?DomainScraper.exe?. Click the ?browse? button to locate the results file you just downloaded, then click ?Rip Domains.? Your results will now start getting processed

    Note: The metrics checking feature for the 80Legs Domain Processor is stripped out, so all metrics are shown as zero. Make sure to check the metrics on your own.

    Enjoy!
     
    • Thanks Thanks x 2