NO-CLUE-SEO
Senior Member
- Jul 31, 2016
- 1,003
- 309
Hi as the title says, how do expired domain scrapers work? Anyone?
Expired domain scrapers how do they work?
- Thread starter NO-CLUE-SEO
- Start date
darkpsycho
Registered Member
- Apr 9, 2017
- 60
- 55
You give them some keywords.
They take results from search engines.
They check these results for broken outgoing links.
Then they check if the domains these links point to are free to register.
That is more or less
They take results from search engines.
They check these results for broken outgoing links.
Then they check if the domains these links point to are free to register.
That is more or less
- Sep 17, 2008
- 6,037
- 3,974
This is true for some yes. There are a few methods that will be applied.
- Searching - take keyword(s) search in search engines for results then crawl these results looking for domains, once domains are found they will check availability
- Crawling - enter a website, tool will crawl it looking for domains, it will then check these domains for availability
- Input list of pages to crawl - tool will take list of pages and crawl them to find domains and then check domains for availability
- Input list of domains - simply check domains for availability
- Reverse crawl - enter a website, tool then gets list of webpages that link to that website and crawls these pages looking for domains that are available (requires 3rd party API for most tools)
- Sep 17, 2008
- 6,037
- 3,974
I forgot to mention that I love your Dwayne Dibley AV lol
NO-CLUE-SEO
Senior Member
- Jul 31, 2016
- 1,003
- 309
A fellow red dwarf fan lol.
Thanks for the answers. I was trying to understand the best method for searching such websites and the broken links method seems viable as it wouldn't involve any 3rd party tools if coded properly. I code quite a few bots.
Mark.
Thanks for the answers. I was trying to understand the best method for searching such websites and the broken links method seems viable as it wouldn't involve any 3rd party tools if coded properly. I code quite a few bots.
Mark.
- Aug 4, 2015
- 4,559
- 679
Hunt From Keywords
Hunt From Websites
Reverse Hunt Domains
Auction Hunter
Web 2.0 Hunter
Filtering/Sorting
Filtering/Sorting
using this stargey will help to find expired blogs
Hunt From Websites
Reverse Hunt Domains
Auction Hunter
Web 2.0 Hunter
Filtering/Sorting
Filtering/Sorting
using this stargey will help to find expired blogs
don chino
Senior Member
- Apr 2, 2011
- 890
- 291
As Furious Man said above, but you will need to do a lot of filtering later to pick up the good domains. Depending on your purpose for expired domains, for backlinking, usually only a very small percentage of scraped domains are good enough to use (backlink profile, web history, metrics, index...).
- Sep 17, 2008
- 6,037
- 3,974
You literally have just taken the list of tools directly out of Domain Hunter Gatherer Pro here lol. I'm not mad at all about it, quite proud actually, but it is what it is lol.
I wasn't going to overtly mention my own tool but if you want to see a complete list of all functions you will need for a complete domain scraping tool you could just look to Domain Hunter Gatherer Pro, it has all the tools and functionality that are required to hunt for, evaluate and resurrect expired domains
digimarket
Regular Member
- Sep 20, 2016
- 351
- 59
Expired Domain Scrapers scrolling in search engines. Find expired domains that are still indexed in the search engines and rebuild them before their de-indexed. Use the expired domains for building a PBN.
Servin Laypa
Registered Member
- Aug 21, 2017
- 79
- 3
So you're going to code your own bot for this using the broken link method I take it. I suppose it's for personal use since a bot BST would be very time consuming to manage; but what do I know?
NO-CLUE-SEO
Senior Member
- Jul 31, 2016
- 1,003
- 309
Thanks for some fantastic answers . As I learn seo it's nice to understand the methods used etc. I like the idea of broken links still indexed in google and I imagine the filtering would be huge to do.
SuperSonicProxy
Power Member
- Sep 3, 2017
- 792
- 91
Well there are other paid tools available as well, such as domcop which would provide you all the information on a particular domain and you can simply browse and review the information easily through their site. This allows you to easily and quickly establish whether that expired domain is worth your time or not. Personally, I'd scrape my own and use my own tools to check the metrics, just to save a couple of bucks.