DogNamedRottles
Registered Member
- Dec 19, 2021
- 52
- 58
It all started from a conversation with a friend about how he managed to make money by manually putting together a speadsheet with business leads by using google search. He managed to sell his list for a $1000. It made me think about how the manual process could be automated and instead spending an entire day putting together a small list (<1000) of leads, there could be a service that could automate the process and shorten the time to build the list dramatically.
When I mentioned how he could automate extraction of the details using special patterns, he was astonished. He said, "Dude, you could make money off of this. Why not create a service and charge a subscription? I know business owners who would pay for such a service"
Thus, my journey began writing such a service. First I started building a smaller scale scraper that grabs sub-directories of businesses just to get my feet wet with web scraping. However, I realized that in order for this to be effective, I need to build a scalable web crawler that can essentially do what Google's crawlers do when indexing the internet. Whilst, extracting the vital business contact data to make available as a lead to a business owner.
This isn't a new idea and it has been around for some time, therefore, I imagine this market is quite saturated. Take for instances, seamless.AI which does the same thing but has added extra features that also provides decision maker contact details. On the other hand, this product idea could be a cheaper alternative and thus more viable. Ultimately, if it doesn't pan out it's a nice technical challenge that I could put down on my resume for future opportunities.
From a high level standpoint, the product design would be split into two sections: the data acquisition, and the end-user interface. The data acquisition being the home brewed crawler which searches the internet find businesses in a specific industry either defined by the end-user or part of a list of industries. The second part is essentially the end product where the business has a dashboard which similarly matches what you see in today's CRMs.
I want to record my journey building the service through here as a well holding myself accountable to see the product through. I'm not so concern that it makes money but the fact the product exists and it was built by me.
When I mentioned how he could automate extraction of the details using special patterns, he was astonished. He said, "Dude, you could make money off of this. Why not create a service and charge a subscription? I know business owners who would pay for such a service"
Thus, my journey began writing such a service. First I started building a smaller scale scraper that grabs sub-directories of businesses just to get my feet wet with web scraping. However, I realized that in order for this to be effective, I need to build a scalable web crawler that can essentially do what Google's crawlers do when indexing the internet. Whilst, extracting the vital business contact data to make available as a lead to a business owner.
This isn't a new idea and it has been around for some time, therefore, I imagine this market is quite saturated. Take for instances, seamless.AI which does the same thing but has added extra features that also provides decision maker contact details. On the other hand, this product idea could be a cheaper alternative and thus more viable. Ultimately, if it doesn't pan out it's a nice technical challenge that I could put down on my resume for future opportunities.
From a high level standpoint, the product design would be split into two sections: the data acquisition, and the end-user interface. The data acquisition being the home brewed crawler which searches the internet find businesses in a specific industry either defined by the end-user or part of a list of industries. The second part is essentially the end product where the business has a dashboard which similarly matches what you see in today's CRMs.
I want to record my journey building the service through here as a well holding myself accountable to see the product through. I'm not so concern that it makes money but the fact the product exists and it was built by me.
