humanAI
Newbie
- Apr 23, 2017
- 18
- 3
Hi,
I have been working on this idea during the last couple of weeks. A little recap:
-Learned some Python
-Learned to use Scrapy library to write web crawlers
-Developed a product crawler for a selected niche and affiliate
-Set up a simple database to store the data
-Made some tinkering with automated link posting
The main idea is to scrape a section of few websites and collect their products (the price), then select the url, add my referral use Google API to obtain a shortlink, automatically post to various niche channels.
I would be able to select true deals, i.e. products that have their price lower than usual (in % or absolute number) above a certain threshold.
That's a lot of work but I think that can be useful and totally automated. Of course websites change and updates will be needed.
Problems I have encountered are:
-Avoid getting banned from websites. I am using my ip but it isn't the best choice I know. Services for random IP requests are around 10$/mo
-Storing data. This is on my pc too for now. automating the scraping job would require an online database so my pc can't provide that.
I just created my channels to push deals but I didn't advertise them for now since I didn't start delivering content.
What do you think? Any suggestion?
I have been working on this idea during the last couple of weeks. A little recap:
-Learned some Python
-Learned to use Scrapy library to write web crawlers
-Developed a product crawler for a selected niche and affiliate
-Set up a simple database to store the data
-Made some tinkering with automated link posting
The main idea is to scrape a section of few websites and collect their products (the price), then select the url, add my referral use Google API to obtain a shortlink, automatically post to various niche channels.
I would be able to select true deals, i.e. products that have their price lower than usual (in % or absolute number) above a certain threshold.
That's a lot of work but I think that can be useful and totally automated. Of course websites change and updates will be needed.
Problems I have encountered are:
-Avoid getting banned from websites. I am using my ip but it isn't the best choice I know. Services for random IP requests are around 10$/mo
-Storing data. This is on my pc too for now. automating the scraping job would require an online database so my pc can't provide that.
I just created my channels to push deals but I didn't advertise them for now since I didn't start delivering content.
What do you think? Any suggestion?