[Need your thoughts] Create an URL scraper

So I had an idea to create and URL scraper, just something basic.

I saw a thread yesterday where someone asked if anyone could make a program that would scrape information for him based on an URL. So this got me thinking, what if I would release a program that would scrape, lets say 10 variable sections, from an url. This would be done by classname or ID and so it will not be completely noob friendly (as you need to know how to enter the developer inspector mode from your browser).

So what it will do:

• You insert a list of urls (can be imported by .txt and .csv
• You select the divs you want to scrape
• You will add a list of proxies (Yes I will include proxy support)
• The program will spit out the URL with all the scraped content representing each div you have selected

Would this be to anyone else be using something like this, or just some weird idea I have? Please let me know so I can get started without having the idea that my effort will be in vain.

P.S. I am not sure where to put something like this, so if a mod doesn't like the place, feel free to move it.

I think you'll want to create a GUI for selecting the divs. Otherwise your target market is very thin between people who know css selectors and people who can't write this themselves + they must need proxies or they can just write a simple jquery query.

Also, not to discourage you but there are already services offering this:
https://data-miner.io/
http://webscraper.io/
https://www.parsehub.com/
So the market is there but a div selection GUI is a must for a thing like this. Good luck mate.

Thanks for your feedback. I forgot to mention that I am planning to give a tool for free and allow people to donate if they find it a useful tool. Furthermore yea I think a GUI would indeed be one of the best options. Maybe I need to investigate this and find a way to make it easier for people to use it.

You should go ahead and do it, and it will be a cool lesson for you at the least

Rather than creating the nth windows interface (GUI), we should rather make our bots in command line, using scrapy for instance.
I am amazed to see that the SEO world still needs "click tools" to work. This shows the low evel of the community for me.

Using scrapy, what you want to do can be achieved in 10 minutes.

Some basic functionality of URL scrapper I already worked with scrapebox, check it out before building new bot.

Ah these all look like good free versions already.

ju111 said:

Rather than creating the nth windows interface (GUI), we should rather make our bots in command line, using scrapy for instance.
I am amazed to see that the SEO world still needs "click tools" to work. This shows the low evel of the community for me.

Using scrapy, what you want to do can be achieved in 10 minutes.

Click to expand...

Don't break an arm jerking yourself off.

uchiha.jain said:

Don't break an arm jerking yourself off.

Click to expand...

LOL, nice one. Ok, i didn't want to break any motivation, and by the way it depends of the language OP wants to learn.
Just read my comment like my own tiredness to see so many Windows tools, while ideally, all tools should run on a linux server (less expansive, more practical to be run 24/7, more scriptable).