My journey to build an open-source bots framework

Hi all

As it is a personal need to have a long-term bot framework, I decided to code it. But instead of keeping everything for myself, I have also decided to release it on Github.
Of course it is not coded yet, but it moves forward progressively.

Features:

• Proxies (of course)
• Headless browsers to be able to scrap Javascript
• Browser parameters management (user-agent, tc.)
  
• Multi-threads
• Web app to get the status, and to read results
  
• And most of all: Features to code new bots
  
• CSV export of results

I will release it as soon as I have built the framework with one bot.

What is the purpose of the bot? Why wrote even before its creation? The bot will run on the desktop? What programming language do you use?

Well, I started the thread because I am already well advanced.

Purpose:

Multiple. The idea is to be very flexible. It is a bot framework and not a bot. I have many scripts in all languages on my computer, and I am tired to try to find the relevant bot. My code in such script is so ugly that it is difficult to maintain, so I want to make a proper solution to capitalize on each bot I code in the future.

To give an example: 2 years ago I scrapped many details about 4 millions websites. Everything has been lost because I don't have one location/one language/one tool for my bots and related data. I cannot count the number of data I scrapped in the past, justifying to make a long-term solution.

Another criteria was that I dislike very much Windows, and I am pretty sure that "bots" should be hosted remotely (in the design), and not on Windows platform (!!). This is why I selected a web application as the architecture. Currently, it is very easy to deploy.

Technologies:

I hesitated a long time between PHP and Python. I know very well PHP and the related frameworks (Laravel especially). However, I choose Python mainly because this is now a well-known language, and additionally, it has probably a better multithreading system and a better interface with PhantomJS.

So it will be:

• Python / Django
• Headless browser (probably phantomjs but maybe another)
• SQLite database (I am so tired to manage MySQL instances... I prefer flat files. Even for my new websites I am chosing flat-files CMS....)

Django is interesting because it enables to define new tables for modular bots very easily through automatic "migrations". PHP/Laravel has such feature too, but a bit less transparent/easy to use. Furthermore, with Django, the interface is immediately partially done. The only issue is the lack of flexibility to customize the main admin page (or I am a dummy, but well... I found this is just a pity and not well documented).

I should be able to post the first screenshots of the GUI in a few days, maybe before.

Great. I want to see what you get. Because write does not mean to do. The main thing that brought the project to the end. Bot will be for some social network?

I just have seen that my thread is in a bad section. Mod, would you mind to move it to IM journeys? Thanks a lot.

A short update...

I am moving forward with a great motivation. I have the feeling that the product will be a never-seen product at the end, because I take the time to think to every details.
Everything is made to let people code bots easily, and to re-use data that has been already crawled/parsed/databased.

Current status

• Web app easily installable
• Authentication
• User management
• Define data format and configuration of bots
• Global configuration (to provide default values to bot)
• Log management (on the web interface)

Although no bot has been coded yet (this is will be the simplest part once the framework will be ready), it is already a promising tool.

A first Screenshot !!



Next steps

Each bot can provide a undefined number of input parameters, which can be entered by the user through the web interface. One of the major next step would be to generate a form (when you click on "Launch") to enter all these parameters depending of the configuration. Planned supported types:

• Numbers
• Text
  
• Columns of a database table
• Conditions support (checkboxes/radios).

Once this major next step will be done, I will focus on the bots facilities in itself.