Hello all
I restart my new journey in this thread as I posted it initially in "Making Money" by mistake.
Goals
My objective is to launch an open-source bot framework, to allow peope to develop more easily customized bots and progressively to build a repository.
Planned features:
Languages/Technologies
I chose Python/Django as the main framework, and jQuery for client's side stuff. Initially, I planned to use Angular or VueJS, but as the learning curve can be difficult, I prefered to stay on a well-known technology, hoping to get the maximum of contributors.
In addition, it was important to me not to do another Windows tool, but rather a tool that can be hosted on a Linux server.
For the database, notably to store results of bots, I chose SQLite to avoid any extra-configuration of the DB.
Current Status
- Authentication: done
- Global configuration: done
- Bot detection / configuration (dynamic forms): done
- Proxies and proxy lists management: done
- Mass-import of proxies: To be done
- Live status about crawling: done
- Custom database table(s) and columns for each bot to store results, easily configurable: done
- Code a first multi-threaded dummy bot: In progress
- Log management (client/server): In progress
Current Screenshots
Dashboard:
Configuration of a bot (customizable fields depending on each bot!):
Proxy Management (supporting group of proxies)
YOUR FEEDBACK IS VERY IMPORTANT! Feel free to comment, to provide ideas, to share what you would want to be implemented (any ideas fr the first three bots to code? It can be everything: a poster, a crawler, etc)
In addition, any encouragement is welcomed. This projects take a lot of time, and a sweet word is always nice to have.
Thanks to all
I restart my new journey in this thread as I posted it initially in "Making Money" by mistake.
Goals
My objective is to launch an open-source bot framework, to allow peope to develop more easily customized bots and progressively to build a repository.
Planned features:
- Proxies (of course)
- Headless browsers to be able to scrap Javascript
- Browser parameters management (user-agent, tc.)
- Multi-threads
- Web app to get the status, and to read results
- And most of all: Features to code new bots
- CSV export of results
Languages/Technologies
I chose Python/Django as the main framework, and jQuery for client's side stuff. Initially, I planned to use Angular or VueJS, but as the learning curve can be difficult, I prefered to stay on a well-known technology, hoping to get the maximum of contributors.
In addition, it was important to me not to do another Windows tool, but rather a tool that can be hosted on a Linux server.
For the database, notably to store results of bots, I chose SQLite to avoid any extra-configuration of the DB.
Current Status
- Authentication: done
- Global configuration: done
- Bot detection / configuration (dynamic forms): done
- Proxies and proxy lists management: done
- Mass-import of proxies: To be done
- Live status about crawling: done
- Custom database table(s) and columns for each bot to store results, easily configurable: done
- Code a first multi-threaded dummy bot: In progress
- Log management (client/server): In progress
Current Screenshots
Dashboard:
Configuration of a bot (customizable fields depending on each bot!):
Proxy Management (supporting group of proxies)
YOUR FEEDBACK IS VERY IMPORTANT! Feel free to comment, to provide ideas, to share what you would want to be implemented (any ideas fr the first three bots to code? It can be everything: a poster, a crawler, etc)
In addition, any encouragement is welcomed. This projects take a lot of time, and a sweet word is always nice to have.
Thanks to all