Best way to store and query some domain data

[davids355]

I am trying to store and query some domain data - just domain, date and scan results (I’m planning to do some periodic vulnerability scanning on a few sites).

What’s the best way of doing it? At the moment I’ve been experimenting with bash scripting and Mariadb but I don’t know if that’s over kill.

I really just want to loop through a list of domains and then keep a record of when the last scan was ran for each domain in the list.

My programming knowledge is quite basic - I’ve only done a little bit of bash scripting really.

This is partly a functional requirement of mine and partly an excuse to learn a bit more about programming.

 

[Juhku]

I’ve been experimenting with bash scripting and Mariadb but I don’t know if that’s over kill.

Unless you specifically want to use a dedicated database (like MariaDB) for learning purposes, yes, that’s an overkill.

My programming knowledge is quite basic - I’ve only done a little bit of bash scripting really.

This is partly a functional requirement of mine and partly an excuse to learn a bit more about programming.

I’d suggest creating a small program in Python.
You can just store the necessary data in a simple .txt or .xlsx file and query the data with some library or even create your own logic if you want to learn even more.

Super basic, gets the job done, and learning to create such a program without any previous Python experience would probably take you ~6 hours.

 

[davids355]

Unless you specifically want to use a dedicated database (like MariaDB) for learning purposes, yes, that’s an overkill.



I’d suggest creating a small program in Python.
You can just store the necessary data in a simple .txt or .xlsx file and query the data with some library or even create your own logic if you want to learn even more.

Super basic, gets the job done, and learning to create such a program without any previous Python experience would probably take you ~6 hours.

Thanks, hadn’t thought of using a csv file. Sounds like a much better idea.

 

[Gogol]

I am trying to store and query some domain data - just domain, date and scan results (I’m planning to do some periodic vulnerability scanning on a few sites).

What’s the best way of doing it? At the moment I’ve been experimenting with bash scripting and Mariadb but I don’t know if that’s over kill.

I really just want to loop through a list of domains and then keep a record of when the last scan was ran for each domain in the list.

My programming knowledge is quite basic - I’ve only done a little bit of bash scripting really.

This is partly a functional requirement of mine and partly an excuse to learn a bit more about programming.

I would personally use node/php and Mysql( aka Mariadb). But that said, bash should work perfectly fine as well (in fact it should be faster to deploy and run. The dx might not be as good though.). You are on the right path.

Also, I would advice against using csv if this data is going to become big. It gets really tough to search/update a csv file when the data gets bigger. It does not scale up as well as a rdbms. Sqlite will be a better option if you want to keep it simple.

 

[SeasonedCode]

As @Juhku suggested, using a CSV file is a great idea. However,
I'd recommend steering clear of Python's built-in CSV module since it can be a bit of a hassle.
Instead, try using Pandas it's much more powerful and will save you a lot of time.

 

[davids355]

Thanks for the pointers guys. It’s going to be very small- less than 50 domains and I’m mainly wanting to loop through them, so some stuff with each one and then update the date field so I know when I last worked on the domain.

Im actually thinking now about using awk to query and update a csv file via a bash script. Mainly because I think I have a fair idea of how to do that already.

 

[PHPInjected]

I am trying to store and query some domain data - just domain, date and scan results

What are the variables coming from the scan results? Just low, medium, & high vulnerability threat?

I'd recommend steering clear of Python's built-in CSV module since it can be a bit of a hassle

So I use IDLE for that presently and haven't had any issues at all with python's CSV module. I upload the CSV via a UI along with proxies and it has pretty much been a breeze outside of my proxies being rate limited.

Im actually thinking now about using awk to query and update a csv file via a bash script

I'd really just recommend a simple python/csv script depending on how the data is being accessed.

Im a bit bias though, I used CSV for literally every list I have, excluding raw keywords with no SV attached.

 

[SeasonedCode]

So I use IDLE for that presently and haven't had any issues at all with python's CSV module. I upload the CSV via a UI along with proxies and it has pretty much been a breeze outside of my proxies being rate limited.

yes you can use the CSV module without any issues but you will write a lot of code compared to pandas + in more complex data manipulation you may make mistakes that will result in weird logic

this is a simple script that updates a column using the CSV module

import csv

# Define the file paths
input_file = 'input.csv'
output_file = 'output.csv'

# Define the column index you want to update (0-based index)
column_index = 2  # Example: Update the third column

# Define the new value to be set in the entire column
new_value = 'UpdatedValue'

# Read the CSV file and update the column
with open(input_file, mode='r', newline='') as infile:
    reader = csv.reader(infile)
    rows = list(reader)

    # Update the desired column
    for row in rows:
        if len(row) > column_index:
            row[column_index] = new_value

# Write the updated data to a new CSV file
with open(output_file, mode='w', newline='') as outfile:
    writer = csv.writer(outfile)
    writer.writerows(rows)

print(f'Updated column {column_index} with value "{new_value}" and saved to {output_file}')

and this is the same logic but using pandas

import pandas as pd

# Define the file paths
input_file = 'input.csv'
output_file = 'output.csv'

# Define the column name or index you want to update
column_name = 'ColumnName'  # Use the column name or index
new_value = 'UpdatedValue'

# Read the CSV file into a DataFrame
df = pd.read_csv(input_file)

# Update the desired column
df[column_name] = new_value

# Write the updated DataFrame to a new CSV file
df.to_csv(output_file, index=False)

print(f'Updated column "{column_name}" with value "{new_value}" and saved to {output_file}')

PS: the provided code examples made by GPT

 

[davids355]

What are the variables coming from the scan results? Just low, medium, & high vulnerability threat?


So I use IDLE for that presently and haven't had any issues at all with python's CSV module. I upload the CSV via a UI along with proxies and it has pretty much been a breeze outside of my proxies being rate limited.



I'd really just recommend a simple python/csv script depending on how the data is being accessed.

Im a bit bias though, I used CSV for literally every list I have, excluding raw keywords with no SV attached.

Im using wpscan wordpress vulnerability database. I’m planning to just run the scan, check if there are vulnerabilities and if so, send an email with scan results attached, then check on remaining credits and if there are sufficient, continue scanning the next domain in the list, each time updating the last scanned date for each domain.

I might build a crude version in bash with awk and then I’ll try to build it in python afterwards; I had a quick. Look at pythons csv library and it looks good. But I know it’ll take me a while to learn everything I need such as reading and writing to csv, looping through the results, emailing and so on.

 

[Deals2Pro]

In my opinion for quantative data with complex relations MySQL or PostgreSQL are good. For flexible,modular data needs MongoDB, Redis, or Elasticsearch data bases are best.

 

[PHPInjected]

yes you can use the CSV module without any issues but you will write a lot of code compared to pandas + in more complex data manipulation you may make mistakes that will result in weird logic

this is a simple script that updates a column using the CSV module


PS: the provided code examples made by GPT

Have you ran that code provided in pandas?

I was gonna say, that's in-depth commented code. I always recommend commenting everything because in 3 months when you come back to it, you can piece what that section/block of code is supposed to be doing, but this is overkill. Lol

 

[TOR SEO]

my suggestion is start coding on py its more more flexibility if you give some time on it sure you will get your best py bot. you mentioned you know bash basic appreciate it but py is more comfortable for you if you start learning & yes OP we have right now chatgpt you can use gpt as a support don't depends full on gpt best wishes for your upcoming project

 

[davids355]

Thought I’d update this thread for rhe record. I ended up cheating a little bit as I found a ready made script online (written in bash) that did most of what I needed.

I just modified it slightly to achieve the desired result, now it’s working perfectly.

I did have a little flurry into python though, for another task. So I have at least started my python journey as well.

 

[TOR SEO]

Thought I’d update this thread for rhe record. I ended up cheating a little bit as I found a ready made script online (written in bash) that did most of what I needed.

I just modified it slightly to achieve the desired result, now it’s working perfectly.

I did have a little flurry into python though, for another task. So I have at least started my python journey as well.

still following your journey

 

[davids355]

my suggestion is start coding on py its more more flexibility if you give some time on it sure you will get your best py bot. you mentioned you know bash basic appreciate it but py is more comfortable for you if you start learning & yes OP we have right now chatgpt you can use gpt as a support don't depends full on gpt best wishes for your upcoming project

still following your journey

Thanks. And incidentally I did use ChatGPT to get my first python script working.

 

[TOR SEO]

Thanks. And incidentally I did use ChatGPT to get my first python script working.

i told you bro. ggs thanks to take the advice appreciate it.

 

[CyberCommander]

I am trying to store and query some domain data - just domain, date and scan results (I’m planning to do some periodic vulnerability scanning on a few sites).

What’s the best way of doing it? At the moment I’ve been experimenting with bash scripting and Mariadb but I don’t know if that’s over kill.

I really just want to loop through a list of domains and then keep a record of when the last scan was ran for each domain in the list.

My programming knowledge is quite basic - I’ve only done a little bit of bash scripting really.

This is partly a functional requirement of mine and partly an excuse to learn a bit more about programming.

i use https://hackertarget.com/ to scan my site with open source tools from the cloud.

 

[davids355]

i use https://hackertarget.com/ to scan my site with open source tools from the cloud.

I’m using wpscan.com

 

[CyberCommander]

I’m using wpscan.com

if you wanna use a free tool to scan CMS i can reccomend also https://siwecos.de/en/