faceplate077
Registered Member
- Jul 25, 2024
- 75
- 19
Hi everyone,
I’m working on a crawler to scan JS and PHP files, along with page content, looking for specific patterns with regex. I’ve developed versions in JS, Python, and tested one in Go. They work fine when I’m running single-threaded scans, but I’m running into serious issues when trying to improve performance with threading, chunking, goroutines, or similar methods. As soon as I add these optimizations, the crawler fails to properly scan each domain on my test list.
The list only includes vulnerable domains I’ve set up for testing, so I’m pretty sure the problem is in the script itself. I’m puzzled because everything works without the performance tweaks, but I can’t get it stable when I try to increase the speed.
If anyone has experience with this or ideas on what might be going wrong, any help would be hugely appreciated!
Thanks!
I’m working on a crawler to scan JS and PHP files, along with page content, looking for specific patterns with regex. I’ve developed versions in JS, Python, and tested one in Go. They work fine when I’m running single-threaded scans, but I’m running into serious issues when trying to improve performance with threading, chunking, goroutines, or similar methods. As soon as I add these optimizations, the crawler fails to properly scan each domain on my test list.
The list only includes vulnerable domains I’ve set up for testing, so I’m pretty sure the problem is in the script itself. I’m puzzled because everything works without the performance tweaks, but I can’t get it stable when I try to increase the speed.
If anyone has experience with this or ideas on what might be going wrong, any help would be hugely appreciated!
Thanks!