500 threads on GSA but I've millions of link and it's not fast enough

Any solution?
Should I increase the number of threads? I'm worried this will crash GSA.
PC spec and internet are not the problem.

Or is it just normal to take weeks to finish the job when you've millions of links?

Update: I'm trying 3.500 threads and works. 3.200 LpM.
But it's still not enough for the amount of links I have.

reznev799 said:

Any solution?
Should I increase the number of threads? I'm worried this will crash GSA.
PC spec and internet are not the problem.

Or is it just normal to take weeks to finish the job when you've millions of links?

Click to expand...

at that volume adding more threads usually stops helping, the real bottleneck becomes how many of those urls are actually postable. if the list is a raw scrape, a big part of your threads is spent downloading pages that don't match any engine or are already dead.

i would clean the list outside the project first, remove duplicate domains, then run it through identify and sort so ser only gets urls it has an engine for. after that split it into a few smaller projects instead of one huge one, and set verification to a longer interval, because verifying millions of submissions eats threads too.

also check html timeout and the proxy pool, with 3500 threads weak proxies turn into timeouts that just look like slow lpm.