How to work with large files in Scrapebox?

Guys, my SB freezes when I try delete duplicated URLs from 500K scraped URL list.

Now, I restarted and trying to open it at least to split it, but no such luck. Can't open - freezes.

Now I tried to split files with Gsplit, but it created some .gns files - no idea how to convert them into txt to cvs.

Can you please advise what is the easiest way to go about working with large scraped lists? (Good thing i was able to sve it, but after that cannot perform any operation)

there's already quite a few threads about this:
- you can follow crazyflx's guide how to deal with large .txt files, it was posted here as well by him, i cannot find now tho, so here it is from his blog: - you can use hjsplit, then rename the results back into .txt
- there was a thread not too long ago in the download section with a tool made by a bhw member for this task (splitting large txt files into smaller files with ease)

Ys, that's exactly the post i've found and his instructions don't seem to work.

s soon as you type the : the cursor will jump to the bottom left hand corner of the screen

Click to expand...

Well, in my case it doesn't - it just edits text instead.

Do you use the same program?

I can clean up your list but it will also remove URLs which differ on subdomain meaning it will leave only subdomain.yahoo.com/etcetc from a list that looks like this:

sport.yahoo.com/etcetc
news.yahoo.com/etcetc
politics.yahoo.com/etcetc

If that is unacceptable, load your file in wordpad and copy and paste chunks of lines in new text files.

observer said:

Ys, that's exactly the post i've found and his instructions don't seem to work.



Well, in my case it doesn't - it just edits text instead.

Do you use the same program?

Click to expand...

yea, i'm using the same program following crazyflx's step by step guide word from word

pay attention to type in the colon character first and just after copy-paste what crazyflx wrote

type the ":" character and then type the ":" character and then type the ":" character and then
it should work!

but as i already mentioned above you can use hjsplit to cut the bigger files into smaller pieces and just then use SB to remove dupes

sweetfunny just released a tool to bulk merge as well as bulk dup check 1mil+ urls though i doesnt seem to be as accurate unless i am missing something