Hi all so i have several very large text files generated from scrapebox some ranging from 2gb upto 9gb and i cannot work with files this big using scrapebox as it will crash. one option has been gvim which is great but when files get to the 5gb and higher it can have some issues so i have gotten textpipe on the recommendation of some of the black hat posts but now im at a loss of how to exactly go about using this program. What im mainly looking todo is remove duplicate urls on some occasions removing duplicate domains and also stripping urls down to the domain level. Anyone have any insight as to how i could go about accomplishing this with textpipe?
Cleaning large text files with Textpipe instead of scrapebox
RedLable
Regular Member
- Feb 16, 2011
- 244
- 32
If you have a file which is 9gb why not break it into smaller lots and just use scrapebox? would this not be quicker then using regx and textpipe? if you are unsure of regx? if not google "regular-expressions-cheat-sheet-v2.pdf" there should be a free version about to help.
HatIsBlacked
Regular Member
- Dec 30, 2010
- 227
- 57
You are better off programming it yourself with C/C++ and saving yourself $400. What's so hard about opening up a text file, reading it in chunks into a buffer, processing, writing out in chunks, and closing the file?
SlimProxies
Supreme Member
- Oct 4, 2011
- 1,423
- 1,324
Yeap but OP may not have any coding skills, and everyone knows that starting coding, even small apps to read text files is very hard.
Well i tried breaking the file up into smaller ones so i could use scrapebox but this was extremly time consuming especially since i have many many gb of text files to go through. I already own textpipe so the cost is not an issue and i know it can handle multiple commands so its an ideal piece of software to work with i just dont know how to go about getting it to work with urls. As for coding my own thing i honestly dont have any experience writing code i know how useful it would be but at the moment and in the near future im not going to learn so im still looking for anyone who knows how to go about achieving these goals using textpipe.
Russian-Czar
Regular Member
- Feb 10, 2012
- 227
- 68
This is a really good freeware for multiple text editing
http://www.replsoft.com/plugins.html
http://www.replsoft.com/plugins.html
- Jan 25, 2009
- 6,378
- 3,774
Are you using the dupe remove addon in scrapebox? It can work with up to 180 million lines. The size files your quoting should have less then 150 million lines. Worst case you could hack it in half, with the dupe remove addon and work with like 3GB files and then once those 2 or 3 are done merge them together and do it.
You can try the "hi speed" tool I have here, it will do the job too.
http://scrapeboxmarketplace.com/free-tools/scrapebox-helper-tools
Although I would consider scrapeboxes dupe remove addon more stable.
You can try the "hi speed" tool I have here, it will do the job too.
http://scrapeboxmarketplace.com/free-tools/scrapebox-helper-tools
Although I would consider scrapeboxes dupe remove addon more stable.
datamystic
Newbie
- Feb 7, 2011
- 1
- 2
For removing duplicate domains, use Filters\Remove\Duplicate lines.
Here is how the filter list should look to strip urls down to the domain level:
|
|--Extract URLs
|
|--Comment...
| | strip domains to root level
| |
| +--Perl pattern [https?://[^/]*?] with [$0\r\n]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [X] Extract matches
| Maximum text buffer size 4096
| [ ] Maximum match (greedy)
| [ ] Allow comments
| [X] '.' matches newline
| [ ] UTF-8 Support
|
|--Ascending ANSI sort (case insensitive), remove duplicates, length 4096
|