Scrapebox- 30,000,000 urls scraped... cant remove dupes

K

keith88

Regular Member

Sep 14, 2010

287

23

• Jan 10, 2014

• #1

hey guys,

I'm trying to clean up a massive list with SB but when i attempt to remove duplicates it says the version only supports ansi text files...

How can i get around this????

 

fxphil

Senior Member

Jul 16, 2010

1,090

512

• Jan 10, 2014

• #2

Split it up maybe into 1mil each, then paste them right into notepad then copy them from notepad into scrapebox.

By pasting them into notepad it turns it into te most basic txt automatically then copying it to SB should work.

 

tahworld

Regular Member

Aug 16, 2013

456

397

• Jan 10, 2014

• #3

yeah you should split it

http://www.gdgsoft.com/gsplit/

 

K

keith88

Regular Member

Sep 14, 2010

287

23

• Jan 10, 2014
• Thread Starter Thread Starter

• #4

fxphil said:

Split it up maybe into 1mil each, then paste them right into notepad then copy them from notepad into scrapebox.

By pasting them into notepad it turns it into te most basic txt automatically then copying it to SB should work.

Click to expand...

hmmm I suppose theres no way around that eh... I did plan to split the file but only after the dupes were removed from the master list.

 

L

LarryKowalsky

Newbie

Jan 7, 2014

14

3

• Jan 10, 2014

• #5

Use gscraper, which dont have limit row in load lists

 

zenlagor

Regular Member

Apr 4, 2013

379

204

• Jan 10, 2014

• #6

Try Textpad or excel.

 

K

keith88

Regular Member

Sep 14, 2010

287

23

• Jan 10, 2014
• Thread Starter Thread Starter

• #7

LarryKowalsky said:

Use gscraper, which dont have limit row in load lists

Click to expand...

I tried but freezes up on me....

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,846

• Jan 10, 2014

• #8

Try using Notepad++
http://notepad-plus-plus.org/

 

K

keith88

Regular Member

Sep 14, 2010

287

23

• Jan 10, 2014
• Thread Starter Thread Starter

• #9

zenlagor said:

Try Textpad or excel.

Click to expand...

Able to handle massive files??

 

W

Winternacht

Junior Member

Jan 7, 2011

113

46

• Jan 10, 2014

• #10

try this...

http://www.scrapebox.com/free-dupe-remove

works with files above 1 Million entries.

 

K

keith88

Regular Member

Sep 14, 2010

287

23

• Jan 10, 2014
• Thread Starter Thread Starter

• #11

Winternacht said:

try this...

http://www.scrapebox.com/free-dupe-remove

works with files above 1 Million entries.

Click to expand...

Yup thats what I'm using bro.....

thats the one that gives me the ansi message...

problem is I cant even open the file because its so large...

 

tahworld

Regular Member

Aug 16, 2013

456

397

• Jan 11, 2014

• #12

Hey Keith you can PM me the file. I'll split it for you for free.

 

R

Robbie1

Junior Member

Jan 16, 2012

112

20

• Jan 11, 2014

• #13

One of scrapebox's free addons is an ansi converter. Try running it through there to see if stops the error from coming up.

 

J

Javado

Junior Member

Jun 26, 2012

188

79

• Jan 11, 2014

• #14

You can send me the list and i will remove the dupes.

 

alaltaierii

Supreme Member

Jun 11, 2010

1,494

388

• Jan 11, 2014

• #15

You don't need to split it. Just use loopline's High Speed Tool.

 

W

We Go

Junior Member

May 8, 2012

164

56

• Jan 11, 2014

• #16

keith88 said:

Able to handle massive files??

Click to expand...

Using Excel 2013, my experience is that doing duplicate removal with the data tool was quick even with 200K rows, but running any macro would hit a known barrier at 32767 rows.

 

K

kinaks

Junior Member

Mar 27, 2012

192

48

• Jan 11, 2014

• #17

there was a software mentioned here: http://www.blackhatworld.com/blackh...ow-easily-build-huge-sites-lists-gsa-ser.html the software name "once is enough" hope this will help you.

parley?

 

bartosimpsonio

Elite Member

Executive VIP

Jr. VIP

Mar 21, 2013

24,828

31,589

• Jan 11, 2014

• #18

There's this little program called Linux which comes with every imaginable text processing tool ever deviced. Then run this

Code:

cat bigfile.txt | sort | uniq > newfile.txt

BTW you can get this stuff for Windows. Search for cygwin

 

B

blackhat777

Elite Member

Jun 25, 2011

1,784

671

• Jan 11, 2014

• #19

Sort the file by name, split it into parts of 1 million each.
Remove duplicates from each file by importing it in scrapebox harvestor. Then again merge the records within single files and again remove the duplicates.
Will need some manual work.


Otherwise, if you have a database, just create a table with one column, insert everything in that table and then remove the duplicates.