How to Clean 1 million backlinks!

If your a frequent user here on BHW you will see free backlinks offered. The problem is that most of these backlink lists have all kinds of porno in them, references to nazis, child porn and more.

My notepad will not handle a huge list so I use Notepad+++. I search for sex as one example and I can actually get all the urls with that keyword in them. Sometimes a list of 3,000. But for some reason I cannot delete them all at one time.

How do you clean your lists?

I would like to know if there is a fast way of doing this.

Thanks

lostgringos said:

If your a frequent user here on BHW you will see free backlinks offered. The problem is that most of these backlink lists have all kinds of porno in them, references to nazis, child porn and more.

My notepad will not handle a huge list so I use Notepad+++. I search for sex as one example and I can actually get all the urls with that keyword in them. Sometimes a list of 3,000. But for some reason I cannot delete them all at one time.

How do you clean your lists?

I would like to know if there is a fast way of doing this.

Thanks

Click to expand...

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

Good question, also wanting to know about this.

i needed something very similar, i wanted to copy those urls only which contain an exact search pattern from a few million lines

that's when Vim walked into the picture , so finally i figured how you can copy all lines containing an exact search pattern to the clipboard using Vim: but you need the direct opposite, which is even more easier, you can use the following command to remove all lines which contain a search pattern (use it w/o the underlines of course): if you want to search and remove lines which contain more complicated patterns, you can do that also using Vim:

For deleting all the websites with "sex" or anything else in the name, I import the list into URL's Harvested in Scrapebox then Remove/Filter and select Remove Urls Containing... and then enter in "sex" and remove all those URLs. Hope this helps!

- gotta love Linux...

I do not intend to hijack this post, but I think what you are referring to here is close to what I need to do.

I have a text document with 1387 lines.

I need to remove duplicates, however I do not know which lines have duplicates.

Is there a way to have a program actually read every line and remove a line if it is an exact match of another line in a text document?

I hope that makes sense.

If you can help, will you please PM me.

Thank you...

You can always use xrumer to do this...

SEORocks said:

I do not intend to hijack this post, but I think what you are referring to here is close to what I need to do.

I have a text document with 1387 lines.

I need to remove duplicates, however I do not know which lines have duplicates.

Is there a way to have a program actually read every line and remove a line if it is an exact match of another line in a text document?

I hope that makes sense.

If you can help, will you please PM me.

Thank you...

Click to expand...

Import the list into SB and remove the duplicate URL's

What a hell are you people talking about, do you even have scrapebox, you can use remove duplicate urls within seconds, and there is a remove urls containing a word you dont need, i cant believe there are posts like this.

I really did not know that I could specify urls to be deleted with keywords. There is so much to learn about Scrapebox. That saves me so much time---> giving my head a shake... I also gave thanks for positive comments here.

Ok I tried that in SB and it is fantastic. I have another question to ask. Do any of you know where I can find a list containing keywords that may offend Google i.e, porn, nazi, kiddie porn etc? Thanks

Mokodoki said:

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

Click to expand...

This.

Or in SB, load your URLs in harvester, and then select "Remove/Filter" > "URL's Containing" and key in your keyword.

Here's how you do it with notepad++

CTRL+H



Then click TextFX>TextFX edit>Delete blank lines

Here is a list of bad words you can use to delete bad urls from these lists...
ahole
anus
ash0le
ash0les
asholes
ass
Ass Monkey
Assface
assh0le
assh0lez
asshole
assholes
assholz
asswipe
azzhole
bassterds
bastard
bastards
bastardz
basterds
basterdz
Biatch
bitch
bitches
Blow Job
boffing
butthole
buttwipe
c0ck
c0cks
c0k
Carpet Muncher
cawk
cawks
Clit
cnts
cntz
cock
cockhead
cock-head
cocks
CockSucker
cock-sucker
crap
cum
cunt
cunts
cuntz
dick
dild0
dild0s
dildo
dildos
dilld0
dilld0s
dominatricks
dominatrics
dominatrix
dyke
enema
f u c k
f u c k e r
fag
fag1t
faget
fagg1t
faggit
faggot
fagit
fags
fagz
faig
faigs
fart
flipping the bird
fuck
fucker
fuckin
fucking
fucks
Fudge Packer
fuk
Fukah
Fuken
fuker
Fukin
Fukk
Fukkah
Fukken
Fukker
Fukkin
g00k
gay
gayboy
gaygirl
gays
gayz
God-damned
h00r
h0ar
h0re
hells
hoar
hoor
hoore
jackoff
jap
japs
jerk-off
jisim
jiss
jizm
jizz
knob
knobs
knobz
kunt
kunts
kuntz
Lesbian
Lezzian
Lipshits
Lipshitz
masochist
masokist
massterbait
masstrbait
masstrbate
masterbaiter
masterbate
masterbates
Motha Fucker
Motha Fuker
Motha Fukkah
Motha Fukker
Mother Fucker
Mother Fukah
Mother Fuker
Mother Fukkah
Mother Fukker
mother-fucker
Mutha Fucker
Mutha Fukah
Mutha Fuker
Mutha Fukkah
Mutha Fukker
n1gr
nastt
nigger;
nigur;
niiger;
niigr;
orafis
orgasim;
orgasm
orgasum
oriface
orifice
orifiss
packi
packie
packy
paki
pakie
paky
pecker
peeenus
peeenusss
peenus
peinus
pen1s
penas
penis
penis-breath
penus
penuus
Phuc
Phuck
Phuk
Phuker
Phukker
polac
polack
polak
Poonani
pr1c
pr1ck
pr1k
pusse
pussee
pussy
puuke
puuker
queer
queers
queerz
qweers
qweerz
qweir
recktum
rectum
retard
sadist
scank
schlong
screwing
semen
sex
sexy
Sh!t
sh1t
sh1ter
sh1ts
sh1tter
sh1tz
shit
shits
shitter
Shitty
Shity
shitz
Shyt
Shyte
Shytty
Shyty
skanck
skank
skankee
skankey
skanks
Skanky
slut
sluts
Slutty
slutz
son-of-a-bitch
tit
turd
va1jina
vag1na
vagiina
vagina
vaj1na
vajina
vullva
vulva
w0p
wh00r
wh0re
whore
xrated
xxx
b!+ch
bitch
blowjob
clit
arschloch
fuck
shit
ass
asshole
b!tch
b17ch
b1tch
bastard
bi+ch
boiolas
buceta
c0ck
cawk
chink
cipa
clits
cock
cum
cunt
dildo
dirsa
ejakulate
fatass
fcuk
fuk
fux0r
hoer
hore
jism
kawk
l3itch
l3i+ch
lesbian
masturbate
masterbat
masterbat3
motherfucker
s.o.b.
mofo
nazi
nigga
nigger
nutsack
phuck
pimpis
pusse
pussy
scrotum
sh!t
shemale
shi+
sh!+
slut
smut
teets
tits
boobs
b00bs
teez
testical
testicle
titt
w00se
jackoff
wank
whoar
whore
damn
dyke
fuck
shit
@$$
amcik
andskota
arse
assrammer
ayir
bi7ch
bitch
bollock
breasts
butt-pirate
cabron
cazzo
chraa
chuj
Cock
cunt
d4mn
daygo
dego
dick
dike
dupa
dziwka
ejackulate
Ekrem
Ekto
enculer
faen
fag
fanculo
fanny
feces
feg
Felcher
ficken
fitt
Flikker
foreskin
Fotze
Fu(
fuk
futkretzn
gay
gook
guiena
h0r
h4x0r
hell
helvete
hoer
honkey
Huevon
hui
injun
jizz
kanker
kike
klootzak
kraut
knulle
kuk
kuksuger
Kurac
kurwa
kusi
kyrpa
lesbo
mamhoon
masturbat
nazi
merd
mibun
monkleigh
mouliewop
muie
mulkku
muschi
nazis
nepesaurio
nigger
orospu
paska
perse
picka
pierdol
pillu
pimmel
piss
pizda
poontsee
poop
porn
p0rn
pr0n
preteen
pula
pule
puta
puto
qahbeh
queef
rautenberg
schaffer
scheiss
schlampe
schmuck
screw
sh!t
sharmuta
sharmute
shipal
shiz
skribz
skurwysyn
sphencter
spic
spierdalaj
splooge
suka
b00b
testicle
titt
twat
vittu
wank
wetback
wichser
wop
yed
zabourah

@paincake excellent use of Notepad++. I use this software all the time. Thanks

now we could run all of these at once it would be really good...

lostgringos said:

now we could run all of these at once it would be really good...

Click to expand...

Macro?

Thanks paincake, your method is the best since Notepad++ is free.

Anyone know how we can run all the negative keywords at once?