1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Clean 1 million backlinks!

Discussion in 'Black Hat SEO' started by lostgringos, Sep 12, 2011.

  1. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    If your a frequent user here on BHW you will see free backlinks offered. The problem is that most of these backlink lists have all kinds of porno in them, references to nazis, child porn and more.

    My notepad will not handle a huge list so I use Notepad+++. I search for sex as one example and I can actually get all the urls with that keyword in them. Sometimes a list of 3,000. But for some reason I cannot delete them all at one time.

    How do you clean your lists?

    I would like to know if there is a fast way of doing this.

    Thanks
     
  2. Mokodoki

    Mokodoki Regular Member

    Joined:
    Feb 26, 2011
    Messages:
    217
    Likes Received:
    354
    Occupation:
    Graphic Artist | Fulltime Student
    I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

    Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

    Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now. :p
     
    • Thanks Thanks x 5
  3. Checkmate

    Checkmate Elite Member

    Joined:
    Aug 9, 2010
    Messages:
    1,534
    Likes Received:
    637
    Good question, also wanting to know about this.
     
  4. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    5,889
    Likes Received:
    7,127
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    i needed something very similar, i wanted to copy those urls only which contain an exact search pattern from a few million lines

    that's when Vim walked into the picture :), so finally i figured how you can copy all lines containing an exact search pattern to the clipboard using Vim:
    HTML:
    qaq
    :g/\<your_search_pattern\>/y A
    :let @+ = @a
    but you need the direct opposite, which is even more easier, you can use the following command to remove all lines which contain a search pattern (use it w/o the underlines of course):
    HTML:
    :g/your_search_pattern_here/d
    if you want to search and remove lines which contain more complicated patterns, you can do that also using Vim:
    Code:
    http://vim.wikia.com/wiki/Delete_all_lines_containing_a_pattern
     
    • Thanks Thanks x 3
  5. thelonewolf

    thelonewolf Newbie

    Joined:
    May 13, 2011
    Messages:
    19
    Likes Received:
    1
    For deleting all the websites with "sex" or anything else in the name, I import the list into URL's Harvested in Scrapebox then Remove/Filter and select Remove Urls Containing... and then enter in "sex" and remove all those URLs. Hope this helps!
     
    • Thanks Thanks x 1
  6. bredsaal

    bredsaal Newbie

    Joined:
    Dec 10, 2009
    Messages:
    13
    Likes Received:
    4
    Occupation:
    Unemployed
    Location:
    Denmark
    Code:
    grep -v sex urllist > clean_list
    - gotta love Linux... :)
     
    • Thanks Thanks x 2
  7. SEORocks

    SEORocks Junior Member

    Joined:
    Jul 14, 2011
    Messages:
    188
    Likes Received:
    104
    I do not intend to hijack this post, but I think what you are referring to here is close to what I need to do.

    I have a text document with 1387 lines.

    I need to remove duplicates, however I do not know which lines have duplicates.

    Is there a way to have a program actually read every line and remove a line if it is an exact match of another line in a text document?

    I hope that makes sense.

    If you can help, will you please PM me.

    Thank you...
     
  8. reinie

    reinie Elite Member

    Joined:
    Jan 16, 2009
    Messages:
    1,574
    Likes Received:
    1,040
    You can always use xrumer to do this...
     
  9. Checkmate

    Checkmate Elite Member

    Joined:
    Aug 9, 2010
    Messages:
    1,534
    Likes Received:
    637
    Import the list into SB and remove the duplicate URL's
     
    • Thanks Thanks x 1
  10. d3ky44

    d3ky44 Regular Member

    Joined:
    Feb 13, 2010
    Messages:
    216
    Likes Received:
    36
    Occupation:
    Web Designer
    Location:
    Google Battlefield
    What a hell are you people talking about, do you even have scrapebox, you can use remove duplicate urls within seconds, and there is a remove urls containing a word you dont need, i cant believe there are posts like this.
     
  11. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    I really did not know that I could specify urls to be deleted with keywords. There is so much to learn about Scrapebox. That saves me so much time---> giving my head a shake...:eek: I also gave thanks for positive comments here.
     
  12. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    Ok I tried that in SB and it is fantastic. I have another question to ask. Do any of you know where I can find a list containing keywords that may offend Google i.e, porn, nazi, kiddie porn etc? Thanks
     
  13. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    This.

    Or in SB, load your URLs in harvester, and then select "Remove/Filter" > "URL's Containing" and key in your keyword.
     
  14. paincake

    paincake Power Member

    Joined:
    Aug 18, 2010
    Messages:
    716
    Likes Received:
    3,099
    Home Page:
    Here's how you do it with notepad++

    CTRL+H

    [​IMG]

    Then click TextFX>TextFX edit>Delete blank lines
     
    • Thanks Thanks x 3
  15. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    Here is a list of bad words you can use to delete bad urls from these lists...
    ahole
    anus
    ash0le
    ash0les
    asholes
    ass
    Ass Monkey
    Assface
    assh0le
    assh0lez
    asshole
    assholes
    assholz
    asswipe
    azzhole
    bassterds
    bastard
    bastards
    bastardz
    basterds
    basterdz
    Biatch
    bitch
    bitches
    Blow Job
    boffing
    butthole
    buttwipe
    c0ck
    c0cks
    c0k
    Carpet Muncher
    cawk
    cawks
    Clit
    cnts
    cntz
    cock
    cockhead
    cock-head
    cocks
    CockSucker
    cock-sucker
    crap
    cum
    cunt
    cunts
    cuntz
    dick
    dild0
    dild0s
    dildo
    dildos
    dilld0
    dilld0s
    dominatricks
    dominatrics
    dominatrix
    dyke
    enema
    f u c k
    f u c k e r
    fag
    fag1t
    faget
    fagg1t
    faggit
    faggot
    fagit
    fags
    fagz
    faig
    faigs
    fart
    flipping the bird
    fuck
    fucker
    fuckin
    fucking
    fucks
    Fudge Packer
    fuk
    Fukah
    Fuken
    fuker
    Fukin
    Fukk
    Fukkah
    Fukken
    Fukker
    Fukkin
    g00k
    gay
    gayboy
    gaygirl
    gays
    gayz
    God-damned
    h00r
    h0ar
    h0re
    hells
    hoar
    hoor
    hoore
    jackoff
    jap
    japs
    jerk-off
    jisim
    jiss
    jizm
    jizz
    knob
    knobs
    knobz
    kunt
    kunts
    kuntz
    Lesbian
    Lezzian
    Lipshits
    Lipshitz
    masochist
    masokist
    massterbait
    masstrbait
    masstrbate
    masterbaiter
    masterbate
    masterbates
    Motha Fucker
    Motha Fuker
    Motha Fukkah
    Motha Fukker
    Mother Fucker
    Mother Fukah
    Mother Fuker
    Mother Fukkah
    Mother Fukker
    mother-fucker
    Mutha Fucker
    Mutha Fukah
    Mutha Fuker
    Mutha Fukkah
    Mutha Fukker
    n1gr
    nastt
    nigger;
    nigur;
    niiger;
    niigr;
    orafis
    orgasim;
    orgasm
    orgasum
    oriface
    orifice
    orifiss
    packi
    packie
    packy
    paki
    pakie
    paky
    pecker
    peeenus
    peeenusss
    peenus
    peinus
    pen1s
    penas
    penis
    penis-breath
    penus
    penuus
    Phuc
    Phuck
    Phuk
    Phuker
    Phukker
    polac
    polack
    polak
    Poonani
    pr1c
    pr1ck
    pr1k
    pusse
    pussee
    pussy
    puuke
    puuker
    queer
    queers
    queerz
    qweers
    qweerz
    qweir
    recktum
    rectum
    retard
    sadist
    scank
    schlong
    screwing
    semen
    sex
    sexy
    Sh!t
    sh1t
    sh1ter
    sh1ts
    sh1tter
    sh1tz
    shit
    shits
    shitter
    Shitty
    Shity
    shitz
    Shyt
    Shyte
    Shytty
    Shyty
    skanck
    skank
    skankee
    skankey
    skanks
    Skanky
    slut
    sluts
    Slutty
    slutz
    son-of-a-bitch
    tit
    turd
    va1jina
    vag1na
    vagiina
    vagina
    vaj1na
    vajina
    vullva
    vulva
    w0p
    wh00r
    wh0re
    whore
    xrated
    xxx
    b!+ch
    bitch
    blowjob
    clit
    arschloch
    fuck
    shit
    ass
    asshole
    b!tch
    b17ch
    b1tch
    bastard
    bi+ch
    boiolas
    buceta
    c0ck
    cawk
    chink
    cipa
    clits
    cock
    cum
    cunt
    dildo
    dirsa
    ejakulate
    fatass
    fcuk
    fuk
    fux0r
    hoer
    hore
    jism
    kawk
    l3itch
    l3i+ch
    lesbian
    masturbate
    masterbat
    masterbat3
    motherfucker
    s.o.b.
    mofo
    nazi
    nigga
    nigger
    nutsack
    phuck
    pimpis
    pusse
    pussy
    scrotum
    sh!t
    shemale
    shi+
    sh!+
    slut
    smut
    teets
    tits
    boobs
    b00bs
    teez
    testical
    testicle
    titt
    w00se
    jackoff
    wank
    whoar
    whore
    damn
    dyke
    fuck
    shit
    @$$
    amcik
    andskota
    arse
    assrammer
    ayir
    bi7ch
    bitch
    bollock
    breasts
    butt-pirate
    cabron
    cazzo
    chraa
    chuj
    Cock
    cunt
    d4mn
    daygo
    dego
    dick
    dike
    dupa
    dziwka
    ejackulate
    Ekrem
    Ekto
    enculer
    faen
    fag
    fanculo
    fanny
    feces
    feg
    Felcher
    ficken
    fitt
    Flikker
    foreskin
    Fotze
    Fu(
    fuk
    futkretzn
    gay
    gook
    guiena
    h0r
    h4x0r
    hell
    helvete
    hoer
    honkey
    Huevon
    hui
    injun
    jizz
    kanker
    kike
    klootzak
    kraut
    knulle
    kuk
    kuksuger
    Kurac
    kurwa
    kusi
    kyrpa
    lesbo
    mamhoon
    masturbat
    nazi
    merd
    mibun
    monkleigh
    mouliewop
    muie
    mulkku
    muschi
    nazis
    nepesaurio
    nigger
    orospu
    paska
    perse
    picka
    pierdol
    pillu
    pimmel
    piss
    pizda
    poontsee
    poop
    porn
    p0rn
    pr0n
    preteen
    pula
    pule
    puta
    puto
    qahbeh
    queef
    rautenberg
    schaffer
    scheiss
    schlampe
    schmuck
    screw
    sh!t
    sharmuta
    sharmute
    shipal
    shiz
    skribz
    skurwysyn
    sphencter
    spic
    spierdalaj
    splooge
    suka
    b00b
    testicle
    titt
    twat
    vittu
    wank
    wetback
    wichser
    wop
    yed
    zabourah
     
    • Thanks Thanks x 2
  16. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    @paincake excellent use of Notepad++. I use this software all the time. Thanks
     
  17. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    now we could run all of these at once it would be really good...
     
  18. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    Macro?
     
  19. shankshera

    shankshera Junior Member

    Joined:
    May 18, 2010
    Messages:
    193
    Likes Received:
    25
    Thanks paincake, your method is the best since Notepad++ is free. ;)
     
  20. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    Anyone know how we can run all the negative keywords at once?