How to Clean 1 million backlinks!

[TheMatrix]

Anyone know how we can run all the negative keywords at once?

SB is now updated to do that.

 

[Freakendorf]

curious to find out myself

 

[xxtoni]

Interesting, I never thought about that but I do remember that I once saw porn related backlinks to my site.

 

[Sweetfunny]

Anyone know how we can run all the negative keywords at once?

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

 

[gortva]

thanks

 

[Checkmate]

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

Is there a way we can get a global badword list block so we all can keep adding to the list like the URL blacklist cloud?

 

[linkdropper4]

I use Excel and a formula like this:

=if(iserror(find("KEYWORD",A1)),0,1)

Then copy and paste values in that column, sort from highest to lowest and delete all the rows that have 1s in them.

 

[lostgringos]

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

i needed something very similar, i wanted to copy those urls only which contain an exact search pattern from a few million lines

that's when Vim walked into the picture , so finally i figured how you can copy all lines containing an exact search pattern to the clipboard using Vim:

qaq
:g/\<your_search_pattern\>/y A
:let @+ = @a

but you need the direct opposite, which is even more easier, you can use the following command to remove all lines which contain a search pattern (use it w/o the underlines of course):

:g/your_search_pattern_here/d

if you want to search and remove lines which contain more complicated patterns, you can do that also using Vim:

http://vim.wikia.com/wiki/Delete_all_lines_containing_a_pattern

For deleting all the websites with "sex" or anything else in the name, I import the list into URL's Harvested in Scrapebox then Remove/Filter and select Remove Urls Containing... and then enter in "sex" and remove all those URLs. Hope this helps!

I use Excel and a formula like this:

=if(iserror(find("KEYWORD",A1)),0,1)

Then copy and paste values in that column, sort from highest to lowest and delete all the rows that have 1s in them.

I am not sure I follow you...where do you put this script?

 

[lostgringos]

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

Thanks so much I am sure this will helps lots of people. Giving you a rep and thanks...vry good information...

 

[lostgringos]

Ok this is excellent information. I think this method would also be great for cleaning some of these fast indexing lists going around too. If anyone has an Indexing List please post it here either cleaned or not. Thanks everyone for sharing here. Those who have shared I have given a "rep" to. Others here can to the same. It's nice to get a pat on the back!

 

[vinaltween]

that are very usefull for me.

 

[lostgringos]

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

It looks like this is a recent update/feature to Scrapebox!

 

[TheMatrix]

It looks like this is a recent update/feature to Scrapebox!

Indeed, it is!

In 1.15.26!

 

[xiojin]

The Fast and stable replace badwords from url list just use my php script.
Put your yourbacklinks.txt and badwords.txt file in the same of this script (index.php) and run at localhost. Your new list (backlinks) file is newbacklinks.txt
Here my script :

<?php

function trim_value(&$value) {
if (is_string($value))
$value = trim($value);
}

function trim_array($arr) {
array_walk($arr, 'trim_value');
return array_filter($arr);
}

$backlinks = trim_array(file("yourbacklinks.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));
$badwords = trim_array(file("badwords.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));

foreach($badwords as $badword) {
$total = count($backlinks);
for($i=0; $i < $total; $i++) {
if (stristr($backlinks[$i], $badword)) {
unset($backlinks[$i]);
}
}
$backlinks = array_values($backlinks);
}

$backlinks = array_unique($backlinks);

file_put_contents("newbacklinks.txt", implode("\r\n", $backlinks));

?>

If you like this script, don't forget to push thanks Enjoy it!

 

[zerackam]

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

Mokodoki can you tell me how exactly to go about stripping lists of links to the root, removing duplicate domains and fixing formatting with textpipe?