How to Clean 1 million backlinks!

lostgringos said:

Anyone know how we can run all the negative keywords at once?

Click to expand...

lostgringos said:

Anyone know how we can run all the negative keywords at once?

Click to expand...

Sweetfunny said:

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

Click to expand...

Mokodoki said:

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

Click to expand...

HoNeYBiRD said:

i needed something very similar, i wanted to copy those urls only which contain an exact search pattern from a few million lines

that's when Vim walked into the picture , so finally i figured how you can copy all lines containing an exact search pattern to the clipboard using Vim:

HTML:

qaq
:g/\<your_search_pattern\>/y A
:let @+ = @a

but you need the direct opposite, which is even more easier, you can use the following command to remove all lines which contain a search pattern (use it w/o the underlines of course):

HTML:

:g/your_search_pattern_here/d

if you want to search and remove lines which contain more complicated patterns, you can do that also using Vim:

Code:

http://vim.wikia.com/wiki/Delete_all_lines_containing_a_pattern

Click to expand...

thelonewolf said:

For deleting all the websites with "sex" or anything else in the name, I import the list into URL's Harvested in Scrapebox then Remove/Filter and select Remove Urls Containing... and then enter in "sex" and remove all those URLs. Hope this helps!

Click to expand...

linkdropper4 said:

I use Excel and a formula like this:

=if(iserror(find("KEYWORD",A1)),0,1)

Then copy and paste values in that column, sort from highest to lowest and delete all the rows that have 1s in them.

Click to expand...

Sweetfunny said:

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

Click to expand...

Sweetfunny said:

Remove URL's containing entries from... Then select your list of bad words, and all URL's in the grid containing those words will be removed. I tried your 459 bad words against a 100k list, and it cleaned it in 4.8 seconds.

Click to expand...

lostgringos said:

It looks like this is a recent update/feature to Scrapebox!

Click to expand...

<?php

function trim_value(&$value) {
if (is_string($value))
$value = trim($value);
}

function trim_array($arr) {
array_walk($arr, 'trim_value');
return array_filter($arr);
}

$backlinks = trim_array(file("yourbacklinks.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));
$badwords = trim_array(file("badwords.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));

foreach($badwords as $badword) {
$total = count($backlinks);
for($i=0; $i < $total; $i++) {
if (stristr($backlinks[$i], $badword)) {
unset($backlinks[$i]);
}
}
$backlinks = array_values($backlinks);
}

$backlinks = array_unique($backlinks);

file_put_contents("newbacklinks.txt", implode("\r\n", $backlinks));

?>

Click to expand...

Mokodoki said:

I use Textpipe as my primary text editor for SB lists as it can easily handle huge lists that programs like Notepad and even Notepad++ can't deal with. With my crappy laptop, I recently coerced it into sorting and removing duplicates from a 2.5gb keyword list although it took it a while.

Using Textpipe, I select the filter to remove lines based on a pattern list. All the exclusion words that I want to remove are in the list, and it will go through and delete all matching URLs.

Textpipe is pretty cool for everything but some unicode manipulation. It can merge, sort, remove duplicates, split much larger lists than even SB's duperemove add-on and do it much quicker. It can also fix formatting errors and a bunch of other stuff I can't think of right now.

Click to expand...